Well... at least they got high-powered routers for $22K apiece. But, it's a shame there isn't a law on the books that selling the government something at 50X the market price / need is fraud.*
* Yes, I realize this would bankrupt half the military contractors as well.
Can't blame them for selling them something they asked for. Who's to say they were even told what they were for. This is government waste and incompetence, nothing more.
You are correct on the military contractors as well though. Lots of government waste and incompetence there as well.
somehow i doubt these sites have multiple uplinks, so they certainly don't need BGP routers. even if they're doing IPSec between some locations this can be handled by an old piece-of-junk $100 computer running BSD or nix, e.g. with isakmpd and pf.
stuff like this is why america's infrastructure is crumbling: dodgy contractors looking for an excuse to upcharge municipalities for irrelevant gear and services.
Here in Australia, our local DMV (RTA) is all MacBook Pros, all running Windows XP in virtualization. Would love to meet the dipshit that authorized this travesty.
You really want to advocate installing "old piece of junk" routers across the entire state? Have you ever managed a project this large? And if so did you use old piece of junk equipment?
I describe this as 'funny money'. The government is spending other people's money.
I imagine the government worker that signed off on this purchase worrying a lot more about a dollar increase in the local lunch special than millions spent on equipment.
It is also funny money because the marginal cost of producing one of these routers is more like $500, not $20k. It is just a transfer payment to the company, it isn't the same as if the government burned $20k worth of barrels of oil. Nothing near that amount in resources was actually wasted.
Not quite. Cisco will probably take this purchase as a signal to produce more unnecessarily large routers. Those people paid to make the unnecessary routers could have used their valuable skills elsewhere. They may convince more people to learn how to build unnecessary routers instead of something more valuable. It's waste all the way down.
Not exactly, unless you think those people will literally use their salaries from the activities you described to buy something and go out and burn it. It might be a mis-distribution, and it might spur misguided investment in an area, but it isn't all squandered to the tune of exactly $20,000 or anywhere close to that. The marginal cost is a much more accurate measure.
it's sad, but nothing new. either verizon is lying about who asked for the specific model, or some inexperienced network "architect" decided not to do the due diligence to spec out what was required, and just asked for the moon.
Private sector FTW. Please, please I hope everyone who reads this begins to understand that governments are not meant to allocate resources. The idea (promoted by politicians, mostly democrats) that we should give the government our money and they will spend it wisely "for the betterment of all" is purely, simply and fundamentally flawed.
Out government should be small. It should perform a few basic functions like making sure we are secure, our trading interests are promoted and our banking and financial markets are healthy. It should educate our children to the best standards in the world. Finally it should make sure we are energy independent and our food and water is uncontaminated. That's pretty much it.
Please don't make this a democrat/republican debate. This is the crooked way that all of our US government is spending our tax money. We need corporate money out of politics if we want to prevent these outlandish contracts from being awarded to the corporations that donate enough money to campaigns.
It's not a democrat/republican or big/small government thing. It's about corruption of the system.
Out of your comment, I agree most with that our government needs to help educate our children. Without an educated population, things only get worse.
how is this not a big/small government thing? We gave away billions of dollars in a stimulus program with little to no oversight that is very clearly getting wasted here.
While the republicans are certainly guilty of lots of spending and many things, democrats are clearly the party that is guilty of more spending and even bigger government.
There are plenty of countries whose governments are far to the left of the Democrats who spend money much more wisely, so I don't see it as inherently tied. For example, the Scandinavian countries have much larger governments proportionally, but generally spend more wisely, and have more transparent bidding/provisioning of their state-funded services, so they get more service and less waste in return for their money.
They probably do spend more wisely, but that doesn't necessarily mean they are spending on something they should be. For example in this case buying the cheaper routers would be spending more wisely, but could pretty easily be argued that this isn't something we should be spending money on. Certainly not at the federal level. Why is my tax money in Tennessee being spent on routers in WV?
There are a few components to this particular screw-up. 1 is big government spending (with little oversight) and 2 is one idiot in WV with a big budget who failed miserably at spending it.
But it could pretty easily be argued that it's something that we should be spending money on. The existence of an argument against a course of action doesn't, on its face, preclude taking that course of action. Such a line of thinking is, well, bizarre.
"1 is big government spending (with little oversight)"
The obvious problem here isn't "big government" spending. It's spending without oversight. This problem occurs all the time everywhere, in sectors both public and private. To claim that private businesses are immune to this by nature of their ability to fail and thus inherently a better mechanism than government is an absolute farce.
Take ownership of your government. To view the elimination of government as a solution to bad governance is both a lazy approach to problem solving and toxic to proper governing.
Thank you. Finally some common sense to counter the libertarian anti-government sentiment. Whenever they tell you that government run schools are failing, ask them why the government run schools in other countries are performing so much better. Whenever they say that government can't allocate resources efficiently, point them to the governments that have balanced their budgets. It is possible. You don't have to throw the baby out with the bathwater. The solution to bad government is not to abandon it, but to work ever harder to improve it.
Getting the money / corporate influence out of government would be the place to start.
democrats are clearly the party that is guilty of more spending and even bigger government.
Let me guess... military spending doesn't count as "more spending". The $800 billion allocated for 10 years of stimulus is a paltry sum compared to the $700+ billion we intended to spend every year on military, and we all know that money is spent very efficiently.
Meanwhile, federal spending has increased every year under both parties for the last two decades, so you don't really have a leg to stand on with regard to your "democrats are the big government spenders" FUD.
consider this statement politically agnostic, but military spending is the closest thing we have to govt subsidized R+D in this country. there several enormously useful and successful military inventions that no commercial company would have touched.
gps and the internet are two of the most notable and visible examples with astronomical infrastructure costs. no private company would undertake a risk like this yet both have substantially improved society. the microwave and digital cameras are two others that a private company may have eventually created.
this isn't an argument for military spending, just the fact that without a government body that focuses entirely on applied research and development the military is the next best thing.
Are you kidding? Look at the budgets (or lack thereof) under Bush and Obama. They aren't even in the same neighborhood. I'll be the first to agree that there is a lot of waste in defense spending. My first job was at a defense contractor working with the air force. I've seen it.
Just because there is waste in defense, doesn't mean there isn't more elsewhere. Defense is one of the few things are government is actually tasked with. All the other shit we waste money on isn't part of what they are supposed to be doing. Roads and armies, that's all I expect. I don't need the government to feed me or wipe my ass.
It's not a big/small government thing because you are being played by what people told you the two party system is about, i.e. republicans are for small government and democrats are for big government. It's a bunch of bullshit.
I'm sorry to burst your bubble, but our current presidents and congress members do what will appease the general population in the very near term, no matter what they are supposed to do according to their party lines. They are all equally guilty of spending.
You know, the TSA in the airports, which has a larger budget than the FBI? Guess who created and nurtured that entity? Your small government friends. And, who spent so much money on the wars in Iraq and Afghanistan that it is bankrupting the country. We can't even get involved in another war right now because we simply can't afford it. It is questionable if we will ever be able to lead a war like that again, due to budget constraints.
Then, we have the wall street bailouts. Which, by the way, aren't republicans supposed to be the only friends of wall street? Not if letting the banks fail mean that you are going to have to line up for the bread line tomorrow morning.
The big government people aren't exactly cherubs either. How many RIAA lawyers have been appointed to the Justice Department by the current administration? I think it's 5, so far. Enjoy additional Internet censorship.
Sorry to be such a downer, but things are just getting worse and the current two party system is a complete facade. I'm envious of countries like Germany that are seeing a surge in pirate party support. Unfortunately, it's going to take decades to undo this mess and, frankly, a large majority of baby boomers are going to need to expire naturally before the voting population comes back into balance.
As a Canadian I find the persistance of the "small vs. big" government narrative in American politics bemusing - there is no small government party in the states, only branding.
Very well said. I agree with your point that governments aren't great at allocating resources, and should concentrate on the basics.
I disagree with the sidebar that it's mostly Democrats trying to take our money. Both parties do it, just with different priorities and in different ways, with all the accepted terms used to discuss it.
They wouldn't be buying any routers without the federal dollars to build fiber so it's kind of a moot point in this situation. Anyone who has worked in the private sector can tell you this type of stuff happens constantly. Too many technically ignorant people making decisions is the problem public or private. A private contractor could have sold them on the same exact package for the same reasons. Maybe even charged more for it.
As for your core point about the role of government... just look at the top 10 economies of the world. Most are not libertarian paradises and they're doing OK. It seems to me both models could probably work if executed properly. I doubt one model is inherently right and one is inherently wrong. Most likely some hybrid is the best solution for everyone.
In other news, JP Morgan just posted a $2 billion loss due to their hedge trading unit. One of the paragons of the capitalist system. Big and old and supposedly "successful" and supposedly run by very smart people who deserve very high pay because what they're doing is very hard and technical, and only they know how to manage resources efficiently, unlike say the government. Let's not even get into the paychecks of all the CEO's, traders, partners, huge sums of money for what in many cases involves just hiring other people and telling them what to do, pointing out the obvious, etc. Meanwhile the little people, the people that do the vast majority of the real and necessary work in our communities --- teachers, doctors, repairmen, cops, firefighters, road crew, waiters, delivery folks, carpenters, electricians, etc. --- get the least amount of pay. Because that's "efficient". If that's an efficient allocation of resources, maybe that's a bad thing.
I think we agree on the idea that government should be small, or at least as small as possible. And we agree on the roles it should play. But I don't buy into the argument that government is inherently bad at allocating resources efficiently and private businesses are inherently good. I've seen way way too many examples that contradict the latter. I think it ultimately comes to individual situations and the choices of individuals, what information is available to them, their goals, where their financial interests pull them.
If you had worked in a big company environment, you would know that this kind of nonsense happens all the time, in both private and public environments. 30000$ routers are pocket money compared to some of the things I have seen (millions of $ spent to develop and maintain in-house software that cannot do 10 % of what a basic open source solution would do).
When you reach a certain size, and you have many competing groups, all with their agenda with the same company, it becomes difficult to allocate resources correctly, especially in a domain that is not considered "core of business" (IT being a common case, sadly).
you should see the crazy amount big companies spend on IBM tools. people pay big money for clearcase.
licenses for clearcase cost enormous amounts. yet clearcase is missing many of the features found in open-source (and free) distributed version control systems. meanwhile, there is no free alternative to the routers, which may actually be necessary at some of the installed locations.
this doesn't include the huge server costs to run clearcase after you buy the license, which aren't necessary for distributed version control systems like git or mercurial. and big companies often spend more money buying other IBM tools to integrate for which free alternatives exist...
the amount of money spent on this purchase is fucking peanuts compared to what many large private sector companies spend on IBM software that lacks most of the features found in your favorite open-source alternative.
Spending other people money is so easy. But $22k apiece instead of the $485 solution quoted in the article - that's madness. It can't be just gross incompetence.
I guess someone involved in the decision got some kind of kickback, gift, or will have a nice cushy job offer waiting for them at Cisco when they get fired as they should be.
Remember this article next time some liberal ask to raise taxes and justifies this by saying how stimulus money is important, and why everyone should have the same opportunities (quoted in the article - as if a 22k router gave any kind of opportunity to people browsing the net in a public library in WV).
And cry when they get trashed at the end of their lifecycle without having ever been connected to optic fiber.
Democrats controlled two of West Virginia's three House seats, both Senate seats, the Governor's mansion, and the state legislature (which they have held since 1928).
One of West Virginia's two current Democrat senators, Joe Manchin, was governor at the time of the stimulus and appointed Mr. Gianato.
The executive branch of the federal government in 2009-2010 was controlled by the Democrats, along with both houses of congress. The federal stimulus was passed with no Republican support.
The entire chain of custody from the grant through request for bids through the eventual approval was controlled by Democrats. The only people involved in this fiasco without a (D) after their name are those people who will have their taxes raised in order to pay for these routers, because of course they were not paid for with existing revenue.
This is very much a failing of Democrat administrations at a federal, state and local levels.
He did actually say "Liberal" and not Democrat. There is a bit of a distinction there. Generally the Liberal party in the US isn't so Liberal and the Conservative party isn't so Conservative. I mean when you get down to it, in US politics...nobody's really much of anything. To peg it all down on a bunch of labels adds about nothing to meaningful discourse, because the labels don't mean anything.
I love West Virginia, and there is so much more important the state government could do with this kind of money if they're going to take it outside of "Homeland Security". As in, better health care or safety for the huge population of coal miners.
why do military closets cost 5000$ a piece? because it costs a sh#tload of money to sell the military a closet. i don't know where i read this but it is a great analogy.
i have been - in my time (2000? 2001? 2002? 2003? 2004? ...) - involved into selling the national austrian television their first video streaming solution (think: free (as in free beer) windows media streaming server) - it took man-years. man-years cost money.
"West Virginia Homeland Security chief Jimmy Gianato, who's leading the state broadband project, defended the $24 million router purchase last week, saying the devices "could meet many different needs and be used for multiple applications."
"Our main concerns were to not have something that would become obsolete in a couple of years," Gianato said. "Looking at how technology evolves, we wanted something that was scalable, expandable and viable, five to 10 years out. We wanted to make sure every place had the same opportunity across the state."
Wow. I've spent some time in West Virginia libraries, and yes the internet connection was slow, but it's not due to the router. And in most towns, libraries and schools are not going to be serving anyone outside their walls with some kind of WAN. Best case will be public wireless near the building.
And how is it that the "West Virginia Homeland Security chief" is "leading the state broadband project"? Seems like bureaucratic overreach, technical ignorance, and budget authority all wrapped in.
Why not spend $50 for a router for libraries and something less than $22k per school?
A router that can handle 5-30 clients at current internet speed standards within a library setting. Yes, bigger installations could benefit from a $20k router, but the vast majority of these routers are a complete waste of money in WV outside of Charleston and Morgantown.
Unless there is some big backbone being built that can service a mostly small-town and very mountainous state, spending $20k (each!) now so that your routing infrastructure within schools and libraries won't be too slow in a couple years is kinda crazy.
I run my office network off of a $50 router quite happily. But that's because I'm right there to take care of it. Earlier this year, for example, our network got a fever:
If I needed to cover 1,000 widely scattered locations for the next decade, there is no way I would try to do that with a bunch of consumer-grade stuff. Especially when trying to manage a bunch of existing T1s and a conversion to fiber.
These routers are pretty good choices for the schools. For the smaller sites it's hard to say without knowing their exact requirements for the future. For example this might be related to your question about the Homeland Security Chief being involved. Libraries are multipurpose buildings. In an emergency they could be used as an emergency response command center. Perhaps the library is used as a polling place and they have specific security requirements. The state may have decided it was in their best interest long term to equip all government buildings with a baseline level of equipment.
I think part of the reaction to this story is people coming to terms with how much real network equipment costs. It's definitely not cheap. You're paying for features, service, long-term support, upgradability, reliability, etc. Some of these routers will probably be in use 15 years from now. I've still got a Cisco 2500 series router chugging away. Next year will be it's 20th birthday. This stuff is built to last.
To be clear, from the article, the routers themselves cost $7,800. And, if I'm rolling out a state infrastructure, and wondering what I can put in place for the next 10 years to serve as a foundation, you could certainly do worse than the 3945 - it's a very flexible ISR, and, all-things being equal, it's probably not worth the hassle of putting 2921s in some locations and 3945s in others. Who knows how much bandwidth you'll want on these high-speed fiber connections 5+ years from now - the 3945 is rated for 350 megabits/second (with features), the 2921 tops out at 75 megabits.
Amortized over 10 years, I would have chosen the 3945 everywhere versus sticking 2921s in some places (the ISR that would have been an alternative) and 3945s in others. Single Security Policy. Single IOS update Policy. Zero doubt as to what features will run in a particular location.
I think what most people have difficulty with is that they are comparing this decision to roll out a state communications infrastructure with the fact that they can go connect a $60 linksys wrt54G in their house and serve a dozen people without breaking a sweat. And get wireless as well! The issues involved in scaling that across the state, while looking to the future, and managing all that gear is a different challenge though.
That seems like a pretty silly argument to me. The library with 4 computers would be just fine with the $60 router. How is it any harder to manage that gear compared to managing the completely unneeded equipment that the local employees have no clue how to run.
Not to mention spending $14k on upgrades for many places that don't need that particular upgrade. I don't care if it's a little simpler to buy all the same, it's absurd to spend millions and millions of taxpayer dollars on something that is completely unnecessary.
I'm not really sure where it came from, but the Slashdot summary states that a lot of these things were never even un-boxed. If true, that's an incredible waste and far-outweighs any convenience or stability benefits gained by using homogeneous hardware.
Interestingly enough - I can see both sides of the argument. When I was building the IT infrastructure for SSN, I ran the company's IT and it's dozen employees on a $75 linksys (and a half-dozen Poweredge Dell Server) for the first 5-6 months. Eventually, we grew to around 20 employees, picked up a data center colo-cabinet for our customer apps and needed a bit more robustness, so I purchased a $300 used Cisco 2621xm, a couple T1 WICs for $500, and ran the company for the next year on that 3 mbit pipe - the company got to around 50 employees before we moved. For the first two years of its life I spent a grand total of $875 on routing hardware and 50 employees were able to get their work done (including code pushes to our data center) over that 3 mbit pipe.
When I handed over network engineering, though, our last purchase order for networking equipment (with around 500 employees) - was north of $500K (including very hefty Cisco Discounts) (This isn't including the Data Center infrastructure - by then we had colo space in three data centers - just the corporate infrastructure for three buildings)
It's hard to explain without living through it - but the decisions that make sense when you are small, and you aren't paying your network engineers $150K/year (fully loaded), and you aren't trying to figure out how to handle pager duty, and warehouse and deal with RMAs, and support various types of hardware, and worry about rotating inventory, and dealing with upgrade cycles, and manage security, and patch levels, and remote administrations - not even considering future feature enhancements and performance (75 mbits sounds great today - but what about the future?) - when you get to scale, the CapEx (the capital cost of the hardware) starts to have fewer consequences on a technology investment (particularly over 10 years) than the other elements. Not to mention that there are also political issues (Layer 8) associated with differing levels of services/features for small population centers versus large population centers.
Yes - this does result in seemingly ludicrous situations like a 4 computer library being run on a ISR capable of running 350 mbits/second without breaking a sweat - but in 5-8 years from now, some technology administrator for the state will take over that infrastructure, and I can guarantee you that they won't be thinking "My God, we're over provisioned on our networking equipment" - but instead, "Thank goodness I have a few more years of runway before we have to replace all this gear."
I realize I haven't fully fleshed out the argument as to why it might make sense to put these hefty branch routers into smaller locales, but hopefully it doesn't sound silly.
Hey - at least they didn't roll out 6509s in all these libraries. Now that would be something I could rant against. :-)
This is a great thought - when talking about a growing business. Comparing it to a public library in a very small town (just over 6k) doesn't make any sense though. For the money, the library can buy a new router every year for next couple of centuries and keep up with current technology. Or we could have bought them a router and then given them $21,940 for new computers, books, facility upgrades etc.
Remember - these decisions weren't being made by the library, or for a 500 person company, they were part of a $126 million dollar infrastructure upgrade for over 1000 sites for the state of west virginia. The point I was trying to make was that the decision that makes sense for that library (individually - say the local librarian was given $25,000 and given freedom to spend it however they chose - they would probably do exactly as you suggested - drop $60 on a linksys and the rest on computers/books), no longer makes sense when you are rolling out $126 million worth of equipment. The most realistic alternative decision would have been to downgrade the library from a $7800 Cisco 3945 to a $2800 Cisco 2921 (an ISR with less capacity, but reasonably similar features and management). The Library would then have been on a 75 mbit/sec platform for the next 10 years. They wouldn't have received any of the savings to spend on books/computers, etc... They would have just gotten less capable networking equipment.
I could probably defend either decision (Going heterogenous 2921 (small) + 3945 (big) vs homogenous 3945 across the state) - but I know the one that would let me sleep easy for the next 10 years.
These problems with managing at "scale" btw, are one of the reasons why consumerization of IT is so popular. The reality is that individual decisions made at small scale, can sometimes be orders of magnitude more efficient than by the enterprise. Example - Our company doesn't have a policy regarding mobile device OS upgrades - which means they don't have to worry about standardizing on a particular mobile device, or test "authorized apps" (or even have authorized apps) against the OS upgrade, or manage the OS updates, etc... Basically, a few early adopters will do an OTA upgrade, watch it for a few days, and if their device doesn't brick or crash more frequently, and the forums seem to indicate it's a good version - they'll send out a note to an internal alias and everybody else will just upgrade their device. Absolutely zero corporate overhead required.
This is true of all sorts of "edge" infrastructure - Skype clients, IM clients, Laptops, backups (now that we have backblaze/crashplan), etc...
Deploying at scale loses all sorts of that efficiency, and should be reserved for "Core" things like your exchange server, filer, and networking. For everything else - try to empower your users - they'll probably be better at it than you will be doing it centrally.
The problem with this argument is that the blanket purchase only provides giant imagined future capacity for the smallest users. If they will actually need this quality of a router in 5-8 years, then the big users will definitely need even bigger ones. But apparently the state is OK with the biggest users not having that much breathing room. So why do the smaller ones need it?
The first thing that comes to my mind is that in 5-8 years, as the State of west Virginia starts to run out of headroom in the larger locations, they can selectively upgrade that 5% or so that require something larger, while leaving the other 95% on the previous iteration.
Also - one thing I've learned about networking sites (real world experience, two companies that went from a dozen to 500+) is that once you are able to satisfy one person with sufficient pipe - that same bandwidth is usually sufficient for the 20-50 people. The reality is that need for bandwidth is very bursty.
And in 5-8 years, there will be more powerful equipment available at lower prices for that 5% of infrastructure. And you still have 1100 5-8 year old routers at all your locations that might not be capable of running whatever physical layer link we're using 5-8 years from now.
Future proofing is one thing. This is like buying a 747 for a route that serves a dozen passengers a flight to prepare them for the future.
> The first thing that comes to my mind is that in 5-8 years ... they can selectively upgrade that 5% or so that require something larger, while leaving the other 95% on the previous iteration.
In other words, they'll be in the exact same position that they were evidently trying to avoid from the start.
This smug, platitudinous self-serving drivel is all that a network executive needs to know to misspend millions of dollars. You conveniently neglect to mention remote network management facilities that will handle configuration and monitoring of the routers that could have been purchased instead of a fully loaded 3945, including the $487 CISCO router that was mentioned in the article. How do you justify this preposterous expense? "What if the state decides on an unprecedented expansion of library facilities?" In this economic climate? "What if homeland security needs emergency facilities and this is the only site available?" You didn't read the article where two identical enterprise level routers were installed in the same little neighborhood. What about the efforts to engage ham radio operators--West Virginia is full of them. So much for the big picture. You live in a world where automating the administrative aspects of networking hasn't kept up with the network devices themselves. At least that's what you tell your superiors. It's as if remote network management facilities don't exist and you're still upgrading IOS on routers state-wide by hand with TFTP. Don't tell me: you support one-size-fits all cable management for the state too.
Yes exactly, homeland security will need emergency facilities for the next terrorist attack and they can conveniently use the libraries. It's a good thing they had an ex-homeland security person to manage these tough decisions and prepare the libraries for the coming internet armaggedon.
The thing is that these libraries are not going to grow to several hundred seats in the next 5-10 years, like the company you worked for. They will remain at their level of 5-20 seats. As such there is no need, nor will there ever be, for such strong infrastructure in a small communal library. They'd easily be able to make do with buying a 100$ router every second year.
If you are putting routers in a school that might serve a few thousand students and teachers, and a branch library that might be serve 5 patrons, you really can't afford to adopt a heterogeneous hardware strategy just to make admin easier.
It is surely more cost effective to use the cheaper router, and replace them when necessary than to buy the uber-model that will still be state of the art in 10 years.
Bucket system needs into three categories (large, medium, small) and find three routers that best manage those needs. Hire a full-time government worker to facilitate categorization of each installation location and manage the installation. Replace upgrade as needed.
Yes - that's actually a reasonable alternative. I'm hoping that someone did the math, and looked at Option #1: $7800 Cisco 3945, $2800 Cisco 2921, $900 Cisco 1921 vs Option #2 - $7800 Cisco 3945 across the board) and then compared it with the costs of having to prematurely replace those 2921s/1921s when they realized the 75 mbits/second wasn't what it was in 2012 (or some service modules with cool feature (Telepresence?) wouldn't work in the 1921)
Another thing to realize - they probably get one chance to upgrade their infrastructure like this. Sticking a 3945 in now means they can save money on upgrades for the next 10+ years.
It hardly seems like managing gear would be all that difficult at this kind of margin. Hire someone full-time at 100K annually who manages the systems and upgrades when necessary.
This is pure, unadulterated nonsense. There are plenty of ways to manage everything you mentioned with a tiered approach to the IT needs of the municipalities of the state without resorting to grossly over accommodating every location. Hell, they could've even used some of that money for regional IT management positions, putting people back to work.
Shame on you for justifying this nonsense in any way shape or form. This is textbook waste ala bureaucratic laziness.
I don't think it does. In the example put forth in the article, the library had only four computers! In the case of just this one library, they could have had: ipads, kindles, more computers, more books and other media.
I think it's kind of a no-brainer that this is a prime example of incompetence in government.
They would also need to hire someone to manage those devices which would cost maybe 2x the router year after year. It'd suck too (not going to get a competent person in that position).
My company provides free I.T. services, support, and sometimes equipment to non-profits and other community organizations. We would be happy to be the on-call techs for stuff like that.
I would be surprised if there wasn't somebody in their area that does the same.
I don't think he said that. For one, cheaper hardware doesn't imply increased management costs; for two, the more expensive hardware didn't come with a support contract included in the price (at least, it's not mentioned in the article); for three, his comment reads to me that they could have optionally put the money towards employing people, to put people back to work -- i.e., it was one example of a better way to spend the money. I don't see anything in his comment that implies, "cheaper hardware and some IT-smarts...".
>This is pure, unadulterated nonsense. There are plenty of ways to manage everything you mentioned with a tiered approach to the IT needs of the municipalities of the state without resorting to grossly over accommodating every location. Hell, they could've even used some of that money for regional IT management positions, putting people back to work. Shame on you for justifying this nonsense in any way shape or form. This is textbook waste ala bureaucratic laziness.
And this BS attitude is part of the reason why the US has one of the worst internet infrastructures in the western world. Let's just use what is adequate for today and save some pennies, sure cable internet (or 640K) should be enough for everybody.
Not to mention that you're suggesting micromanaging and needlessly complicating the municipality IT infrastructure for marginal gains (compared to IT staff wages). Not to mention ignoring the "economy of scale" and mass-ordering benefits.
>The benefits from economies of scale and mass-ordering still apply to much lower priced hardware.
If you break the uniformity of the order, to lower priced for some and higher priced for others (for which the lower priced won't do) then you break the economy of scale benefit.
Instead of buying 100 same machines, you then buy a 100 machines of A/B/C etc types. You still buy bulk (within every machine class), but not as "bulk" as before.
Sure, the savings from buying bulk the more expensive unit may not totally offset the savings of buying units with different price points, according to each deployment's specific needs. But uniformity has a lot more benefits, too.
>The IT staff wages will still have to be paid because hardware needs support regardless of the unit price.
Sure, and nobody argued that. What we said is that since the price of the hardware is a drop in the bucket (marginal) compared to the total cost (mainly the IT staff costs), then it doesn't matter. Paying 30,100,000 compared to 30,020,000 is not much of a difference (this one is 0.2%, but even if it was 5-10% it would not matter much).
Also there will be savings in the staff training (the have to only learn one unit), fixing costs, etc.
The most important reason though, is that we are very bad at anticipating future needs in internet infrastructure. Better get something that has a lot of headroom for future needs.
When smart and experienced people tell you that things work differently in large orgs and govs, they don't primarily mean that in a negative way (...that the system is broken), they mean that as a matter of fact (...that the system is completely different).
It's not broken, it's not "fixable", it's just the nature of it all.
Let me put it to you this way, if they purchased $60 dollar routers, there would be a scandal about the incompetence and lack of forward vision at the top of HN right now.
The nonsense about $60 linksys devices was addressed in the article. Did you bother to read it? CISCO quoted a $487 router. You expect the taxpayer to foot the bill because you're too lazy to keep track of which router you've installed at each location? I deal with bureaucrats like you all the time. This is the last place I would have expected to find them.
The bureaucrats I deal with would buy the various routers at best prices and then proceed to build a software system to track which router was installed where that would cost > top of the line routers and would be built by a company friendly to the person making the decision. Initial costs would be low to win the bid but many "required changes" would ramp the bill up. They would create software jobs and get optimal router pricing - if that's not efficiency I don't know what is !
I can assure you that any existing software would fail to meet the requirements (hint - primary requirement being related to/friends with/lobbying the decision maker).
You're right if nepotism is a requirement. But if we could suspend our cynicism long enough to consider available enterprise products (which is a tremendous strain admittedly for anyone who agrees with Paul Graham about "enterprise" computing) then as a comment below points out, CISCO sells Monitoring as a service and there are other vendors that provide this.
The other thing is that the bids had to be submitted very quickly or they risked not getting any funding at all. In that light, they can't really be blamed for not doing research into the most efficient allocation of money! (To be clear, I blame the stupid bid process that rewards rushed decisions and punishes anyone who stops to think.)
I'm surprised this article actually does a decent job covering both sides of the story. When you look at how flexible the ISRs are, and the likelihood these devices will be in service for probably the next 10 years or more, it's not not the worst investment they could make. They're actually well sized for the high schools and other larger sites. Obviously any 3900 series is going to be overkill for a little library but you do have to look at support costs, service contracts, future requirements, future upgrade costs, etc to get the full picture. This is not a 'bridge to nowhere' situation.
You can get an ASA 5505 firewall with all the features you'll ever need for a small library installation for about $1200 retail (hell of a lot less if you're buying, say, 50 of them), and any network engineer who can't handle ASA configuration in his sleep should be fired immediately for gross incompetence.
The "state" is completely clueless. You don't spec out technology for 10 years down the road anymore. The pace changes too quickly. You get the most affordable solution NOW, and when you need the big stuff you get something recent with even more longevity. The decision they made was stupid from a thousand angles. Retarded.
Am I the only person that is (not really of course!) tempted to put on official-looking clothing and confidently walk in with a laptop and Cisco console cable, nonchalantly telling the librarian their "Internet box" "needs a service upgrade".
And then quietly replace it (after dumping the config and replicating the configuration) with a $100 half-depth Supermicro server bought off Ebay, running OpenBSD?
And then walk out, like the Grinch after he artfully puts Cindy Lou Who to bed, with a nice new 3945 for my own use?
EDIT: forgot to mention, I need a CF card reader also to get the config off the Cisco flash.
If you then sold the router on the second-hand market and gave the money to the library to spend on materials that they actually needed, your plan could do some real good.
Considering you're talking about pulling the config off flash, which is where the OS and not the config is stored, I'm not too worried about what you plan to do.
From the article: "I'm not an expert on the technical side," [Gianato] said, "but these have all kinds of capabilities and applications."
Earlier: "Gianato acknowledged that he didn't heed Dunlap's advice or wait for an evaluation.
'The routers already had been bid out,' Gianato said. 'I think John was looking at our needs now, not looking at our needs into the future."
---
So not only does Gianato admit he doesn't know anything (which means he's not even suffering from the Dunning-Kruger effect), but he also admits that he's not sure what his IT advisor was thinking about, but, eh, it doesn't really matter because the people who are the experts obviously don't think about the long-term and I'm not going to bother asking them about it anyway, because as we all know, bids are binding contracts. Right.
Regarding planning for the future, this seems like someone who was used to buying machinery was put in charge of buying technology. Sure, in five years the library might need a gigabit of bandwidth, but by then the Linksys WRT may well handle it for $100. It's not like you're buying a work truck, which may well serve the government for 5, 10, or even 15 years (assuming low usage), or a lathe that might be used intermittently for 35 years.
There's a lot to be said for uniformity in deployment, but that's why you would define, say, three tiers of supported hardware instead of kitting out everyone with top-of-the-line.
Interestingly, the CEO of Cisco is from West Virginia and went to WVU - I believe he's one of the largest donors to WVU. (only know this because I'm also from WV, not suggesting any conspiracy here)
I have a buddy who sells routing equipment at this level. Doesn't have a degree in Fahrenheit, but just cleared a mil in his bank account, owns his 6 bedroom Connecticut home outright and keeps a beach house in Mystic. He's 38. I'm 36, a doc, and still paying off student loans and working for Uncle Sam.
Sounds like a buffoon was in charge of the operation. Gianato suggesting that the unused/unneeded T1 interface cards could still be used for "video conferencing, wireless Internet and "voice over Internet protocol." is just painful to read. I am not sure how they can be used for such things if you don't actually stick them in a router!
I would still like to see a breakdown of the costs and what services are provided with the routers before I call it a total waste though. If it was just the routers then it was a waste. If it includes other things like a long term warranty & onsite service, then maybe not so much.
141 comments
[ 4.4 ms ] story [ 229 ms ] thread* Yes, I realize this would bankrupt half the military contractors as well.
You are correct on the military contractors as well though. Lots of government waste and incompetence there as well.
somehow i doubt these sites have multiple uplinks, so they certainly don't need BGP routers. even if they're doing IPSec between some locations this can be handled by an old piece-of-junk $100 computer running BSD or nix, e.g. with isakmpd and pf.
stuff like this is why america's infrastructure is crumbling: dodgy contractors looking for an excuse to upcharge municipalities for irrelevant gear and services.
Anything more than that for a moderately large library or most schools is an outlandish, shameful waste of money and equipment.
I imagine the government worker that signed off on this purchase worrying a lot more about a dollar increase in the local lunch special than millions spent on equipment.
Out government should be small. It should perform a few basic functions like making sure we are secure, our trading interests are promoted and our banking and financial markets are healthy. It should educate our children to the best standards in the world. Finally it should make sure we are energy independent and our food and water is uncontaminated. That's pretty much it.
It's not a democrat/republican or big/small government thing. It's about corruption of the system.
Out of your comment, I agree most with that our government needs to help educate our children. Without an educated population, things only get worse.
While the republicans are certainly guilty of lots of spending and many things, democrats are clearly the party that is guilty of more spending and even bigger government.
There are a few components to this particular screw-up. 1 is big government spending (with little oversight) and 2 is one idiot in WV with a big budget who failed miserably at spending it.
"1 is big government spending (with little oversight)"
The obvious problem here isn't "big government" spending. It's spending without oversight. This problem occurs all the time everywhere, in sectors both public and private. To claim that private businesses are immune to this by nature of their ability to fail and thus inherently a better mechanism than government is an absolute farce.
Take ownership of your government. To view the elimination of government as a solution to bad governance is both a lazy approach to problem solving and toxic to proper governing.
Nobody said anything about elimination of government, just less of it.
Getting the money / corporate influence out of government would be the place to start.
Let me guess... military spending doesn't count as "more spending". The $800 billion allocated for 10 years of stimulus is a paltry sum compared to the $700+ billion we intended to spend every year on military, and we all know that money is spent very efficiently.
Meanwhile, federal spending has increased every year under both parties for the last two decades, so you don't really have a leg to stand on with regard to your "democrats are the big government spenders" FUD.
gps and the internet are two of the most notable and visible examples with astronomical infrastructure costs. no private company would undertake a risk like this yet both have substantially improved society. the microwave and digital cameras are two others that a private company may have eventually created.
this isn't an argument for military spending, just the fact that without a government body that focuses entirely on applied research and development the military is the next best thing.
Just because there is waste in defense, doesn't mean there isn't more elsewhere. Defense is one of the few things are government is actually tasked with. All the other shit we waste money on isn't part of what they are supposed to be doing. Roads and armies, that's all I expect. I don't need the government to feed me or wipe my ass.
I'm sorry to burst your bubble, but our current presidents and congress members do what will appease the general population in the very near term, no matter what they are supposed to do according to their party lines. They are all equally guilty of spending.
You know, the TSA in the airports, which has a larger budget than the FBI? Guess who created and nurtured that entity? Your small government friends. And, who spent so much money on the wars in Iraq and Afghanistan that it is bankrupting the country. We can't even get involved in another war right now because we simply can't afford it. It is questionable if we will ever be able to lead a war like that again, due to budget constraints.
Then, we have the wall street bailouts. Which, by the way, aren't republicans supposed to be the only friends of wall street? Not if letting the banks fail mean that you are going to have to line up for the bread line tomorrow morning.
The big government people aren't exactly cherubs either. How many RIAA lawyers have been appointed to the Justice Department by the current administration? I think it's 5, so far. Enjoy additional Internet censorship.
Sorry to be such a downer, but things are just getting worse and the current two party system is a complete facade. I'm envious of countries like Germany that are seeing a surge in pirate party support. Unfortunately, it's going to take decades to undo this mess and, frankly, a large majority of baby boomers are going to need to expire naturally before the voting population comes back into balance.
I disagree with the sidebar that it's mostly Democrats trying to take our money. Both parties do it, just with different priorities and in different ways, with all the accepted terms used to discuss it.
As for your core point about the role of government... just look at the top 10 economies of the world. Most are not libertarian paradises and they're doing OK. It seems to me both models could probably work if executed properly. I doubt one model is inherently right and one is inherently wrong. Most likely some hybrid is the best solution for everyone.
I think we agree on the idea that government should be small, or at least as small as possible. And we agree on the roles it should play. But I don't buy into the argument that government is inherently bad at allocating resources efficiently and private businesses are inherently good. I've seen way way too many examples that contradict the latter. I think it ultimately comes to individual situations and the choices of individuals, what information is available to them, their goals, where their financial interests pull them.
When you reach a certain size, and you have many competing groups, all with their agenda with the same company, it becomes difficult to allocate resources correctly, especially in a domain that is not considered "core of business" (IT being a common case, sadly).
licenses for clearcase cost enormous amounts. yet clearcase is missing many of the features found in open-source (and free) distributed version control systems. meanwhile, there is no free alternative to the routers, which may actually be necessary at some of the installed locations.
this doesn't include the huge server costs to run clearcase after you buy the license, which aren't necessary for distributed version control systems like git or mercurial. and big companies often spend more money buying other IBM tools to integrate for which free alternatives exist...
the amount of money spent on this purchase is fucking peanuts compared to what many large private sector companies spend on IBM software that lacks most of the features found in your favorite open-source alternative.
I guess someone involved in the decision got some kind of kickback, gift, or will have a nice cushy job offer waiting for them at Cisco when they get fired as they should be.
Remember this article next time some liberal ask to raise taxes and justifies this by saying how stimulus money is important, and why everyone should have the same opportunities (quoted in the article - as if a 22k router gave any kind of opportunity to people browsing the net in a public library in WV).
And cry when they get trashed at the end of their lifecycle without having ever been connected to optic fiber.
Unless I misread, the article is talking about West Virginia. I would hardly call WV a bastion of liberal thinking.
Never attribute to malice (or political leaning) that which is adequately explained by stupidity (or incompetence)
One of West Virginia's two current Democrat senators, Joe Manchin, was governor at the time of the stimulus and appointed Mr. Gianato.
The executive branch of the federal government in 2009-2010 was controlled by the Democrats, along with both houses of congress. The federal stimulus was passed with no Republican support.
http://www.nytimes.com/2009/01/29/us/politics/29obama.html
The entire chain of custody from the grant through request for bids through the eventual approval was controlled by Democrats. The only people involved in this fiasco without a (D) after their name are those people who will have their taxes raised in order to pay for these routers, because of course they were not paid for with existing revenue.
This is very much a failing of Democrat administrations at a federal, state and local levels.
Governments that waste money just raise taxes, borrow more money, or print more money. Corrections when they do come are often catastrophic in nature.
I love West Virginia, and there is so much more important the state government could do with this kind of money if they're going to take it outside of "Homeland Security". As in, better health care or safety for the huge population of coal miners.
i have been - in my time (2000? 2001? 2002? 2003? 2004? ...) - involved into selling the national austrian television their first video streaming solution (think: free (as in free beer) windows media streaming server) - it took man-years. man-years cost money.
I could further elaborate on the reasons why this matters to me, but I'm just relieved that it works OK now.
"Our main concerns were to not have something that would become obsolete in a couple of years," Gianato said. "Looking at how technology evolves, we wanted something that was scalable, expandable and viable, five to 10 years out. We wanted to make sure every place had the same opportunity across the state."
Wow. I've spent some time in West Virginia libraries, and yes the internet connection was slow, but it's not due to the router. And in most towns, libraries and schools are not going to be serving anyone outside their walls with some kind of WAN. Best case will be public wireless near the building.
And how is it that the "West Virginia Homeland Security chief" is "leading the state broadband project"? Seems like bureaucratic overreach, technical ignorance, and budget authority all wrapped in.
Why not spend $50 for a router for libraries and something less than $22k per school?
Unless there is some big backbone being built that can service a mostly small-town and very mountainous state, spending $20k (each!) now so that your routing infrastructure within schools and libraries won't be too slow in a couple years is kinda crazy.
http://nerdfeed.needfeed.com/blog/2012/02/our-network-has-a-...
If I needed to cover 1,000 widely scattered locations for the next decade, there is no way I would try to do that with a bunch of consumer-grade stuff. Especially when trying to manage a bunch of existing T1s and a conversion to fiber.
I think part of the reaction to this story is people coming to terms with how much real network equipment costs. It's definitely not cheap. You're paying for features, service, long-term support, upgradability, reliability, etc. Some of these routers will probably be in use 15 years from now. I've still got a Cisco 2500 series router chugging away. Next year will be it's 20th birthday. This stuff is built to last.
Amortized over 10 years, I would have chosen the 3945 everywhere versus sticking 2921s in some places (the ISR that would have been an alternative) and 3945s in others. Single Security Policy. Single IOS update Policy. Zero doubt as to what features will run in a particular location.
I think what most people have difficulty with is that they are comparing this decision to roll out a state communications infrastructure with the fact that they can go connect a $60 linksys wrt54G in their house and serve a dozen people without breaking a sweat. And get wireless as well! The issues involved in scaling that across the state, while looking to the future, and managing all that gear is a different challenge though.
I don't see any huge scandal here.
Not to mention spending $14k on upgrades for many places that don't need that particular upgrade. I don't care if it's a little simpler to buy all the same, it's absurd to spend millions and millions of taxpayer dollars on something that is completely unnecessary.
http://yro.slashdot.org/story/12/05/11/1937235/west-virginia...
When I handed over network engineering, though, our last purchase order for networking equipment (with around 500 employees) - was north of $500K (including very hefty Cisco Discounts) (This isn't including the Data Center infrastructure - by then we had colo space in three data centers - just the corporate infrastructure for three buildings)
It's hard to explain without living through it - but the decisions that make sense when you are small, and you aren't paying your network engineers $150K/year (fully loaded), and you aren't trying to figure out how to handle pager duty, and warehouse and deal with RMAs, and support various types of hardware, and worry about rotating inventory, and dealing with upgrade cycles, and manage security, and patch levels, and remote administrations - not even considering future feature enhancements and performance (75 mbits sounds great today - but what about the future?) - when you get to scale, the CapEx (the capital cost of the hardware) starts to have fewer consequences on a technology investment (particularly over 10 years) than the other elements. Not to mention that there are also political issues (Layer 8) associated with differing levels of services/features for small population centers versus large population centers.
Yes - this does result in seemingly ludicrous situations like a 4 computer library being run on a ISR capable of running 350 mbits/second without breaking a sweat - but in 5-8 years from now, some technology administrator for the state will take over that infrastructure, and I can guarantee you that they won't be thinking "My God, we're over provisioned on our networking equipment" - but instead, "Thank goodness I have a few more years of runway before we have to replace all this gear."
I realize I haven't fully fleshed out the argument as to why it might make sense to put these hefty branch routers into smaller locales, but hopefully it doesn't sound silly.
Hey - at least they didn't roll out 6509s in all these libraries. Now that would be something I could rant against. :-)
I could probably defend either decision (Going heterogenous 2921 (small) + 3945 (big) vs homogenous 3945 across the state) - but I know the one that would let me sleep easy for the next 10 years.
This is true of all sorts of "edge" infrastructure - Skype clients, IM clients, Laptops, backups (now that we have backblaze/crashplan), etc...
Deploying at scale loses all sorts of that efficiency, and should be reserved for "Core" things like your exchange server, filer, and networking. For everything else - try to empower your users - they'll probably be better at it than you will be doing it centrally.
Also - one thing I've learned about networking sites (real world experience, two companies that went from a dozen to 500+) is that once you are able to satisfy one person with sufficient pipe - that same bandwidth is usually sufficient for the 20-50 people. The reality is that need for bandwidth is very bursty.
Future proofing is one thing. This is like buying a 747 for a route that serves a dozen passengers a flight to prepare them for the future.
In other words, they'll be in the exact same position that they were evidently trying to avoid from the start.
It is surely more cost effective to use the cheaper router, and replace them when necessary than to buy the uber-model that will still be state of the art in 10 years.
Another thing to realize - they probably get one chance to upgrade their infrastructure like this. Sticking a 3945 in now means they can save money on upgrades for the next 10+ years.
Shame on you for justifying this nonsense in any way shape or form. This is textbook waste ala bureaucratic laziness.
I think it's kind of a no-brainer that this is a prime example of incompetence in government.
I would be surprised if there wasn't somebody in their area that does the same.
I don't think he said that. For one, cheaper hardware doesn't imply increased management costs; for two, the more expensive hardware didn't come with a support contract included in the price (at least, it's not mentioned in the article); for three, his comment reads to me that they could have optionally put the money towards employing people, to put people back to work -- i.e., it was one example of a better way to spend the money. I don't see anything in his comment that implies, "cheaper hardware and some IT-smarts...".
Hell, a college kid could do the job.
And this BS attitude is part of the reason why the US has one of the worst internet infrastructures in the western world. Let's just use what is adequate for today and save some pennies, sure cable internet (or 640K) should be enough for everybody.
Not to mention that you're suggesting micromanaging and needlessly complicating the municipality IT infrastructure for marginal gains (compared to IT staff wages). Not to mention ignoring the "economy of scale" and mass-ordering benefits.
The IT staff wages will still have to be paid because hardware needs support regardless of the unit price.
If you break the uniformity of the order, to lower priced for some and higher priced for others (for which the lower priced won't do) then you break the economy of scale benefit.
Instead of buying 100 same machines, you then buy a 100 machines of A/B/C etc types. You still buy bulk (within every machine class), but not as "bulk" as before.
Sure, the savings from buying bulk the more expensive unit may not totally offset the savings of buying units with different price points, according to each deployment's specific needs. But uniformity has a lot more benefits, too.
>The IT staff wages will still have to be paid because hardware needs support regardless of the unit price.
Sure, and nobody argued that. What we said is that since the price of the hardware is a drop in the bucket (marginal) compared to the total cost (mainly the IT staff costs), then it doesn't matter. Paying 30,100,000 compared to 30,020,000 is not much of a difference (this one is 0.2%, but even if it was 5-10% it would not matter much).
Also there will be savings in the staff training (the have to only learn one unit), fixing costs, etc.
The most important reason though, is that we are very bad at anticipating future needs in internet infrastructure. Better get something that has a lot of headroom for future needs.
It's not broken, it's not "fixable", it's just the nature of it all.
Let me put it to you this way, if they purchased $60 dollar routers, there would be a scandal about the incompetence and lack of forward vision at the top of HN right now.
Fix the process.
There is no excuse for this insanity.
No you fool! These are designed to handle large loads, not protection from obsolescence!
And then quietly replace it (after dumping the config and replicating the configuration) with a $100 half-depth Supermicro server bought off Ebay, running OpenBSD?
And then walk out, like the Grinch after he artfully puts Cindy Lou Who to bed, with a nice new 3945 for my own use?
EDIT: forgot to mention, I need a CF card reader also to get the config off the Cisco flash.
Earlier: "Gianato acknowledged that he didn't heed Dunlap's advice or wait for an evaluation.
'The routers already had been bid out,' Gianato said. 'I think John was looking at our needs now, not looking at our needs into the future."
---
So not only does Gianato admit he doesn't know anything (which means he's not even suffering from the Dunning-Kruger effect), but he also admits that he's not sure what his IT advisor was thinking about, but, eh, it doesn't really matter because the people who are the experts obviously don't think about the long-term and I'm not going to bother asking them about it anyway, because as we all know, bids are binding contracts. Right.
A little known fact in the tech community is that a $7,800 router provides exactly 16.016 times more opportunity to a student than a $487 router.
Anyone familiar with telecom and living in PA, WV, other states that have Verizon as the ILEC, knows this.
There's a lot to be said for uniformity in deployment, but that's why you would define, say, three tiers of supported hardware instead of kitting out everyone with top-of-the-line.
I would still like to see a breakdown of the costs and what services are provided with the routers before I call it a total waste though. If it was just the routers then it was a waste. If it includes other things like a long term warranty & onsite service, then maybe not so much.