I just got through speaking in DC about Agile, startups, and functional programming, mostly to an audience that is intimately familiar with government work. It's really bad when you try to be diplomatic about how bad things are fucked up and get open laughter from the crowd. There's no BSing people who live this everyday.
Talking with folks afterwards, I heard about systems that I could have rewritten in the late 90s in a few months that are still in the process of being rewritten, 15 years later. After literally hundreds of people have worked on them costing the taxpayer millions of dollars. And these are not unique or oddball edge cases. Stuff like that is standard operating procedure. I can't impress on the HN audience enough how terrible federal IT is overall. (Aside from some really great outliers). There's a vast and increasingly large gap between political rhetoric and the reality on the ground. This has been true for at least the last 20 years.
I take issue with some of the details the supporting argument makes here, but that's mostly a quibble. The author is definitely headed in the right direction.
I'm a veteran of several federal IT programs, and a real veteran. I have seen some really good federal teams, but never a good federal team of teams. It all falls apart once you get past a few people and start throwing some kind of structure and management in place. I understand that Agile is becoming required for some federal work, but I'll bet you a thousand bucks this kind of "Agile" isn't going to be the kind that the rest of us use. Think big tools, big management, lots of stupid measurements, and no accountability. All with the word "Agile" stamped on the front of it.
...The bottom line is that federal IT programs' success is measured by things that have nothing to do with how successful they are or by the metrics most of the world uses. While the business world (and Web companies in particular) now monitor user experience and productivity as a metric for IT success, the government keeps throwing out numbers that mask the truth: the only people who would use their systems are the ones that are forced to...
"Agile" when federalized will probably turn out like "Lean". Lean in manufacturing and similar processes, when done right, is a good thing. However, the government version of it is almost a complete perversion of the intent. Instead of bottom up, employee-driven changes, it's top down based strictly on metrics and the changes typically effected so that whoever's currently in charge can say, "I made <process> 10% more efficient by reducing the number of wasted <things>." Employees resist the changed processes/taskings because they fear a loss of employment, and are kept out of the loop when the changes are being devised. Federalized Agile would likely be similar.
We saw this same thing with source control. Who doesn't like source control? But the government, in an effort to make sure there was absolutely no risk, mistakes, or accountability associated with software development, started adding more and more mandatory forms that had to be completed each time a developer checked something back in.
This created such a mess for developers that it wasn't unusual for one check in to take hours to accomplish. You'd read a book while the system tried to accept your code changes.
Agile is all about failing fast and learning quickly. Most contractual work for the government does not include the possibility of failure. At all. In fact, the way the contracts are written is such that the contractor takes on all responsibility for everything, even things there's no way they could control. In response to this idiotic situation, contractors start bidding at crazy low rates on these things, say 40 bucks an hour, then actually make their money on changes. So it's always in the contractors interest to capture and execute changes, while it's the government is trying to get the world delivered to them on a platter for five bucks. People use the term "perverse incentives", but this is like that taken to the next level. It's a multi-layered thing. It's total systemic failure. We'd call it systemic collapse, but the money keeps flowing in anyway, so that term isn't applicable, although it certainly feels like it.
This kind of distrust doesn't make for anything but a living hell for everybody, government employees and contractors alike. So, with tons of rules and structures in place, the way things really work is through relationships. People who are easy to get along with and helpful get more government work. Both sides of the fence can't control the process, and nobody rocks the boat and everybody gets a piece of the pie. You end up from the government side with some sort of informal and unspoken version of I don't care if you make a lot of money, just make sure me and my friends look good Also it'd be nice to pick up a little lucrative consulting work once I've put in my 20 and can start double-dipping.
Your average developer has absolutely no idea about this. If they've worked in IT in federal work, they probably have horror stories, but the more you see the worse it gets. Oddly enough, in the best cases you end up with people who really like working together, are motivated, but spend most of their time fighting a Sisyphean fight against the idiocy of the system.
> But the government, in an effort to make sure there was absolutely no risk, mistakes, or accountability associated with software development, started adding more and more mandatory forms that had to be completed each time a developer checked something back in.
That's because "the government" is operating under different pressures than corporations. A corporation, big or small, screws up, only a handful of stakeholders are affected and raise a stink. If the government screws up, you're looking at headlines in the Washington Post.
A nit: This isn't plausible deniability, but rather CYA (cover your ass).
Plausible Deniability: Delegation of tasks and lack of records of communications letting the actual responsible party deny knowledge of the "insubordinate" or "rogue" underlings.
CYA: A process to change the process to define the process for tracking changes to the process, with 20 signatures or more required at each step of the process, digital forms filled out in triplicate. Or, making proper recordings of communications with underlings so when they screw up you can point out that they didn't act according to your direction, and that they failed to report issues back up the chain.
Yet another reason why nationalization of government functions is a bad idea. Much easier to move to a new state than to a new country and there are very few things beyond national defense that can't be handled by a state or local government (public works, education, health insurance/care, first responders, disaster response, etc.).
That isn't to say that state governments can't screw things up also but at least the damage is contained and state and local governments generally respond better to their constituents complaints/needs than the federal government.
> I understand that Agile is becoming required for some federal work, but I'll bet you a thousand bucks this kind of "Agile" isn't going to be the kind that the rest of us use.
Agile can't be mandated; you can mandate methodologies (e.g., Scrum), but by the very act of mandating a methodology you are not agile. Agile is a culture, and it is a culture of flexibility in methodology with focus on the goal.
And its a culture that is very much contrary to the entrenched culture of most large bureaucracies, government or private (and, in government, often in tension with binding mandates where the authority to make changes to the mandate is very distant from the locus at which one would normally expect efforts to adopt development approaches.)
Isn't this how any government project works? IT, construction, etc. Sees like the government is terrible at building anything, especially when contractors are used to do the job.
Probably why the government is so big and expense. I can only imagine what that $600+ billion gets us for the military. -_-
> Sees like the government is terrible at building anything, especially when contractors are used to do the job.
Strange to contrast that with a tremendously active and efficient era of public works from the 1890s through the 1950s. Just thinking of New York City, I suspect that the total cost of all the bridges, tunnels and highways (adjusted for inflation) might add up to a lot less than the 800 billion USD that disappeared into our recent stimulus bill. What changed?
edit: Apparently the Brooklyn Bridge cost ~15 million USD to originally construct, which now would be somewhere between 300 million to 400 million USD (http://historical.whatitcosts.com/facts-brooklyn-bridge.htm). Even though these are rough figures and inflation over large time horizons isn't a perfect measure, I'm still astounded. The government's totally broken healthcare website cost more to build than the Brooklyn Bridge.
The relationship between government and business, the much more short term approach to making money for businesses and individuals, a loss of the concept of "public good" or "common good".
I'm sorry, I don't really see how any of those explain the government being granted money (in staggering quantities), then spending it, then having nothing. Are you implying that because businesses have a short-term approach to making money that therefore, the government is forced to give money to unproductive contractors? Are you saying that because "the people" don't have a concept of "public good" that when the government bids something out they can't help but use a bidding approach that virtually guarantees failure?
Vague platitudes about society at large aren't crashing government projects. Society at large aren't really involved in them, after all. Something specific between the interaction of the government and the government-chosen contractors is probably the problem.
In a way, you're probably proving far more than you mean, accidentally; if the problem really is "society at large", then there is no solution and these large projects can't work, and therefore the most effective thing the government could possibly do is stop trying and wasting money. (A project with a prerequisite of "change the larger culture's attitudes" is simply a guaranteed failure, after all.) I suspect that's not the point you're going for. You probably really want to be arguing for a much more local reason.
Sorry, it wasn't really a thorough answer or intended as one.
Bad oversight: whoever's managing the project, doesn't understand the project. Not just, "I don't know how to read <some language>" lack of understanding, but "I don't know the difference between a spec and code". They don't specify terms that keep contractors in line which allow projects to run years and millions or billions over budget. Most of these contract managers aren't engineers, they may have engineers on staff, but it's not uncommon for those engineers to be employees of the same contractor that receives the benefits of the contract. The system is both incompetent and corrupted.
The companies receiving contracts that habitually run late and over budget are also some of the largest. Small contractors don't make it when this happens, but NG, Lockheed Martin, Boeing, they can survive the political fallout thanks to their clout.
If the companies have no oversight, and no accountability internally because they have no real concern for the other stakeholders (ie, the people paying them, the citizens/taxpayers), and the desire to make high quarterly profits, they'll rig the game by rent seeking, rather than completing a project effeciently and effectively and moving on to the next one. See NG and the maintenance of the JointSTARS aircraft as one such example. A rushed, but militarily valuable, project that had no maintenance plan in place, ends up contracted out to a company that destroyed one of the (few) planes and received no real punishment for it.
Thanks to the tenure-like nature of civil service, a large chunk of government employees are retired in place, new employees have no place to start (few positions opening), and those that get in have a terrible example in place and become retired in place after a handful of years themselves. They may start off dedicated to the job, and many do remain dedicated, but they are also frustrated by the processes involved and either quit or ride it out to retirement.
And regarding the platitudes, if companies and citizens (and consequently many of the elected and appointed officials) see profit as motive, rather than the future of the country or the society, then they make decisions based on that altered morality. Decisions based on short-term profit may not be harmful to long-term objectives, but they often are or are neutral to such objectives. A modern works project on the scale of the construction of the interstate system would likely never see the light of day, and if it did it would be mismanaged so as to never achieve its objectives.
Normally I type these up in emacs and copy them over, this is not a well formatted message, sorry about that.
And, because of the political clout that these large companies have and the revolving door between senior government and the large companies, government engineers and project managers who are competent and capable are routinely overruled when they try to raise red flags.
So very true. Fortunately, they occasionally get called on it and people realize there were kickbacks or back-scratching going on before they moved from one employer to the other and someone goes to jail. Unfortunately the contracts rarely get changed or cancelled as a result, it's usually just the employee that gets (meaningfully) penalized.
The usual form of quid-pro-quo between government agents and private contractors is the implied offer of a cushy "consulting" job with the company once they retire from public service. This happens with regularity and it's not clear that anyone ever gets penalized, except the taxpayers being fleeced.
> I'm sorry, I don't really see how any of those explain the government being granted money (in staggering quantities), then spending it, then having nothing.
Would it help to know that the same government (altho Federal Reserve is private but heavily regulated) print the money for their own use?
Imagine you have a small house with a garage inside and printing money machine in it. Now you want to expand the house and you know you can pay for the materials labor, etc, thanks to your printing machine. Wouldn't you do it?
The answer is those who care about US, about their outlook on the world would be very careful with using this machine. Those who would "own the house" for just a little bit would abuse it. We are at this stage where government human resources are so overbloated, ,you need so many signatures to buy a roll of toilet paper that eventually at the end of the day nobody is responsible for anything.
That can least if you own the world and don't have to answer to anyone. Un/fortunately this is not US case. If we default, we lose credibility with other major player such as China. Imagine if China stops producing for US (would be insane embargo if you think about out), can you imagine what would happen with us economy? all of sudden the soap dispencer you buy at wallmart cost $45 (made in USA) instead of 4.99 (made in China). Unfortunately, like I said there is no credibility even Chief in Charge Barrack Husain Obama is not aware (in most part) what is really going on under his watch.
I think that often, it's not just the interaction between the government and its contractor, but the activities within the contractor as well, though the interaction between government and contractor can certainly exacerbate any existing issues within the contractor.
Ironically, a large amount of the waste comes from the effort to avoid waste.
Weeks of development time have been spent dealing with how to keep our memory load under 1Gb, since that's how much RAM we have. Granted, 4GB of RAM would have been more than enough, but all that developer time saved the tax payer $20.
On the reverse side, I've seen half a million spent on hardware for a project where the sys-admin was making $35k.
If the hardware meant they could get rid of the sysadmin, it was probably a good deal. The loaded labour cost (salary plus facilities, insurance, redundancy payments, etc) is probably upwards of 200k per year. That means the hardware is cheaper.
I'd also add changes (improvements, in my opinion) in labor and safety laws. I suppose that falls under "[changes in] the relationship between government and business" but I think it's worth calling out specifically.
Lots of things changed, especially unions and labor costs. The guy who digs a subway or water tunnel in NYC today probably makes $250k, retires at 55, and makes $150k for the rest of his life.
The guys who built the Brooklyn bridge were day laborers who made nothing and died young.
I highly doubt construction works make 250k a year at least if you account for down time. RegardlessLabor costs don't explain massive cost over runs. If the amount of labor were accurately predicted, it would just inflate the cost, not over run those estimates.
The young guys probably make 125-160k, the old guys get overtime preference. IMO they deserve it -- it's a really dangerous job.
In any large scale project, any issue cascades and costs a shit-ton of money. For government contracts in many states, you have stuff like "prevailing wage" (where the gov pays too much for labor), "project labor agreements" (where the gov agrees to employ X number of people), minority and women owned business quotas (a big pit of corruption).
Also remember that governments are run by politicians, and politicians embrace magical thinking. Painting a rosy picture to get something approved and blowing huge money with no tangible result is a good way to raise capital in govt. (whoops, $X didn't do the trick, but to avoid wasting it, give me $X+Y)
France's rights of way were in no small part established by Napoleon simply building roads where he wanted them to go.
The USA, like many countries, has rules about simply seizing land that are of the form "well, OK, if it's for a good cause, but you better pay for it".
The inefficiency is primarily due to the scale and customization of the projects, not that it's government per se. In any project of sufficient scale, 1) you very likely use contractors at some level 2) you will have to invest into some sort of oversight infrastructure 3) with that much money invested, you want a strongly guaranteed outcome, so you'll err on the side of reliability over cost efficiency (or pay for the cost of failure and rework in subsets of the project).
I agree though that the mix of in-house expertise vs outside contractor effort does play a big part in the outcome. Look at Boeing's 777 vs 787 aircraft development for different sides of cost to use internal resources, vs leaning more heavily on external resources. The 787 with it's heavy use of subcontractors has had a much more difficult introduction.
The government in general has been upping the contractor content in a lot of projects with the belief that more "competitive private contractor" content automatically means better efficiency. But, in truth, it's no silver bullet, often increasing costs if it's blindly applied - especially if you fall below some threshold of in-house expert judgement.
I disagree that "government projects" are uniformly terribly executed between industries.
I can't vouch for the American case, but New Zealand has a similar history of disastrous government-run IT projects - from a terrible teacher payroll system[1] to a police information system[2]. The IT community hear is desperately trying to stop the same mistake with a massive tax system overhaul[3].
Nevertheless, civil infrastructure projects appear to be very well executed. Recent major roading engineering projects like the Victoria motorway tunnel in Auckland were completed on time and under budget, for example. There has been an active discussion in this country about why the IT industry, in particular as manifested as major systems projects executed by foreign software companies, are such consistent clusterfucks.
It's a big discussion and I won't go into many of the points raised here, but the issues is, IMHO, very much a "software industry" one, and not a "government" one.
Most first-world governments have a century or more of experience in proper administration and project management of construction contracts. Designing, estimating, awarding and administering nine or ten figure construction projects is extremely complex but by and large, we know how to do it properly.
Construction management is a serious discipline that is studied by civil engineers. It helps that construction methods don't change very often. Software technologies can become obsolete in a matter of months.
I wouldn't start calling IT procurement a failure compared to civil infrastructure until software engineering has matured to the point that civil engineering has.
I live in Perth, Australia. Almost every civil project there is blowing out like a balloon. Opera houses, hospitals, fancy developments. All going over budget and schedule.
I was in Toronto a few days ago. What's on the cover of the newspaper? A civil project. A subway station running way over cost and time.
Industrial and civil megaprojects routinely go way off course. Bad project management is universal, good project management is hard, any project of any serious size will require owner-commissioner organisations to not be useless idiots.
Heh, you could probably blame many of Perth's failures on the role of the mining industry there. Engineers have to be flown in from as far away as New Zealand to make up for the shortage, a flush city administration happy to double-down on pet projects, and so on.
But, obviously these are all anecdotal, I just know from my time at a large civil engineering company that the failure rate appeared to be very low. Part of that is excellent risk management, but also long-term client relationships, clear and long established success criteria and a distinct lack of political meddling (once the project is underway, at least).
Large projects fail or run over budget because of organizational failures: poor goals, lack of accountability, internal gridlock, etc. Nothing about that is specific to government – and if we wanted to change that in the United States we could reach the level of, say, Scandinavia if we were willing to make significant cultural changes.
That's just small potatoes. The really big stuff is almost invisible. That's because the liability is too large, but every now and then you can see how things are going. Check out the IRS modernization efforts. (The quote and link are probably 5-10 years old. Haven't heard much about it one way or the other lately)
...The IRS has failed twice to modernize its system, in the 1970’s when taxpayer privacy issues surfaced and in the mid-1990’s when the IRS spent ten years and $2 billion on the project and had little to show for it. The current project launched in 1999 with a budget of $8 billion. The project is late and over-budget; problems have been well publicized after the release of the IRS Oversight Board investigation...
AFAIK, the IRS dropped $10 Billion in IT money in just over a decade with little to nothing to show for it.
I used to joke around and say heck, they could have paid me 2 Billion and I would have also provided them nothing, but look at the money they would have saved!
The general rule is: the larger, older, and more politicized the agency is, the more awful the technology development.
So NSA, which is fairly young, stays out of the political eye (until recently), and small? They do some really cool stuff. But the older, more controversial agencies, covering things like immigration, defense, taxes, health, and so on? Not good.
This is an excellent analysis. Particularly the point about setting the go-live date long before the scope of the project was understood, not to mention the fact that there's no way they could have had an open bidding process AND meet the deadline. In some ways it's a miracle this went live on-time, flaws and all.
However, I suspect the true extent of the flaws with Healthcare.gov and the exchanges is not understood yet.
If anything good comes out of this, I hope it's that the government finally starts paying attention to the woeful state of government IT projects.
A new version of Windows Server doesn't address the problem of loading 64 JE files and 11 CSS files on every page. There is some programming ineptitude at play as well, though I realize that the article was focusing specifically on the new ObamaCare site.
That's also a minor side issue: it'll cost you a sub-1-second page load time but it doesn't cause the kinds of problems people are talking about where the backend components are overloaded to the point of failure.
The size of healthcare.gov is actually well under the average:
If you look carefully, though, you'll see a) that they aren't setting Cache-Control headers and are setting ETags (both performance killers) and b) there's this weird staggered response behaviour which indicates something is seriously wrong. It seems to handle much better in Chrome - 3s vs 70! - so presumably there's some interaction with the browser going on as well:
At an individual level: How many programmers who are good enough to get a job that pays well in the private sector want to take a job (either in the government or for its contractor) with all that bureaucratic overhead?
More than I think most people here realize. Some people find particular, unique problems fascinating. Others have a sense of duty that outweighs material or personal concerns. Others just don't know better.
Large orgs want guarantees and certainty, and so typically look for software support from other large organizations. Easier to hold IBM to account than a small independent vendor who may have gone out of business by the time you need them. Yes, this is an expensive approach.
As per the article, they're very (very very) slowly migrating from Windows XP to Windows 7.
I suspect Puppet and Chef are not even on their radar at this point.
The reason it fails is because of articles like this disincentivizing the talent they need from being interested in working for either a) .gov/.mil or b) government contractors. Additionally, the contracting process being divorced from the project execution weighs heavily.
Articles like this don't disincentivize a lot of people, and burying it would just let them be lured in and find out the hard way how bad it is. Articles like this air a problem that needs to be corrected.
What disincentivizes talented (specifically young talent) people is the ageism inherent in the system and its perverse method of rewarding people (specifically talking civil service here, contracting's another beast). Unlike the ageism you hear about in the occasional article about Silicon Valley, its the young that have a hard time in civil service. Positions aren't available because too many people are waiting to retire (for various reasons). When they do get in there are few leadership roles open for advancement (see previous sentence). Their pay is strictly time and not merit based (well, not 100%, but NSPS is gone and only a handful of places use its equivalent of merit based pay increase). Career fields are disincentivized, for instance engineers are treated better than computer scientists and mathematicians even in offices where the latter make more sense (bonuses, faster promotion schedules for trainees, pay differential for the degree). Lack of proper training programs (this is office by office, so not sure how universal the problem is, but it's pretty shabby in a lot of places). People are expected to be hired in and, within months, become experts on arcane and outdated systems, with no support because the actual expert has just retired (that's whey there was an opening). People leave quickly, similar to the Teach for America article earlier in the day, because they burn out or realize there's no where to go, neither up nor down, just steady.
On the other hand, in some regions older, talented folks can get a good deal. They can come in as a high grade GS employee with a middling step, automatically enrolled in a decent 401k (5% matching), good insurance (less critical with Obamacare?), and, after a couple years, effective tenure so they have stability which they might value for their family.
This article is only touching on a small portion of why government IT is terrible.
People like to blame contractors but I think a large part of the problem is that the federal employees who are responsible for these big projects are rarely held accountable for failure. In fact, the lack of accountability for federal employees is why so much work was farmed out to contractors in the first place.
Then there's the fact that a lot of the decisions in government IT are made by people who have little understanding of how successful software gets built. People don't understand the importance of defining the scope of the project properly. Or you get people that change requirements willy nilly. Or you find people that are just trying to protect their little fiefdom so you end up with wasteful duplication of effort or withholding of vital information.
The Ars article indirectly touches on the fiefdom issue. When you have dysfunctional IT departments or applications, often times instead of fixing the dysfunction, someone just creates their own little separate IT effort to get around it. So the VIP's might get something functional while the peons get classic ASP & Access.
What I find very interesting about the Healthcare.gov problem is that this may be the first time that a major federal IT project failure has major political consequences for elected officials. If Healthcare.gov ends up being a crash and burn failure, it could result in Republicans gaining enough power in the next election to overturn the whole ACA.
I don't think lack of accountability is what pushed the work out to contractors. And if that is the case, does that mean the accountability "stick" got pushed out to the contractors. I think work got pushed out to contractors from a desire to out-source and because contractors, for various reasons, generally do a much better job of staying abreast of best practices than the government employees. And I say this as someone who has worked on both sides of the fence.
I agree that requirements are a big issue. In addition to the requirements issues you mention, there is the problem with the "business line owners" attempting to unnecessarily dictate implementation details, poorly defining the functional requirements in the first place and ignoring interfacing systems.
Where lack of accountability comes is the idea that contractors can be fired if they don't do a good job, whereas it is much more difficult to fire a full-time employee. So rather than solving that problem, most of the government's software development is farmed out to contractors. The other reasoning for contractors is that contractors don't get retirement benefits that FTE's do, so in theory that saves the govt money but in practice it probably doesn't, given that contractor bill rates are usually 200-300% or in excess of what FTE salaries are.
Not to say that FTE's are great or anything, contractors do tend to me more up-to-date on technology but I think that is a result of the government employee hiring practices and compensation scale.
Yeah I've seen a lot of "business line owner" issues like the ones you've mentioned. Another common one is that software is often built to mirror dysfunctional and wasteful business processes. So you either get software that is needlessly complex, or you build the software then have to re-do it because people realize how bad the underlying business process was.
> Where lack of accountability comes is the idea that contractors can be fired if they don't do a good job, whereas it is much more difficult to fire a full-time employee.
When was the last time you saw that happen? The only major .gov IT screwup which I can recall having any consequences for the vendor was the FBI's virtual Case File system. Everything else tends to get brushed under the rug either to avoid disrupting the relationship (!) or because nobody wants to have to explain why they didn't say anything before the failure was publicly obvious.
Exactly. The idea is that contractors can be fired for a poor job, but the reality is that it rarely happens. Federal employees are rarely held accountable for their failures so in turn they rarely hold contractors accountable. Maybe that contractor doesn't get to stay on for option years, but many times they do for a myriad of reasons.
Then there's the incestual revolving door between government and contracting companies. Government lead retires, then goes to work for the contractor they used to oversee. The biggest contractors use that as a prime mechanism for earning new business. In some companies the best path to get into senior management positions is by being a former Fed.
NYC had a runaway project where they spent something like $300M on an employee time card system (google "city time"). There was no consequence until some bank teller noticed that two guys were literally stuffing duffel bags of money siphoned from the project into safe deposit boxes.
Here in the UK most government contracts just go to the big enterprise shops (think Accenture, IBM, Logica, Cap Gemini, ATOS etc) who manage to negotiate fairly hefty long term deals and charge a healthy commission for taking them on and running them for years.
This is why you run into legacy codebases that are 10-20 years old and will keep on going.
I've done IT at both megacorps and govt. Frankly, I don't see a difference. If anything, govt work has greater transparency and accountability.
Contributing factors are, but not limited to:
- scale
- complexity
- legacy / installed base
- laws, regulations, rules
- organizational processes (workflows)
- quality of developers
- resources
- politics and agendas
Just trying to support all the entrenched chaotic mutant use cases and edge cases will wreck any project plan.
I much prefer working at younger, smaller companies. There's fewer roadblocks. But the healthcare and other benefits working for government rock, so here I am.
65 comments
[ 2.8 ms ] story [ 132 ms ] threadTalking with folks afterwards, I heard about systems that I could have rewritten in the late 90s in a few months that are still in the process of being rewritten, 15 years later. After literally hundreds of people have worked on them costing the taxpayer millions of dollars. And these are not unique or oddball edge cases. Stuff like that is standard operating procedure. I can't impress on the HN audience enough how terrible federal IT is overall. (Aside from some really great outliers). There's a vast and increasingly large gap between political rhetoric and the reality on the ground. This has been true for at least the last 20 years.
I take issue with some of the details the supporting argument makes here, but that's mostly a quibble. The author is definitely headed in the right direction.
I'm a veteran of several federal IT programs, and a real veteran. I have seen some really good federal teams, but never a good federal team of teams. It all falls apart once you get past a few people and start throwing some kind of structure and management in place. I understand that Agile is becoming required for some federal work, but I'll bet you a thousand bucks this kind of "Agile" isn't going to be the kind that the rest of us use. Think big tools, big management, lots of stupid measurements, and no accountability. All with the word "Agile" stamped on the front of it.
...The bottom line is that federal IT programs' success is measured by things that have nothing to do with how successful they are or by the metrics most of the world uses. While the business world (and Web companies in particular) now monitor user experience and productivity as a metric for IT success, the government keeps throwing out numbers that mask the truth: the only people who would use their systems are the ones that are forced to...
Yep.
We saw this same thing with source control. Who doesn't like source control? But the government, in an effort to make sure there was absolutely no risk, mistakes, or accountability associated with software development, started adding more and more mandatory forms that had to be completed each time a developer checked something back in.
This created such a mess for developers that it wasn't unusual for one check in to take hours to accomplish. You'd read a book while the system tried to accept your code changes.
Agile is all about failing fast and learning quickly. Most contractual work for the government does not include the possibility of failure. At all. In fact, the way the contracts are written is such that the contractor takes on all responsibility for everything, even things there's no way they could control. In response to this idiotic situation, contractors start bidding at crazy low rates on these things, say 40 bucks an hour, then actually make their money on changes. So it's always in the contractors interest to capture and execute changes, while it's the government is trying to get the world delivered to them on a platter for five bucks. People use the term "perverse incentives", but this is like that taken to the next level. It's a multi-layered thing. It's total systemic failure. We'd call it systemic collapse, but the money keeps flowing in anyway, so that term isn't applicable, although it certainly feels like it.
This kind of distrust doesn't make for anything but a living hell for everybody, government employees and contractors alike. So, with tons of rules and structures in place, the way things really work is through relationships. People who are easy to get along with and helpful get more government work. Both sides of the fence can't control the process, and nobody rocks the boat and everybody gets a piece of the pie. You end up from the government side with some sort of informal and unspoken version of I don't care if you make a lot of money, just make sure me and my friends look good Also it'd be nice to pick up a little lucrative consulting work once I've put in my 20 and can start double-dipping.
Your average developer has absolutely no idea about this. If they've worked in IT in federal work, they probably have horror stories, but the more you see the worse it gets. Oddly enough, in the best cases you end up with people who really like working together, are motivated, but spend most of their time fighting a Sisyphean fight against the idiocy of the system.
That's because "the government" is operating under different pressures than corporations. A corporation, big or small, screws up, only a handful of stakeholders are affected and raise a stink. If the government screws up, you're looking at headlines in the Washington Post.
Plausible deniability is a powerful force.
Plausible Deniability: Delegation of tasks and lack of records of communications letting the actual responsible party deny knowledge of the "insubordinate" or "rogue" underlings.
CYA: A process to change the process to define the process for tracking changes to the process, with 20 signatures or more required at each step of the process, digital forms filled out in triplicate. Or, making proper recordings of communications with underlings so when they screw up you can point out that they didn't act according to your direction, and that they failed to report issues back up the chain.
That isn't to say that state governments can't screw things up also but at least the damage is contained and state and local governments generally respond better to their constituents complaints/needs than the federal government.
Agile can't be mandated; you can mandate methodologies (e.g., Scrum), but by the very act of mandating a methodology you are not agile. Agile is a culture, and it is a culture of flexibility in methodology with focus on the goal.
And its a culture that is very much contrary to the entrenched culture of most large bureaucracies, government or private (and, in government, often in tension with binding mandates where the authority to make changes to the mandate is very distant from the locus at which one would normally expect efforts to adopt development approaches.)
That's hardly confined to Federal IT projects, however.
Probably why the government is so big and expense. I can only imagine what that $600+ billion gets us for the military. -_-
http://en.wikipedia.org/wiki/Military_budget_of_the_United_S...
Strange to contrast that with a tremendously active and efficient era of public works from the 1890s through the 1950s. Just thinking of New York City, I suspect that the total cost of all the bridges, tunnels and highways (adjusted for inflation) might add up to a lot less than the 800 billion USD that disappeared into our recent stimulus bill. What changed?
edit: Apparently the Brooklyn Bridge cost ~15 million USD to originally construct, which now would be somewhere between 300 million to 400 million USD (http://historical.whatitcosts.com/facts-brooklyn-bridge.htm). Even though these are rough figures and inflation over large time horizons isn't a perfect measure, I'm still astounded. The government's totally broken healthcare website cost more to build than the Brooklyn Bridge.
Vague platitudes about society at large aren't crashing government projects. Society at large aren't really involved in them, after all. Something specific between the interaction of the government and the government-chosen contractors is probably the problem.
In a way, you're probably proving far more than you mean, accidentally; if the problem really is "society at large", then there is no solution and these large projects can't work, and therefore the most effective thing the government could possibly do is stop trying and wasting money. (A project with a prerequisite of "change the larger culture's attitudes" is simply a guaranteed failure, after all.) I suspect that's not the point you're going for. You probably really want to be arguing for a much more local reason.
Bad oversight: whoever's managing the project, doesn't understand the project. Not just, "I don't know how to read <some language>" lack of understanding, but "I don't know the difference between a spec and code". They don't specify terms that keep contractors in line which allow projects to run years and millions or billions over budget. Most of these contract managers aren't engineers, they may have engineers on staff, but it's not uncommon for those engineers to be employees of the same contractor that receives the benefits of the contract. The system is both incompetent and corrupted.
The companies receiving contracts that habitually run late and over budget are also some of the largest. Small contractors don't make it when this happens, but NG, Lockheed Martin, Boeing, they can survive the political fallout thanks to their clout.
If the companies have no oversight, and no accountability internally because they have no real concern for the other stakeholders (ie, the people paying them, the citizens/taxpayers), and the desire to make high quarterly profits, they'll rig the game by rent seeking, rather than completing a project effeciently and effectively and moving on to the next one. See NG and the maintenance of the JointSTARS aircraft as one such example. A rushed, but militarily valuable, project that had no maintenance plan in place, ends up contracted out to a company that destroyed one of the (few) planes and received no real punishment for it.
Thanks to the tenure-like nature of civil service, a large chunk of government employees are retired in place, new employees have no place to start (few positions opening), and those that get in have a terrible example in place and become retired in place after a handful of years themselves. They may start off dedicated to the job, and many do remain dedicated, but they are also frustrated by the processes involved and either quit or ride it out to retirement.
And regarding the platitudes, if companies and citizens (and consequently many of the elected and appointed officials) see profit as motive, rather than the future of the country or the society, then they make decisions based on that altered morality. Decisions based on short-term profit may not be harmful to long-term objectives, but they often are or are neutral to such objectives. A modern works project on the scale of the construction of the interstate system would likely never see the light of day, and if it did it would be mismanaged so as to never achieve its objectives.
Normally I type these up in emacs and copy them over, this is not a well formatted message, sorry about that.
Would it help to know that the same government (altho Federal Reserve is private but heavily regulated) print the money for their own use?
Imagine you have a small house with a garage inside and printing money machine in it. Now you want to expand the house and you know you can pay for the materials labor, etc, thanks to your printing machine. Wouldn't you do it?
The answer is those who care about US, about their outlook on the world would be very careful with using this machine. Those who would "own the house" for just a little bit would abuse it. We are at this stage where government human resources are so overbloated, ,you need so many signatures to buy a roll of toilet paper that eventually at the end of the day nobody is responsible for anything.
That can least if you own the world and don't have to answer to anyone. Un/fortunately this is not US case. If we default, we lose credibility with other major player such as China. Imagine if China stops producing for US (would be insane embargo if you think about out), can you imagine what would happen with us economy? all of sudden the soap dispencer you buy at wallmart cost $45 (made in USA) instead of 4.99 (made in China). Unfortunately, like I said there is no credibility even Chief in Charge Barrack Husain Obama is not aware (in most part) what is really going on under his watch.
And;
Your awkward attempt at slurring the president actually involved spelling two of his names wrong.Weeks of development time have been spent dealing with how to keep our memory load under 1Gb, since that's how much RAM we have. Granted, 4GB of RAM would have been more than enough, but all that developer time saved the tax payer $20.
On the reverse side, I've seen half a million spent on hardware for a project where the sys-admin was making $35k.
The guys who built the Brooklyn bridge were day laborers who made nothing and died young.
http://articles.latimes.com/2006/jul/02/news/adna-sand2
The young guys probably make 125-160k, the old guys get overtime preference. IMO they deserve it -- it's a really dangerous job.
In any large scale project, any issue cascades and costs a shit-ton of money. For government contracts in many states, you have stuff like "prevailing wage" (where the gov pays too much for labor), "project labor agreements" (where the gov agrees to employ X number of people), minority and women owned business quotas (a big pit of corruption).
Also remember that governments are run by politicians, and politicians embrace magical thinking. Painting a rosy picture to get something approved and blowing huge money with no tangible result is a good way to raise capital in govt. (whoops, $X didn't do the trick, but to avoid wasting it, give me $X+Y)
The USA, like many countries, has rules about simply seizing land that are of the form "well, OK, if it's for a good cause, but you better pay for it".
http://en.wikipedia.org/wiki/Interborough_Rapid_Transit_Comp...
http://en.wikipedia.org/wiki/Brooklyn%E2%80%93Manhattan_Tran...
Busses and ferries were private too:
http://en.wikipedia.org/wiki/Fifth_Avenue_Coach_Company
I agree though that the mix of in-house expertise vs outside contractor effort does play a big part in the outcome. Look at Boeing's 777 vs 787 aircraft development for different sides of cost to use internal resources, vs leaning more heavily on external resources. The 787 with it's heavy use of subcontractors has had a much more difficult introduction.
The government in general has been upping the contractor content in a lot of projects with the belief that more "competitive private contractor" content automatically means better efficiency. But, in truth, it's no silver bullet, often increasing costs if it's blindly applied - especially if you fall below some threshold of in-house expert judgement.
I can't vouch for the American case, but New Zealand has a similar history of disastrous government-run IT projects - from a terrible teacher payroll system[1] to a police information system[2]. The IT community hear is desperately trying to stop the same mistake with a massive tax system overhaul[3].
Nevertheless, civil infrastructure projects appear to be very well executed. Recent major roading engineering projects like the Victoria motorway tunnel in Auckland were completed on time and under budget, for example. There has been an active discussion in this country about why the IT industry, in particular as manifested as major systems projects executed by foreign software companies, are such consistent clusterfucks.
It's a big discussion and I won't go into many of the points raised here, but the issues is, IMHO, very much a "software industry" one, and not a "government" one.
1. http://en.wikipedia.org/wiki/Novopay 2. http://en.wikipedia.org/wiki/INCIS 3. http://www.stuff.co.nz/business/industries/9261063/Row-deepe...
Construction management is a serious discipline that is studied by civil engineers. It helps that construction methods don't change very often. Software technologies can become obsolete in a matter of months.
I wouldn't start calling IT procurement a failure compared to civil infrastructure until software engineering has matured to the point that civil engineering has.
I live in Perth, Australia. Almost every civil project there is blowing out like a balloon. Opera houses, hospitals, fancy developments. All going over budget and schedule.
I was in Toronto a few days ago. What's on the cover of the newspaper? A civil project. A subway station running way over cost and time.
Industrial and civil megaprojects routinely go way off course. Bad project management is universal, good project management is hard, any project of any serious size will require owner-commissioner organisations to not be useless idiots.
But, obviously these are all anecdotal, I just know from my time at a large civil engineering company that the failure rate appeared to be very low. Part of that is excellent risk management, but also long-term client relationships, clear and long established success criteria and a distinct lack of political meddling (once the project is underway, at least).
http://marginalrevolution.com/marginalrevolution/2006/08/the...
Large projects fail or run over budget because of organizational failures: poor goals, lack of accountability, internal gridlock, etc. Nothing about that is specific to government – and if we wanted to change that in the United States we could reach the level of, say, Scandinavia if we were willing to make significant cultural changes.
...The IRS has failed twice to modernize its system, in the 1970’s when taxpayer privacy issues surfaced and in the mid-1990’s when the IRS spent ten years and $2 billion on the project and had little to show for it. The current project launched in 1999 with a budget of $8 billion. The project is late and over-budget; problems have been well publicized after the release of the IRS Oversight Board investigation...
http://www.kellogg.northwestern.edu/student/courses/tech914/...
AFAIK, the IRS dropped $10 Billion in IT money in just over a decade with little to nothing to show for it.
I used to joke around and say heck, they could have paid me 2 Billion and I would have also provided them nothing, but look at the money they would have saved!
So NSA, which is fairly young, stays out of the political eye (until recently), and small? They do some really cool stuff. But the older, more controversial agencies, covering things like immigration, defense, taxes, health, and so on? Not good.
However, I suspect the true extent of the flaws with Healthcare.gov and the exchanges is not understood yet.
If anything good comes out of this, I hope it's that the government finally starts paying attention to the woeful state of government IT projects.
The size of healthcare.gov is actually well under the average:
http://www.webpagetest.org/result/131010_ZG_X5C/1/details/
(see http://www.webperformancetoday.com/2013/06/05/web-page-growt...)
For comparison, look at WashingtonPost.com: it uses considerably more data and more total requests and yet the page loads much faster:
http://www.webpagetest.org/result/131010_GG_XDK/
If you look carefully, though, you'll see a) that they aren't setting Cache-Control headers and are setting ETags (both performance killers) and b) there's this weird staggered response behaviour which indicates something is seriously wrong. It seems to handle much better in Chrome - 3s vs 70! - so presumably there's some interaction with the browser going on as well:
http://www.webpagetest.org/result/131010_ZW_XD6/1/details/
So why hasn't the likes of puppet and chef caught on? Genuinely curious.
Who is selling support for Puppet and Chef?
What disincentivizes talented (specifically young talent) people is the ageism inherent in the system and its perverse method of rewarding people (specifically talking civil service here, contracting's another beast). Unlike the ageism you hear about in the occasional article about Silicon Valley, its the young that have a hard time in civil service. Positions aren't available because too many people are waiting to retire (for various reasons). When they do get in there are few leadership roles open for advancement (see previous sentence). Their pay is strictly time and not merit based (well, not 100%, but NSPS is gone and only a handful of places use its equivalent of merit based pay increase). Career fields are disincentivized, for instance engineers are treated better than computer scientists and mathematicians even in offices where the latter make more sense (bonuses, faster promotion schedules for trainees, pay differential for the degree). Lack of proper training programs (this is office by office, so not sure how universal the problem is, but it's pretty shabby in a lot of places). People are expected to be hired in and, within months, become experts on arcane and outdated systems, with no support because the actual expert has just retired (that's whey there was an opening). People leave quickly, similar to the Teach for America article earlier in the day, because they burn out or realize there's no where to go, neither up nor down, just steady.
On the other hand, in some regions older, talented folks can get a good deal. They can come in as a high grade GS employee with a middling step, automatically enrolled in a decent 401k (5% matching), good insurance (less critical with Obamacare?), and, after a couple years, effective tenure so they have stability which they might value for their family.
People like to blame contractors but I think a large part of the problem is that the federal employees who are responsible for these big projects are rarely held accountable for failure. In fact, the lack of accountability for federal employees is why so much work was farmed out to contractors in the first place.
Then there's the fact that a lot of the decisions in government IT are made by people who have little understanding of how successful software gets built. People don't understand the importance of defining the scope of the project properly. Or you get people that change requirements willy nilly. Or you find people that are just trying to protect their little fiefdom so you end up with wasteful duplication of effort or withholding of vital information.
The Ars article indirectly touches on the fiefdom issue. When you have dysfunctional IT departments or applications, often times instead of fixing the dysfunction, someone just creates their own little separate IT effort to get around it. So the VIP's might get something functional while the peons get classic ASP & Access.
What I find very interesting about the Healthcare.gov problem is that this may be the first time that a major federal IT project failure has major political consequences for elected officials. If Healthcare.gov ends up being a crash and burn failure, it could result in Republicans gaining enough power in the next election to overturn the whole ACA.
I agree that requirements are a big issue. In addition to the requirements issues you mention, there is the problem with the "business line owners" attempting to unnecessarily dictate implementation details, poorly defining the functional requirements in the first place and ignoring interfacing systems.
Not to say that FTE's are great or anything, contractors do tend to me more up-to-date on technology but I think that is a result of the government employee hiring practices and compensation scale.
Yeah I've seen a lot of "business line owner" issues like the ones you've mentioned. Another common one is that software is often built to mirror dysfunctional and wasteful business processes. So you either get software that is needlessly complex, or you build the software then have to re-do it because people realize how bad the underlying business process was.
When was the last time you saw that happen? The only major .gov IT screwup which I can recall having any consequences for the vendor was the FBI's virtual Case File system. Everything else tends to get brushed under the rug either to avoid disrupting the relationship (!) or because nobody wants to have to explain why they didn't say anything before the failure was publicly obvious.
Then there's the incestual revolving door between government and contracting companies. Government lead retires, then goes to work for the contractor they used to oversee. The biggest contractors use that as a prime mechanism for earning new business. In some companies the best path to get into senior management positions is by being a former Fed.
NYC had a runaway project where they spent something like $300M on an employee time card system (google "city time"). There was no consequence until some bank teller noticed that two guys were literally stuffing duffel bags of money siphoned from the project into safe deposit boxes.
The prime contractor was SAIC.
This is why you run into legacy codebases that are 10-20 years old and will keep on going.
Contributing factors are, but not limited to:
- scale
- complexity
- legacy / installed base
- laws, regulations, rules
- organizational processes (workflows)
- quality of developers
- resources
- politics and agendas
Just trying to support all the entrenched chaotic mutant use cases and edge cases will wreck any project plan.
I much prefer working at younger, smaller companies. There's fewer roadblocks. But the healthcare and other benefits working for government rock, so here I am.