55 comments

[ 4.9 ms ] story [ 31.8 ms ] thread
This situation has reminded me of the Perl Success Stories booklet that O'Reilly used to spread around. And how, for example, the Bank of Scotland or some such institution ended up keeping the Perl prototype for production, it worked so well.

I can help thinking that a few good Perl programmers, and a few high-level bureaucrats to force open the maws of the respective agencies, could have had this sorted in much shorter, and more cost-effective, order.

Also, as I commented elsewhere before, first make it work. Then make it pretty.

I guess I'm being a bit snarky. But I'm also being more than a bit serious.

The goal of a contractor is to extract as much money as possible from their client, not necissarily to do efficient quality work. In some cases it makes sense for a contractor to do quality efficient work, meaning that is the best path to profits, and sometimes it is not.
If contractor is IBM or Kumaran, then yes, their only goal is to extract profit. They are not concerned about product they make it look like they are building. If contractor is Joe Blow Perl coder, then most likely his goal will be to deliver a good system he can take pride in.
Which will also mean further contracts and recommendations from others.

Contractors who do shitty work are putting most of their effort in to landing new clients to replace the ones who will never repeat.

I read that CGI Federal developed the site, but did they get the contract to maintain it? I can imagine that, compared to a non-gov't contract, they have to go through a lot of red tape to fix the issues and build up capacity.
The WA state one is full of XSS and SQL injection vulnerabilities. It's quite beautiful.
They're making progress at least. When the site first opened I couldn't get in, but I had no trouble today. It works just like eHealthInsurance.com -- the site I used to buy my own insurance 3 years ago.

http://i.imgur.com/GwXTkdc.png

Sort of short on details or perspective.

I do remember reading articles six months ago or so that the federal government was disappointed that they would be managing the exchanges for so many of the states. They were originally hoping that they'd only be managing the exchanges for a handful of states, rather than the 36 they're having to shoulder the load for. I can see how that level of political uncertainty adds an extra challenge for estimating load.

Ah, and here I thought $600,000,000 was too much for a website.
I think it's mostly red states that aren't running their own ones.
Idaho is the only solidly Red state that's doing its own.

A number of Blue or Purple states are also punting to the federal exchange, or doing a "partnership" where the federal exchange is doing the heavy lifting:

New Hampshire, New Mexico for the moment for individuals, Utah permanently for individuals, Arkansas, Iowa, Michigan, Illinois.

In many of these cases of partnership, whatever the governor of whatever party wanted, the legislature didn't buy off. New York is the only case I know of where the governor told the legislature to jump in a lake and set up one for the state through an executive order using the temporary money the Feds are donating early on.

And there's Missouri, who's people outlawed a state run exchange, and New Jersey's governor vetoed 2 bills to set up one.

On NPR the other day they said solidly red Kentucky is running their own site, and is doing really well. It sounded like they were more specific in what they wanted, and perhaps had reduced functionality thus complexity. Either way, kudos!

I would have thought California (note: I live in CA) would have had a good healthcare site, considering all the techies, but I guess in this case that didn't work out.

map of state/fed/joint-run sites: http://www.dailykos.com/story/2013/10/03/1243855/-Kentucky-s...

A state with Democratic governor, lieutenant governor and one of two legislative bodies is "solidly red"??? (https://en.wikipedia.org/wiki/Kentucky#Executive_branch https://en.wikipedia.org/wiki/Kentucky_House_of_Representati...) Not to mention the top vote counter (Secretary of State) and Attorney General? In Wikipedia's list of elective state offices, only the Commissioner of Agriculture is a Republican.

Kentucky is as about as "red" as the current Federal government; even the DailyKos article you link to is self-refuting, first claiming it's "one of the only reliably red states" and then claiming "the biggest factor of all [in its claimed success] is Democratic Gov. Steve Beshear".

As I note elsewhere in this thread, Idaho is the only truly Red state implementing a full state exchange, although the map you cite claims it's not full. And that turns out to be in theory temporary, further reading my source (http://www.healthinsurance.org/idaho-state-health-insurance-...) it's in the process of building its own and depending on the Feds for 2014.

Kentucky is as about as "red" as the current Federal government

Mitch McConnell and Rand Paul would disagree... (although of course, given that Obama is somewhere to the right of Nixon on most issues, it's not so far off)

But do Speaker Greg Stumbo, and most especially Larry Clark, the Speaker pro Tempore since January 4, 1993 (sic), disagree?

Do I really have to explain that state officials run state governments, not the state's elected Federal representatives who do their thing in far off D.C.?

I had someone visit one of my classes at PSU and describe the backend system as it is being implemented in Pennsylvania. The system that was described acts as one big middle man and interops multiple legacy systems mostly written in COBOL. I was told they have the time constraint of 7 seconds to process approvals, which to me seems like a lot, but I'm sure the older systems are probably the real constraint. If these legacy systems supported modern APIs and data transfer formats, this could probably all fit cleanly into any web framework and you wouldn't need a backend dedicated to doing tricky data cleansing. I hope the plan is to use this system as a shim while they improve the legacy systems.
Those systems are legacy systems for a reason. They're never being upgraded.

Also, before anyone gets all "hurr durr government!" on this - this is the exact same situation at any major bank or company founded before about 1990.

Good point, before judging the price or quality we have to know what the Government asked them to do and the legacy systems they had to deal with.
Spot on this is normal. I work in the financial services sector and stuff still arrives on dialup and ftp from mainframes regularly in all sorts of odd formats. SOAP is what is in the executive bathrooms and JSON is the cleaner's name and that is it :)
I ordered a part at the local franchise of a major auto parts place (I forget which one). The guy acted a little embarrassed as we stood there listening to the dial-up modem connect to place the order with the corporate system. That was in 2012 when even the small corner bookstore down the road had free wifi.
From what I can tell it's not just legacy systems that are the problem. I just tried the system for the first time today and there are multiple form issues (errors like "Error: id" with no further explanation) and broken links (a link to https://www.healthcare.gov/help/how-do-i-choose-marketplace-... for more info results in a 404).
(comment deleted)
When will we finally hold contractors that build shit products for the public accountable for wasting a huge amount of tax dollars?
They are being held accountable. The press prints a story, you get all angry, hopefully you vote somebody into office, then they create new rules, and in ten or twenty years we can start making generalizations about how well those rules are working (or not).

How else would you do it? Public executions of any public-funded program that looks bad? We have an accountability system -- it just might not be designed to optimize the things we would like for it to optimize.

Why not have the federal government have an actual organization of software developers (with possibly a standards committee) that do it themselves instead of contracting it out to private companies?

Something like a NASA for non-space software development?

It used to be that I would look at something like this and pronounce judgment. As I get older, however, I've learned that trying things is always good. See what happens!

My question would simply be: how would this be any different than using a small contractor? Or having really great people in a small team do the same thing?

In other words, what does a new organization and standards bring to the table that's not already there? Simply a new name? What do they do that you currently cannot do separately if you wanted to? Are you really suggesting something new, or just adding a bunch of buzzwords together in an effort to try to make something sound good? Or, to be even more provocative, is the goal here to actually do something, or create something that sounds like it might do something? Because they're two different things. We've been down this road of "Ok, let's set aside this special group, with these special criteria, and a new name" a hundred times. Best case -- it works at a very small scale. Worst case, same old, same old, just with new titles over all the stuff.

You do realize that NASA contracts everything out to private companies? All of the NASA employees I know are either project or program management (the two sides of the contracting coin).
From an obit, http://www.washingtonpost.com/wp-dyn/content/article/2007/05...

Walter Schirra; Fifth Astronaut in Space

By Patricia Sullivan

Washington Post Staff Writer

Walter M. Schirra Jr., 84, one of the original seven astronauts and the only man to fly in the Mercury, Gemini and Apollo space programs....

[...]

[His second] mission was delayed a second time when the Titan II engines beneath the space capsule ignited at countdown and then shut down.

For several heart-stopping minutes, Capt. Schirra and astronaut Tom Stafford, sitting atop a highly explosive mass of rocket fuel, chose not to pull the ejection handle, which would have scrapped the mission. It was a calculated risk. Capt. Schirra trusted that the booster rocket would not explode and that the first attempt to rendezvous with another spacecraft, Gemini 7, could still occur. The risk paid off, and three days later, the launch was successful.

Asked later what he thought while sitting on the launchpad, Capt. Schirra replied, "This was all put together by the lowest bidder."

E.g. the story of the reasearch and developemtn of the legendary Apollo Saturn V first stage engines by Rocketdyne, first for the Air Force and then for NASA, is also legendary: https://en.wikipedia.org/wiki/F-1_(rocket_engine)#History. They used explosives to debug it! That's got to count as a hack....

When agency heads stop being appointed as favors by beauty pagent winners.
(comment deleted)
> Officials say that too many people tried to log on at once; within three days, 8.6m had visited healthcare.gov.

That is an order of magnitude less traffic than I expected. 33 visits per second isn't child's play, but my WordPress blog hosted on a 1GB Linode VPS has easily handled bursts of that kind of traffic in the past.

That's totally an apples and oranges comparison, of course, but it's more and more difficult to understand how this project has failed so miserably. I half bought the excuse that it was one of the highest traffic site launches in history, but apparently that was more speculation than truth.

Except your making the classic mistake of assuming that it's 33 visits per second because you are looking at the 8.6m visitors happening evenly over the course of 3 days. You're also making the mistake of confusing the number of visitors with the number of hits on the site. 20 requests to the server for information is still one visit. And yes, 33 hits per second on a static site, cached at various points is far, far easier to develop.

> but it's more and more difficult to understand how this project has failed so miserably.

So, imagine someone estimated 8.6m visitors over three days, and decided to load test the site for 33 visits per second... =)

Then it becomes really easy to see why it could fall over.

Disclaimer: I've been doing load testing the last few weeks, and have been beating up on other peoples server side applications. It's a lot of fun when it's not your code getting hammered. =)

Agreed. Capacity planning is not about the average case; it's about the 99th percentile case and whether you can afford it.
(comment deleted)
That's incorrect. The entire system, far beyond just the website included those costs. The company that actually built the site got it for a $93.7 million contract. The HALF A BILLION DOLLARS you are remarking also include exchanges facilitated by the government for individual states. Finally, that $93.7 million was over 3 years, 2011, 2012, and 2013.

Now, you can armchair things all you want, but the technical elements of building the healthcare.gov website did not cost HALF A BILLION DOLLARS.

CAPS LOCK IS CRUISE CONTROL FOR COOL!!1!

(comment deleted)
You make some good points, so how about the point that's really getting to me?

THREE AND A HALF YEARS

That is, March 23, 2010 to October 1, 2013, 2 days after the Congress was finished with it. The Administration's CTO should have been bird dogging this from about then, along with the HHS specialists who we know have a clue (CMS.gov and Medicare.gov).

They've had plenty of time to get the basic signup system working ... that should have been undergoing stress testing a long time ago, not failing on people as attested to in this discussion.

We often talk about the triad of time, money and quality. They've had plenty of time, and obviously quality was punted. Is anyone claiming they didn't have enough money to the point it could have caused this level of failure?

Okay, so whoever I replied to deleted their message. They are the ones who wrote "HALF A BILLION DOLLARS." Anyways...

> THREE AND A HALF YEARS

Seriously... I just did it to highlight the fact that's it stupid. Please tell me you're only doing it for the same reason? =)

But anyways, no, the site didn't take three and a half years to make, as is clear by anyone taking a few minutes to do a bit of research.

Anyways, getting to the heart of the matter:

> Is anyone claiming they didn't have enough money to the point it could have caused this level of failure?

I don't know and I don't care. I honestly think that question is irrelevant. Who cares if anyone is asking that?

Doing the emphasis that way is stupid in Hacker News context, I just did it to mirror the now deleted message. Although I do think it deserves really serious emphasis, not because "the site didn't take three and a half years to make" but because they had three and a half years to make the whole system. And the very simplest part of it, the up front registration, is reliably being reported to frequently fail hard. If this rather small and uncomplicated part of the system is badly screwed up, the horrors lurking below....

As for the money, I care because I'm a student of success and failure, ever since a year after I got heavy exposure to a great Version 6 UNIX(TM) system I then learned about Multics, and wondered why the latter failed (in shore, GE sold their computer business to Honeywell).

So I listed the three things that are said to be in tension, "you can have only two" is a common saying, and pointing out that this system did not seem to fail due to two of them, it had lots of time (one reason to sole source it), and quality was sacrificed, so what about the third, did it have enough money?

If it had sufficient money then we've got to look deeper.

Your WordPress blog doesn't interact with hundreds of APIs with varying quality and reliability. Healthcare.gov does and while many departments it interacts with are all furlough.
In phone networks, the difference between peak and average load is typically at least one order of magnitude higher. Every day there is a difference of 4x between min and max, with the peak being sustained for multiple hours.

That's why telco systems are sized as "BHCA" (busy hour call attempts) and "CAPS" (call attempts per second), which are peak rates, not average rates. Well, they are both averages, but not over the whole day or year!

Then, as others have pointed out, there's a difference between a static site and a non-static one. This is mostly determined by I/O. A single disk can do between 100-150 IO operations per second (IOPS). Reading a row from a database will take several IOPS. In the systems I work with, we have to keep the entire database in memory to meet our performance numbers, and even then we are still O(database reads).

Doing a SOAP request to a legacy platform that was designed to only support batch requests? Lots of opportunity for slowness. Particularly when you start to run into TCP timers controlling port reuse.

Doesn't excuse the 404's though. That stuff is easy to test automatically.

I work with a lot of technology teams, in a lot of different situations. I'm also a programmer.

I think the tendency in these Monday-morning quarterback sessions is to blame the people or the tech. If they only used X,Y, and Z! people will say, or Why can't they find qualified people who can do this, it's not that hard!

When that fails, we start blaming groups of people. Those damn contractors. Always screwing us over! or Dang government bureaucrats! Lazy bums!

These patterns of analysis are almost always mistaken. You don't drop a billion bucks and get screwed because you hired dumb people, because you're lazy, or because they decided to use WhizBang 1.0 instead of 2.0 You get screwed because you have smart, motivated people working inside of a system of delivery which is hosed. Incentives are wrong, feedback loops are loose (or non-existent), and so on.

So while this is a nice political topic, and sure to get everyone's emotions roiling, be aware that we see this same stuff all of the time. (I would make the case that government is far worse, for exactly the reasons I mentioned above, but I won't because it's only going to push many of your buttons).

It's not the simple stuff that kills projects. It's certainly not tech. It's particular configurations of people that choose bad tech, or that refuse to change bad tech once it's identified. Most all of technology development problems boil down to social problems, not the other stuff. If you've got a good social structure with ten qualified people, you can rule the world. If you have a bad one with a thousand geniuses, you'll be lucky to get your shoes tied in the morning.

Like Weinberg said in Secrets of Consulting:

   No matter how it looks at first, it's always a people problem.
The evidence in this article is:

  1. Jason Lahoz couldn't sign up.
  2. @Ginnyproffitt was frustrated.
I tried signing up a couple of days ago. The site loaded quickly and worked fine. The only quirk I noticed was that I received two registration confirmation emails instead of one.

We should not be so quick to jump on the government without evidence that there is a problem, especially when you can just go look at the site yourself and see if it works.

I tried signing up on Thursday. I filled out the account information, submitted, the site spun for a few seconds and replied 'Important: Your account couldnt be created at this time. The system is unavailable.' (Note no apostrophe in couldn't, verbatim from site.) Twenty minutes later I received an email 'Your Marketplace account has been created.', with a link to verify my email address. I read the email an hour later, clicked the link, and was informed that the link had expired and that I would need to re-enter my account information.

I realize that the plural of anecdote is not data, but there's got to be some multiplier of plural that is data.

Agreed, but I think it's a pretty big multiplier.
I think a lot of this "gov is incompetent" is overblown. Complex systems like this rarely work exactly right on launch (and rarely launch on time). People have another 2.5 months to register before any coverage actually kicks in. This whole episode was just a layup for right wingers to torch government.
I'm not a right-winger but am disappointed that the healthcare.gov site isn't working at all.

It's 2013 and there are tons of technologies that allow this type of system to be set up fast and scalable. The fact that I can't even login (with a registration I made a month ago) and that it doesn't even give me an error message, and that support (phone and chat) just say "it's experiencing problems right now" is inexcusable when you consider that they had half a billion dollars to work with.

Is a secure registration and login system that at least gets people started on the site too much to ask?

"Is a secure registration and login system that at least gets people started on the site too much to ask?"

Evidently so, even given 3 and a half years after passage of the bill that's this Administration's signature accomplishment to date. While it's hardly unknown in other sectors, I've read one of the problems was HHS/Sebelius making important decisions way late in the process, and I think I recall the usual required changes as well.

But unless the signup process hooks into outside live databases for verification or what have you there's as you note no excuse, especially since disallowing window shopping is one of the biggest modern UI hindrances, something pointed out to the people doing this (some information gathering is needed to offer the right plans, but others and I have compared it to the medicare.gov Part D sub-site, which collects that but doesn't require giving all your information prior to you officially selecting a plan, after which the site informs the chosen provider).

Anyway, I am a "right-winger" who's upfront about wanting to ... eliminate many parts of government, and just from memory I can point out several massive and notorious IT procurement failures that would kill or cripple most private concerns. E.g. last time I checked (which is some years ago):

The FAA is still using their 1940s (sic) techniques, and every attempt to really modernize the initial system architecture that automated it has failed. (They famously hired a couple of retired IBM engineers to make microcode patches for some of their old machines for Y2K). This one has really serious economic consequences.

Every comprehensive attempt to upgrade the IRS following their very first system has failed. They're still using code for the 1959 IBM 1401 (https://en.wikipedia.org/wiki/IBM_1401) in their front end processes.

The FBI is legendary for their failures to ditch paper. I remember at least one failed attempt pre-9/11, there's one acknowledged failure after it, and I don't know if they're really using the Sentinel system as they claim as of 2012, after 7 (!) years of troubled development.

These are the really big ones that come to mind. Which of course is not to say the government is always incompetent. The Census Bureau was famously cutting edge, using punched cards in 1890 (sic), buying a UNIVAC I in 1951 (sic), and when I talked to some of their people in the mid-90s I got a sense of serious competence.

As noted in previous discussions, the same contractor operations CMS.gov and Medicare.gov, and I can attest that the latter site, while a bit clunky, works just fine. CMS's backends seem to work as well.

Off the top of my head, the CDC and FDA do well in getting out timely info. The DoJ's statistics sections are good, the NIH's sites are stellar. Heck, the BATF has a nice web page where you can type in some of the numbers of an FFL (firearms dealer, gunsmith, etc.) and confirm it is legitimate and still valid. Etc.

I was surprised that they had anything ready on October 1st. I expected at least a six month delay. Not because of hurr durr government can't do anything right, but that's the nature of large scale IT projects. They're extremely hard to manage, especially when there are a lot of third party systems involved.
I wonder if this would still be the case if fewer states had punted the work to the federal government. The fact is that the federal government is running 36 exchanges was never the plan.