the difference is that there is a government shutdown involved, which is 100% pure politics, as opposed to topics like the NSA, which aren't necessarily politics, at least not obviously.
At first I thought you used "political" to mean opinionated commentary about software development, platforms, and frameworks. And I thought, "bring it on!" And then I realized you meant political. Oh. That.
A bunch of other better articles on it have been posted to HN, but not upvoted enough to make the front page. I too find this surprising, this sort of highly visible large scale web app rollout failure seems like a topic HN would like.
Yeah, I've figured there'd be more here about it, but it's sort of like shooting fish in a barrel at this point. Plus... it will inevitably devolve in to politics. And lastly, it's not actually working yet. We don't have any postmortem about what's actually going wrong - maybe we'll get that soon (or never?)
Sure, they could have been prevented. By spending more money. Isn't that always the answer?
Creating a website that can handle, well, nearly the entire country, out of the gate, is not trivial. (Facebook etc got to scale up to that level, they didn't need to do it out of the gate). But it also can be done.
How? Well, you hire real experts in doing this sort of thing. And you give them enough calendar time, and enough billable hours, to do it right. And you don't give them your own 'business requirements' which contradict their expertise on what's neccesary to make it work.
Obviously the government did not do one or more of those things, right?
(I will confess that my opinion is not high of the likely expertise level of typical IT government contracting firms.).
Your opinion shouldn't be high. There are talented IT people working at government contractors, but they aren't going to get put where they are needed, because the incentives for contractors are not performing the work at a top-notch level.
Gov't contracting is, in economic terms, a rent-seeking business. The best and brightest in a given contracting firm are dedicated to pursuing new work. I work for a contractor, and my fun IT projects are all proof of concept items. Stellar execution simply isn't rewarded. The gov't puts idiots in charge of things: not idiots in the sense that they aren't smart, but idiots in the sense that they have no expertise in anything. They are hired for paper qualifications (ever heard of a Project Management Professional cert? It's what they hire, and its useless), and the bright/motivated ones move on to interesting work, perhaps in other parts of the gov't, or the private sector. I built a great tool, using all open source technology for the gov't once. I built it in 3 weeks (it is now officially gov't owned software), and it replaced an $80K piece of software (commercial off the shelf) that was inadequate for the task. My reward was complaints that the security people weren't familiar with Postgres, and an immediate request to migrate to Oracle. There was no functional reason for this, but I complied after the gov't shelled out huge sums of money for Oracle license. Why? Because paper pushing idiots in the gov't are in charge of IT security, and their credentials for the job are based entirely on passing certification tests. Their entire incentivization at their job is to minimize their own workload, rather than maximize the happiness of their gov't customers.
Do you think anybody who could even guage whether proper testing was being performed or not was in charge of this project from the gov't level? Of course not. If they had, it would have been properly tested.
I'm mad as hell about this because I want the ACA to succeed. I don't think its ideal, but its vastly better than the existing system of hospitals as clinics for the uninsured hordes.
The Federal gov't has an awful hiring process, and until it is fixed, you are lucky to encounter competent and driven individuals within it. You are foolish to expect it.
I'm mad as hell about this because I want the ACA to succeed. I don't think its ideal, but its vastly better than the existing system of hospitals as clinics for the uninsured hordes.
I couldn't agree more. You only get one chance to make a first impression and they blew it. (sigh)
I don't think PMP certifications are any worse than any other certifications, though often, as with other certifications, they often aren't worth the paper they are printed on. And I agree that the project managers often have little-to-no technical expertise in the project they are managing; but that's not supposed to be their job. Their job is supposed to be to manage the job that the experts are working on and let the experts do their thing. Although they (PMs) should have at least enough knowledge to understand discussions at a high level, facilitate when necessary and recognize when one of the experts is blowing smoke out his/her ass.
I am not optimistic that they'll get it working by the deadline, sadly. Are you?
I mean, let's start admitting that it's _not_ easy to make a site that can handle that level of traffic. It's do-able, but it's not easy.
Once they have it handling the proper traffic, only _then_ are they going to start finding out about all the bugs that are actually logic related instead of load/traffic related. Which I'm sure there are plenty of, because all software has bugs, and because the QA for those (including integration with IRS system or whatever?) isn't trivial either, and I'm sure they didn't do any better with it then they did with preparing for the load.
Well, you can give them your own business requirements, which should be high-level and state "I need a system that performs these functions for this many people". What you should not do is dictate design and implementation details. That's when you rely on their expertise. . .unfortunately, as you stated, that's rarely how it works.
Also, it would have really helped to have someone - likely not a developer - to bridge the gap and ask the customer "do you really need people to set up user access before viewing plans or you just want that? why?" and then turn to the developer and say "ok, what does that mean technically? what the pros and cons?"
"A scrappy little startup working out of a DC garage that completely influenced the course of how the web-facing portions of the Affordable Care Act were to be implemented"
Premiums vary based on region, age, and smoking status. It's not so much information that you couldn't handle it with static pages.
Subsidy is based on income, and it's simple enough that NPR and others have already build subsidy calculators. Put that in javascript and you've still got static pages, running calculations client-side.
I'm surprised they don't have a separate system for their own staff to use when doing signups over the phone and in person. From all the articles it sounds like staff are going through the main healthcare.gov to attempt signups.
The biggest improvement they could make in the short term is setting up a light weight version so you can just enter the needed information to view your plans/options prior to the signup/verification process. Then once you've decided on a plan go into the signup process.
A lot of traffic is probably just people comparing prices, deductibles, etc putting unnecessary strain on the signup system.
It seems odd that you can't even view the log in page when you click log in on the home page.
Definitely looking forward to details about the site, backend, db, hosting, traffic, etc . . . once everything shakes out.
> The biggest improvement they could make in the short term is setting up a light weight version so you can just enter the needed information to view your plans/options prior to the signup/verification process.
Washington's site did just that. Still blew up. For the first couple of days, just trying to get available plans didn't actually work.
There aren't that many websites out there that experience the kind of traffic levels the exchanges have. Those that do have mostly grown (if sometimes quite quickly) to those levels, not launched with them on day 1.
Competent people with prior experience scaling to these levels are (over-)employed with high compensation. They didn't work on these sites.
I've dealt with having to scale a site to millions of pageviews and honestly, if you've never had to deal with a traffic spike, you probably aren't going to build for the scale issues you will have.
For example, there is a lot of caching you just have to do. Tons of it. Cache as much as you can. Memcache and Varnish are your friends, use them as much as you can. Unfortunately, if Healthcare.gov was in a very write-heavy situation, they are somewhat limited in how much caching will help.
One thing they could have done that would have saved A TON of load is not require users to sign up before giving them a list of available plans. That whole part didn't need to be database driven at all. The data could have been stored in redis and they could have used javascript to filter it based on a form. That would have been ridiculously fast. They also could have prerendered all the plan list possibilities and stored those in varnish. That also would have been ridiculously fast. My guess is millions of people just wanted to check prices and eliminating the database load for those users would have probably kept things running fast and smooth.
Slow DB queries are the enemy and you don't realize how bad they are until you are at scale. Sure, it only takes a few seconds on your local machine, but multiply that times thousands of concurrent users and your DB gets swamped. If you are using an ORM, it is MUCH harder to track down where in your code that 3 way join that scans every record is happening. Ideally you'd be using straight SQL and maybe use comments to tag a query. Also tools like newrelic might help if only because many databases don't offer great visibility of performance data.
Sharding your database is something that is probably possible, and in the case of healthcare.gov, they probably could have had totally separate infrastructure on a per-state basis that would have made scaling a lot easier than putting everything on the same database. Also, put reporting and things that aren't mission critical on a slave database. The last thing you want is a reporting job bringing down the live site in the background.
Getting good hardware with fast IO is going to save a lot of developer time required to scale things. Using fast SSD's is probably the easiest win to speed up your database. Developer time costs a lot more than hardware and giving yourself cheap headroom up front gives you breathing room on launch.
Performance testing is also something worth doing, but the tricky part is until you roll out, it is hard to know exactly where the hotspots are going to be. In this case, new user signup would be the obvious place to test, so they probably should have tested up to the limits of their servers and tried to extrapolate an expected number of users and maybe increased that by 1.5x or something to have some leeway.
On a rollout where you don't know what you are getting into user wise, being on a cloud where you can scale out fast as demand requires is something worth doing. They could have saved a lot of bad press and headache by being on the cloud initially and migrating to less hardware after the initial peak died down.
There are a ton of little things like that you have to think about if you are dealing with massive scale. I don't know if the engineers building healthcare.gov had ever dealt with something like this before, but I'm sure they're learning these lessons now.
One thing they could have done that would have saved A TON of load is not require users to sign up before giving them a list of available plans. ...
My guess is millions of people just wanted to check prices and eliminating the database load for those users would have probably kept things running fast and smooth.
That's all I wanted to do (I have employer-based coverage for now). I turned back once I realized you needed an account to see this, but assumed it was due to some policy issue or agreement with the insurers.
I wonder if the list of available plans is customized, not everyone has access to every plan, and it can't show you the list of plans available to you until it knows who you are.
"Slow DB queries are the enemy and you don't realize how bad they are until you are at scale. Sure, it only takes a few seconds on your local machine, but multiply that times thousands of concurrent users and your DB gets swamped. If you are using an ORM, it is MUCH harder to track down where in your code that 3 way join that scans every record is happening. Ideally you'd be using straight SQL and maybe use comments to tag a query. Also tools like newrelic might help if only because many databases don't offer great visibility of performance data."
Problem is your ORM might generate 300 fast queries to get 300 records where 1 would be faster than 300. using an ORM make things tricky since you basically surrender most of Db control .
Scaling databases is hard,developpers like ORMs , but ORMs make scaling databases even harder.
The big thing with an ORM is that you must keep those lazy loads in mind when you work with it. Lazy loads are the big place where you get "300 queries instead of 1".
ORMs aren't bad on principal, imho... but they give you more than enough rope to hang yourself while promising a magical efficient user-friendly rope.
I wonder why most ORM's don't let you turn off lazy loads -- "only fetch when I ask for a fetch, if I try to access something without having asked you to fetch it first -- raise an exception!" Just as an option.
This would not be a particularly difficult feature for an ORM to implement.
Microsoft's LInQ2Entities offers this. In 3.5 it was on by default, but EF3.5 was the buggiest mess ever. They rushed it out when they realized that LInQ2SQL was unmaintainable, and then made an even worse mess.
It offered a manual "Load" operation that manually initiated the lazy load for a given FK relation, providing a worst-of-both-worlds option.
The thing I could never figure out in LinQ was how to make an explicit load of related objects fetch more aggressively. Like, say I have object Foo. I find out I need Foo->Bar, but realistically I'm also going to need Bar->Baz and Bar->Quux to make Bar useful. I could make a stand-alone query to pull down Bar, Bar->Baz and Bar->Quux in a single query, but then Bar wouldn't be loaded attached to Foo, and if I attached it to Foo it would confuse the object context. I could use the Foo->Bar to load Bar, and then Bar->Baz and Bar->Quux to load Baz and Quux... but that would be too many hits to the DB. I could never figure out how to put those things together, and fetching Foo->Bar->[Baz|Quux] proactively every time I wanted a Foo wasn't preformant.
Of course, this was back when I was obsessed with normalization and was using UUIDs for all my primary keys, so I might be missing the point.
But to be picky, let's note that "manually initiated the lazy load" -- there's nothing "lazy" about that load, heh. The "lazy" part specifically means "on-demand, as asked for, automatically."
In ActiveRecord, you can quite easily set 'eager loading', which is setting certain loads to batch for efficiency, like you're talking about -- and this can apply to lazy loads (when you lazy load X, ALSO load y and z together, in as few SQL statements as possible), OR to manual loads (right NOW, load x, y, and z together in as few loads as possible). But AR doesn't let you turn off lazy loads, ha.
One side effect of the complexity of government contracting processes (designed, in principle, to ensure that the government doesn't get taken advantage of by either internal or external actors) is that the people who get awarded government contracts are often the people with the most skill in negotiating the government contracting process, not the people with the most skill in the problem domain.
We know who built Healthcare.gov; it was Development Seed. There were a bunch of stories about it earlier this year. Development Seed also built and runs MapBox, so they should have some idea how to run a high volume web service.
What's surprising to me at this point is that nobody has published the actual plans, separately from this website. Best thing the government could do right now is get those plans and their rates out to news organizations, so people can see them without going to healthcare.gov, which could go back to being just the enrollment application that they seem to have intended.
The website includes the plans with the subsidies, which is why they require you to sign up and answer a bunch of questions. I believe there's a lot of verification of subsidy eligibility going on in the background. Just having the rates out there would make it appear that the plans are much more expensive than they will be for most people.
If anyone has seen my posts in the past, you'll know that I'm not a fan of the ACA, but disregarding that, let me be the first to say that this probably isn't the fault (or, at least not solely the fault) of those building the web frontends.
In Maryland, our exchange website is poorly designed and written, just on the frontend (I obviously can't see the backend), but at the same time, all of the frontend issues could be fixed with a clever caching scheme.
Where the real bottleneck almost certainly lies is when the system takes your user submitted data and has to post it into what is surely an old, legacy federal system so that it can verify your identity. That old legacy system may have seen upgrades in preparation for this, but probably not -- even so, there's simply no way to prepare for the onslaught of users accessing (indirectly) what was certain to have been an isolated, government-only database in the past.
You can't see what's running at the moment of course, but why shouldn't you expect to be able to see the source? Why are new government IT projects closed source?
This is only the content site, not the marketplace. See: "This project does not include any source code for the Federal Health Insurance Marketplace (the online systems located under www.healthcare.gov/marketplace."
Setting aside the politics, I really don't think it could have been prevented.
The specs were not exactly realistic on number of users at a time. Heck, even Apple still gets slammed and they know its coming.
Plus, the experience just isn't in the DC contracting community. They can do websites, but not high availability transactional. If we were talking back-ends, then yes they have high transaction experience, but not with websites.
I will be expecting second (third?) day stories about bad data problems and failed transactions. This won't be a simple thing.
But isn't the DC contracting community involved in other high availability transactional systems? For example, isn't one of the ACH systems a government or pseduo-government system?
Oh yes they are, but not anything with a web-front end. They do really well with mainframe and back-end, just don't expect anything that has to touch a web stack.
Regardless of any technological or architectural decisions, simple load testing before launch would have prevented this.
1. Load test, discover issues.
2. Fix issues.
3. Repeat steps 1-2 until no more issues.
That is the flabbergasting part. Nobody actually tried applying real load to the system before release? It fell over so easily and quickly the only assumption is either no loadtesting was performed, the results were ignored, or massively incorrect load numbers were used (all of which are signs of incompetent mangement).
They probably load-tested based on the number of sign-ups they expected, and that estimate may even have been accurate. They don't seem to have realized that lots more people would be interested in what's available, even if they aren't ready to enroll. If they had realized that, you wouldn't have to "apply" before seeing the plans, and the plans could have been static content, distributed all over a CDN by now.
The official story right now is that they underestimated. Medicaid has had at max 30k users; they doubled the estimate for hc.gov, but instead of ~50k users they're seeing 250k. From what I'd read (in a story linked from this hn page) they did test for load, but nowhere near 200k users.
FWIW, I don't think the issues right now are load so much. I think there's some insanely bad logical pieces that are messing things up, and possibly have corrupted accounts that were created early on. I base this on a few things.
1. The "username" requirements.
The username is case sensitive. Choose a username that is 6-74 characters long and must contain a lowercase or capital letter, a number, or one of these symbols _.@/-
Really? I know some systems use case-sensitive usernames, but given that you're already forcing some odd characters and whatnot in there, why not normalize to lowercase? This just feels like it's going to cause more support problems (mobile safari automatically uppercasing a username that should be lowercased, etc).
Also... the English description of that username is nowhere near intuitive. "A lowercase or capital letter". As opposed to what other kind of letter?
2. Multiple accounts with a single email address.
I've been able to 'successfully' register multiple different username accounts with the same email address. When I do a password reset based on email (the couple times it worked) which username was I resetting the password for? I say 'successfully' because no login attempt has ever worked. And now trying to register yet another username with the same email address doesn't work, but the error message is so vague ("there was a problem" IIRC) that I can't tell if that was a factor or not.
OK... 2 things. That's about as far as I've been able to see in to the system so far, so that's all I can judge it on. But it seems that they've probably allowed some logical inconsistencies in my own signups that may be causing more problems now, and I don't think they are related to load.
Of course, I could be 100% wrong, but it doesn't feel like load is the culprit right now.
I immediately get an error page after logging and and trying to view plans. That could be load but it certainly feels like some kind of logical error to me.
Exactly. They've got the 'waiting room' technique to slow down apps/logins. The login process, once done, is quite fast in getting me to a blank page. (https://www.healthcare.gov/marketplace/auth/userprofile page in question)
53 comments
[ 3.1 ms ] story [ 123 ms ] threadhttp://www.usatoday.com/story/news/nation/2013/10/05/health-...
Didn't that site cost millions to make?
Wait, is this blog entry trying to sell their own products?
This is a better round-up of stories about the website:
http://www.theatlanticwire.com/politics/2013/10/obama-admini...
Not sure why this one finally made it, heh.
Creating a website that can handle, well, nearly the entire country, out of the gate, is not trivial. (Facebook etc got to scale up to that level, they didn't need to do it out of the gate). But it also can be done.
How? Well, you hire real experts in doing this sort of thing. And you give them enough calendar time, and enough billable hours, to do it right. And you don't give them your own 'business requirements' which contradict their expertise on what's neccesary to make it work.
Obviously the government did not do one or more of those things, right?
(I will confess that my opinion is not high of the likely expertise level of typical IT government contracting firms.).
Gov't contracting is, in economic terms, a rent-seeking business. The best and brightest in a given contracting firm are dedicated to pursuing new work. I work for a contractor, and my fun IT projects are all proof of concept items. Stellar execution simply isn't rewarded. The gov't puts idiots in charge of things: not idiots in the sense that they aren't smart, but idiots in the sense that they have no expertise in anything. They are hired for paper qualifications (ever heard of a Project Management Professional cert? It's what they hire, and its useless), and the bright/motivated ones move on to interesting work, perhaps in other parts of the gov't, or the private sector. I built a great tool, using all open source technology for the gov't once. I built it in 3 weeks (it is now officially gov't owned software), and it replaced an $80K piece of software (commercial off the shelf) that was inadequate for the task. My reward was complaints that the security people weren't familiar with Postgres, and an immediate request to migrate to Oracle. There was no functional reason for this, but I complied after the gov't shelled out huge sums of money for Oracle license. Why? Because paper pushing idiots in the gov't are in charge of IT security, and their credentials for the job are based entirely on passing certification tests. Their entire incentivization at their job is to minimize their own workload, rather than maximize the happiness of their gov't customers.
Do you think anybody who could even guage whether proper testing was being performed or not was in charge of this project from the gov't level? Of course not. If they had, it would have been properly tested.
I'm mad as hell about this because I want the ACA to succeed. I don't think its ideal, but its vastly better than the existing system of hospitals as clinics for the uninsured hordes.
The Federal gov't has an awful hiring process, and until it is fixed, you are lucky to encounter competent and driven individuals within it. You are foolish to expect it.
I couldn't agree more. You only get one chance to make a first impression and they blew it. (sigh)
I mean, let's start admitting that it's _not_ easy to make a site that can handle that level of traffic. It's do-able, but it's not easy.
Once they have it handling the proper traffic, only _then_ are they going to start finding out about all the bugs that are actually logic related instead of load/traffic related. Which I'm sure there are plenty of, because all software has bugs, and because the QA for those (including integration with IRS system or whatever?) isn't trivial either, and I'm sure they didn't do any better with it then they did with preparing for the load.
Also, it would have really helped to have someone - likely not a developer - to bridge the gap and ask the customer "do you really need people to set up user access before viewing plans or you just want that? why?" and then turn to the developer and say "ok, what does that mean technically? what the pros and cons?"
"A scrappy little startup working out of a DC garage that completely influenced the course of how the web-facing portions of the Affordable Care Act were to be implemented"
Subsidy is based on income, and it's simple enough that NPR and others have already build subsidy calculators. Put that in javascript and you've still got static pages, running calculations client-side.
The biggest improvement they could make in the short term is setting up a light weight version so you can just enter the needed information to view your plans/options prior to the signup/verification process. Then once you've decided on a plan go into the signup process.
A lot of traffic is probably just people comparing prices, deductibles, etc putting unnecessary strain on the signup system.
It seems odd that you can't even view the log in page when you click log in on the home page.
Definitely looking forward to details about the site, backend, db, hosting, traffic, etc . . . once everything shakes out.
Washington's site did just that. Still blew up. For the first couple of days, just trying to get available plans didn't actually work.
There aren't that many websites out there that experience the kind of traffic levels the exchanges have. Those that do have mostly grown (if sometimes quite quickly) to those levels, not launched with them on day 1.
Competent people with prior experience scaling to these levels are (over-)employed with high compensation. They didn't work on these sites.
For example, there is a lot of caching you just have to do. Tons of it. Cache as much as you can. Memcache and Varnish are your friends, use them as much as you can. Unfortunately, if Healthcare.gov was in a very write-heavy situation, they are somewhat limited in how much caching will help.
One thing they could have done that would have saved A TON of load is not require users to sign up before giving them a list of available plans. That whole part didn't need to be database driven at all. The data could have been stored in redis and they could have used javascript to filter it based on a form. That would have been ridiculously fast. They also could have prerendered all the plan list possibilities and stored those in varnish. That also would have been ridiculously fast. My guess is millions of people just wanted to check prices and eliminating the database load for those users would have probably kept things running fast and smooth.
Slow DB queries are the enemy and you don't realize how bad they are until you are at scale. Sure, it only takes a few seconds on your local machine, but multiply that times thousands of concurrent users and your DB gets swamped. If you are using an ORM, it is MUCH harder to track down where in your code that 3 way join that scans every record is happening. Ideally you'd be using straight SQL and maybe use comments to tag a query. Also tools like newrelic might help if only because many databases don't offer great visibility of performance data.
Sharding your database is something that is probably possible, and in the case of healthcare.gov, they probably could have had totally separate infrastructure on a per-state basis that would have made scaling a lot easier than putting everything on the same database. Also, put reporting and things that aren't mission critical on a slave database. The last thing you want is a reporting job bringing down the live site in the background.
Getting good hardware with fast IO is going to save a lot of developer time required to scale things. Using fast SSD's is probably the easiest win to speed up your database. Developer time costs a lot more than hardware and giving yourself cheap headroom up front gives you breathing room on launch.
Performance testing is also something worth doing, but the tricky part is until you roll out, it is hard to know exactly where the hotspots are going to be. In this case, new user signup would be the obvious place to test, so they probably should have tested up to the limits of their servers and tried to extrapolate an expected number of users and maybe increased that by 1.5x or something to have some leeway.
On a rollout where you don't know what you are getting into user wise, being on a cloud where you can scale out fast as demand requires is something worth doing. They could have saved a lot of bad press and headache by being on the cloud initially and migrating to less hardware after the initial peak died down.
There are a ton of little things like that you have to think about if you are dealing with massive scale. I don't know if the engineers building healthcare.gov had ever dealt with something like this before, but I'm sure they're learning these lessons now.
That's all I wanted to do (I have employer-based coverage for now). I turned back once I realized you needed an account to see this, but assumed it was due to some policy issue or agreement with the insurers.
1. Slow Query Log 2. Grep commands
Scaling databases is hard,developpers like ORMs , but ORMs make scaling databases even harder.
ORMs aren't bad on principal, imho... but they give you more than enough rope to hang yourself while promising a magical efficient user-friendly rope.
This would not be a particularly difficult feature for an ORM to implement.
It offered a manual "Load" operation that manually initiated the lazy load for a given FK relation, providing a worst-of-both-worlds option.
The thing I could never figure out in LinQ was how to make an explicit load of related objects fetch more aggressively. Like, say I have object Foo. I find out I need Foo->Bar, but realistically I'm also going to need Bar->Baz and Bar->Quux to make Bar useful. I could make a stand-alone query to pull down Bar, Bar->Baz and Bar->Quux in a single query, but then Bar wouldn't be loaded attached to Foo, and if I attached it to Foo it would confuse the object context. I could use the Foo->Bar to load Bar, and then Bar->Baz and Bar->Quux to load Baz and Quux... but that would be too many hits to the DB. I could never figure out how to put those things together, and fetching Foo->Bar->[Baz|Quux] proactively every time I wanted a Foo wasn't preformant.
Of course, this was back when I was obsessed with normalization and was using UUIDs for all my primary keys, so I might be missing the point.
But to be picky, let's note that "manually initiated the lazy load" -- there's nothing "lazy" about that load, heh. The "lazy" part specifically means "on-demand, as asked for, automatically."
In ActiveRecord, you can quite easily set 'eager loading', which is setting certain loads to batch for efficiency, like you're talking about -- and this can apply to lazy loads (when you lazy load X, ALSO load y and z together, in as few SQL statements as possible), OR to manual loads (right NOW, load x, y, and z together in as few loads as possible). But AR doesn't let you turn off lazy loads, ha.
One question would be whether they hired people/firms who had had to deal with that level of scale before.
I'd assume not.
But apparently they hired someone who didn't even know enough to know that they were not qualified to do it.
In Maryland, our exchange website is poorly designed and written, just on the frontend (I obviously can't see the backend), but at the same time, all of the frontend issues could be fixed with a clever caching scheme.
Where the real bottleneck almost certainly lies is when the system takes your user submitted data and has to post it into what is surely an old, legacy federal system so that it can verify your identity. That old legacy system may have seen upgrades in preparation for this, but probably not -- even so, there's simply no way to prepare for the onslaught of users accessing (indirectly) what was certain to have been an isolated, government-only database in the past.
You can't see what's running at the moment of course, but why shouldn't you expect to be able to see the source? Why are new government IT projects closed source?
https://github.com/CMSgov/healthcare.gov
The specs were not exactly realistic on number of users at a time. Heck, even Apple still gets slammed and they know its coming.
Plus, the experience just isn't in the DC contracting community. They can do websites, but not high availability transactional. If we were talking back-ends, then yes they have high transaction experience, but not with websites.
I will be expecting second (third?) day stories about bad data problems and failed transactions. This won't be a simple thing.
1. Load test, discover issues. 2. Fix issues. 3. Repeat steps 1-2 until no more issues.
That is the flabbergasting part. Nobody actually tried applying real load to the system before release? It fell over so easily and quickly the only assumption is either no loadtesting was performed, the results were ignored, or massively incorrect load numbers were used (all of which are signs of incompetent mangement).
FWIW, I don't think the issues right now are load so much. I think there's some insanely bad logical pieces that are messing things up, and possibly have corrupted accounts that were created early on. I base this on a few things.
1. The "username" requirements.
The username is case sensitive. Choose a username that is 6-74 characters long and must contain a lowercase or capital letter, a number, or one of these symbols _.@/-
Really? I know some systems use case-sensitive usernames, but given that you're already forcing some odd characters and whatnot in there, why not normalize to lowercase? This just feels like it's going to cause more support problems (mobile safari automatically uppercasing a username that should be lowercased, etc).
Also... the English description of that username is nowhere near intuitive. "A lowercase or capital letter". As opposed to what other kind of letter?
2. Multiple accounts with a single email address.
I've been able to 'successfully' register multiple different username accounts with the same email address. When I do a password reset based on email (the couple times it worked) which username was I resetting the password for? I say 'successfully' because no login attempt has ever worked. And now trying to register yet another username with the same email address doesn't work, but the error message is so vague ("there was a problem" IIRC) that I can't tell if that was a factor or not.
OK... 2 things. That's about as far as I've been able to see in to the system so far, so that's all I can judge it on. But it seems that they've probably allowed some logical inconsistencies in my own signups that may be causing more problems now, and I don't think they are related to load.
Of course, I could be 100% wrong, but it doesn't feel like load is the culprit right now.