Sunday is the best day for HealthCare.gov to go down for maintenance. Only 1-2% of your traffic through-out the week is done on Sunday for any healthcare. Plus the system has trouble scaling when Obama was in office. So planned down time seems reasonable to flatten data and process files to and from the Exchanges.
They could phrase it better though. The downtime is announced as "Sundays, 12am-12pm ET". If there is, I'm not aware of any standard as to what that means.
> Uhh.. It means 00:00 - 12:00, every Sunday. From midnight to noon. The first half of the day.
Perhaps likely, but it's not as obvious as your phrasing makes it out to be.
> To avoid confusion in, for example, an insurance certificate, it is always better to use the 24-hour clock, when 12:00 is 12 noon and, for example, 24:00 Sunday or 00:00 Monday both mean 12 midnight Sunday/Monday. It is common in transport timetables to use 23:59 Sunday or 00:01 Monday (in this example), or 11:59 p.m. or 12:01 a.m., to further reduce confusion.
> There are no standards established for the meaning of 12 a.m. and 12 p.m. It is often said that 12 a.m. Monday is midnight on Monday morning and 12 p.m. is midday.
We are talking about America, not the UK. "Sundays, 12am-12pm" has a well established meaning in the US. It's ambiguity in the UK (or elsewhere) is irreverent to a US audience.
We should all move to 24 hour time, yes, there's absolutely no disagreements from me there. AM/PM confusion is completely unnecessary.
However, 24 hour time is not used for anything in the US outside of a few industries (military, aviation, probably transportation). Not even train or bus schedules are given to the public in 24 hour time in the US (in my experience). If you used 24 hour time for anything at all while communicating with the general public you will confuse people. Especially the elderly. 00:00 is complete jibberish to the average American, so in this case they did the right thing given the audience.
We do some (paper) logging at work and a (larger than I imagined) percentage of the workforce (including most people older than 50%) were very, very confused by it. (Even though the instructions are the very simple "if it's after noon add 12 to the hour.")
If you did a survey on this, what fraction of adults do you think would get it right? 80%? Remember, this system is supposed to be used by anyone, including the poorly educated and recent immigrants.
If you assume that people remember the solutions to every problem they've ever encountered, I would not like to use the kind of user interface you would create!
Incidentally, 99.9% is going to be a bad guess for almost any survey answer because of the "lizard people effect" (4% of US adults say that the country is controlled by lizard people, 7% are not sure).
> The style manual of the United States Government Printing Office used 12 a.m. for noon and 12 p.m. for midnight until its 2008 edition, when it reversed these designations.[14][15]
9999 out of 10000? Seriously? Even in a group of geniuses, 10 in 10,000 would answer wrong simply by being absentminded. And the US population is not all geniuses.
That is incorrect. It was running effectively without nearly that much maintenance even last year, and there is no earthly reason for a site to need 12 hours of weekly downtime. It's not as if there are floors that need to be mopped or shelves that need to be stocked.
- Is that for all Healthcare.gov functions or just for signups?
- How do the scaling issues that were solved over 4 years ago matter now?
- Has the lack of this extreme and unheard-of level of "maintenance" downtime lead to prior problems large enough to make this a legitimate action rather than another disgusting attempt to sabotage the American healthcare system by making it harder for Americans to sign up?
For some sites only 1-2% of traffic is on weekends. But weekends are the only time many people have to research and select complicated health plan offerings.
Sunday mornings tend to be a convenient time for churches to do things like help people navigate the exchange.
I'm not sure how much churches have done in past years though. Some information about how the site has been used during the announced downtimes in previous years would be a courageous act of journalism. Maybe some hero will tweet it so that all the online journalists can toss something up.
I don't know which is harder to swallow, this administration's overt hatred of the American people as a whole, or the fact that the people it is hurting the most are its most fervent supporters.
>As part of its mission to translate the false narrative that the Affordable Care Act is failing into reality, Tom Price’s Health and Human Services Department has already announced massive cuts to the navigator programs that fund Obamacare outreach, as well as the budget for ads that alert Americans to the existence of the open-enrollment period.
It just seems like a strange coincidence they need to shutdown the site during open enrollment.
It's a valid complaint. The site's technical issues have been fixed for a long time. As the article says:
given the administration’s previous efforts to sabotage the insurance markets, it’s not paranoid to view this maintenance schedule as a plan to keep down enrollment
We're all technical folks here. What possible reason could there be to close the site every Sunday morning for 12 hours? In what situation would that ever be the best answer for users?
> We're all technical folks here. What possible reason could there be to close the site every Sunday morning for 12 hours? In what situation would that ever be the best answer for users?
What? No. You just bring-up a replica, compact that (or do any other slow operation you want), wait for it to catch-up, then fail-over, making the replica the new master.
There's no excuse for doing slow, blocking maintenance operations on a live user-facing database.
> There's no excuse for doing slow, blocking maintenance operations on a live user-facing database.
You can't judge a government system against the best-of-class standards of the private sector (note that I did not say the typical standards, just the top-of-line).
Government systems are rarely built or staffed by the top talent, often outsourced to lowest-bid contractors, and may not even have people on call for issues. It's not unusual for government sites to be scheduled to function only during standard business hours.
So yeah, it sucks that this site is going to be down during some of the least likely times for people to use it (midnight to noon on Sunday), but that's not at all unusual for government sites.
> Government systems are rarely built or staffed by the top talent, often outsourced to lowest-bid contractors, and may not even have people on call for issues. It's not unusual for government sites to be scheduled to function only during standard business hours.
Except that's not what happened, the Obama administration set up The US Digital Service to fix Healthcare.gov and specifically staffed it with top talent.
Can I just make a guess that you don't know anything about the Healthcare.gov infrastructure? It was extremely widely reported at the time how the Obama administration fired the first contractor (CGI Federal) for lack of quality, and how a number of people from the private sector came in to design it (sometimes at far below market rates).
Anyway, you're are making general platitudes about "government systems" without regard for the level, department, or any other factor that would influence the quality. And you think, "not unusual for government sites to be scheduled to function only during standard business hours". It would be a genuine shock to anyone knowledgeable about these sort of things if this were truly technically necessary.
> Can I just make a guess that you don't know anything about the Healthcare.gov infrastructure?
You can, but you couldn't be further from the truth.
> Anyway, you're are making general platitudes about "government systems" without regard for the level, department, or any other factor that would influence the quality.
I guess, but that's not really relevant to my point, which is to refute the statement "There's no excuse for doing slow, blocking maintenance operations on a live user-facing database.". There is an excuse in government systems for this. You may not like it, and you may think it's an excuse that they should avoid, but that's one reason, and these sorts of issues are pretty common in government systems.
You're correct that healthcare.gov is actually one of the better-functioning sites in that regard, but even if some aspects of it mitigate the shortcomings, that doesn't mean it's not subject to some of the other same limitations.
> It would be a genuine shock to anyone knowledgeable about these sort of things if this were truly technically necessary.
I'm emphasizing that what people on HN think of as "necessary" is different from what the government thinks of as "necessary". The example that I gave was that, if you don't have anyone on-call after hours (which is incredibly common), it makes sense to keep an important system offline during those hours, rather than risk a silent failure.
Eh, not to metacomment but I upvoted you. I don't know why you were downvoted. What about the healthcare.gov website do you know that would require this sort of downtime window, when it didn't require it in prior years?
When I say knowledgeable about these things, I mean these contracted, designed-to-spec, CMMI level 5 projects. Even one of those would be unlikely to require this sort of maintenance. Again, open to correct if you can offer specifics.
> We're all technical folks here. What possible reason could there be to close the site every Sunday morning for 12 hours?
Sometimes the reason isn't always purely technical. Especially in the government, there could be bureaucratic reasons that cause really un-intuitive behavior that's pretty much unheard of in the commercial world.
They could be exercising migrating to their COOP site and when asked "how long does it take for the site to FULLY come up?" the contractor said 12 hours to be on the safe side. If you account for things like acceptance testing done by government employees, 12 hours is unfortunate but not unreasonable for the government.
Planned maintenance at varying times, on varying dates, might be reasonable. Planned maintenance every Sunday (i.e., the weekend day that the largest number of people have off, and which would be the hardest off time to staff up for for planned maintenance), a little less so.
Wait, now we're complaining that the government is doing maintenance of some type outside of normal working hours? Talk about damned if you do, damned if you don't.
> No explanation was given for the outages, but given the administration’s previous efforts to sabotage the insurance markets, it’s not paranoid to view this maintenance schedule as a plan to keep down enrollment. At a minimum, it displays an indifference to the needs of the population it is supposed to be serving.
They have no regard for what is helpful to the common person. All they care about is dismantling what the previous administration did and pleasing special interests.
What you consider helpful to the common person is not what they consider helpful to the common person is not what someone else considers helpful to the common person.
Adding variables like time-frame, varying definitions of "common person" only make it more complicated.
People have beliefs about what is good for society at scale that no not align with yours. The current administration thinks that deregulation of industries and trade policies that protect domestic industry are the right thing to do. That belief may be misguided but it's what they believe.
Whether they're right or not is still to be determined. You never really get a good picture of that until the next administration takes office anyway.
> Whether they're right or not is still to be determined.
In what possible way? The free market has utterly failed healthcare. Costs are higher than ever, insurance is more expensive and covers less than ever, the #1 route to bankruptcy is a medical emergency, our doctors are offing themselves at record rates, and we have boatloads of people addicted to painkillers. Exactly what way has the market done anyone any good, other than the people on the top?
But if you're rich you have low wait times, so it's all good.
Frankly, Obamacare is a bit of a failure for some - I have friends who have "insurance plans" that don't actually cover anything without a deductible that they can't afford. Care for ongoing conditions is essentially not covered as a result.
The thing is... they'd have no healthcare at all under the previous system either, so going back to that isn't a solution for them.
The only real solution is single-payer healthcare paid out of taxes - unless you believe that somebody's right to live depends on how much money they make.
> You can dictate prices and have a private service model. See Medicare Advantage for a US based example of this that is fairly popular with consumers.
Medicare Advantage doesn't dictate prices. And indirectly, this is the entire reason that it it's popular (the service is substantially better).
> For additional services, but what about Part A and Part B? Seems that Advantage plans get similar pricing to traditional:
Parts A and B are Original Medicare. If you're on Medicare Advantage, it replaces Part A/B coverage.
(It's also a bit more complicated than that, because Medicare Advantage plans are provided by the same private insurers who cover non-Medicare patients, and they'll structure their agreements in such a way that the extra payments are hidden. For example, "for every Medicare Advantage patient of ours you see, we'll reimburse an additional X% for non-Medicare patient" - it's not literally like that or that explicit, but that's the shape of how it shakes out).
> You can dictate prices and have a private service model.
Perhaps, but at that point, you usually have a system where providers see their only way of making a profit as cutting costs - except those cuts are never passed onto the taxpayer. It's the worst of both worlds. NHS England has been forced into selling off parts of its operation in a similar manner to what you're suggesting for ideological reasons, and it's only cost us more money for worse service.
Perhaps it works where there's actually competition - e.g. where there's a choice of half a dozen providers owned by entirely separate entities that you could reasonably go to, which might be the case in some cities - but it doesn't work where there's few options for service users.
The ACA is far from perfect, though it doesn't help that the more aggressive components were kneecapped by Republicans so Republicans could bitch about the weak points for the next 7 years or so.
I'm a big believer in Single Payer, I mean come on, we have huge rolls of the public already on Medicare for a variety of reasons. Why not just put everyone on it and be done already?
> I mean come on, we have huge rolls of the public already on Medicare for a variety of reasons. Why not just put everyone on it and be done already?
1) Medicare isn't single-payer; it's a multipayer system
2) The government-run portion of Medicare underperforms the private payers in cost, medical outcomes, and patient satisfaction (the three primary axes on which success is measured).
If you want to point to data that suggests that a single-payer system would be successful in the US, Medicare is the last example you want to pick, because it suggests the exact opposite.
> The government-run portion of Medicare underperforms the private payers in cost, medical outcomes, and patient satisfaction (the three primary axes on which success is measured).
Would you be willing to cede that this is likely because their patient base grows constantly and yet their funding is cut at every turn? I'm not saying it's their only issue, but constantly being drained of resources while getting additional work to do makes it somewhat incredible that they operate as well as they do.
> Would you be willing to cede that this is likely because their patient base grows constantly and yet their funding is cut at every turn?
Medicare Advantage is growing faster than Medicare and operates on the same budget, so no, that's not an explanation for Original Medicare's shortcomings.
>And taking the thing down for 12 hours, 1/4 of every weekend could possibly be good in what way?
Some stakeholder somewhere probably refuses to go down during the business week for whatever reason. Maybe that stakeholder isn't even in government. Stuff like this is not uncommon in the world of enterprise.
I am usually the first to quote Hanlon's razor and attribute something like this to incompetence, but in this case it really hard not to see the malice in the actions taken.
I have family members that live paycheck to paycheck, have very little of their future saved/set-aside, and rely on government programs and subsidies far more than they certainly realize. Challenge them on ANYTHING regarding Trump and you'll get a never-ending earful. The number of people who are in the group of "Get rid of Obamacare!" + "I have insurance through the Affordable Care Act" is startling.
It absolutely shocks me the ignorance (literal use) of so many people when it comes to politics, particularly at the Federal level. Trump isn't the first time we've seen this, but it certainly feels like the most severe manifestation.
I don't know if it is related, but a certain three letter agency we do business with is having an IT freeze from 9/21 to 10/3 which means no changes of any kind to any production system. In over 10 years this is the first I've seen something like it.
it wouldn't surprise many if a FOIA analysis were to discover that this scheduled downtime were to coincide with the peak usage time for prior enrollments
Wow. This is so petty. They might hate it but I thought they would at least try to replace it and not hurt those that are using it. Shutting it down to reduce enrollment is beyond evil.
86 comments
[ 2.9 ms ] story [ 138 ms ] threadPerhaps likely, but it's not as obvious as your phrasing makes it out to be.
> To avoid confusion in, for example, an insurance certificate, it is always better to use the 24-hour clock, when 12:00 is 12 noon and, for example, 24:00 Sunday or 00:00 Monday both mean 12 midnight Sunday/Monday. It is common in transport timetables to use 23:59 Sunday or 00:01 Monday (in this example), or 11:59 p.m. or 12:01 a.m., to further reduce confusion.
> There are no standards established for the meaning of 12 a.m. and 12 p.m. It is often said that 12 a.m. Monday is midnight on Monday morning and 12 p.m. is midday.
http://www.npl.co.uk/reference/faqs/is-midnight-12-am-or-12-...
It wouldn't cost them anything to be more explicit.
We should all move to 24 hour time, yes, there's absolutely no disagreements from me there. AM/PM confusion is completely unnecessary.
However, 24 hour time is not used for anything in the US outside of a few industries (military, aviation, probably transportation). Not even train or bus schedules are given to the public in 24 hour time in the US (in my experience). If you used 24 hour time for anything at all while communicating with the general public you will confuse people. Especially the elderly. 00:00 is complete jibberish to the average American, so in this case they did the right thing given the audience.
We do some (paper) logging at work and a (larger than I imagined) percentage of the workforce (including most people older than 50%) were very, very confused by it. (Even though the instructions are the very simple "if it's after noon add 12 to the hour.")
And the more confusion, the fewer people will sign up, because confused people will plan their schedules around the wrong time.
It's a brilliantly executed dark pattern! You'd think Karl Rove was still calling the shots.
Here's the Wikipedia entry on the subject, which describes how 12am/12pm is actually a common confusion. https://en.wikipedia.org/wiki/12-hour_clock#Confusion_at_noo...
Incidentally, 99.9% is going to be a bad guess for almost any survey answer because of the "lizard people effect" (4% of US adults say that the country is controlled by lizard people, 7% are not sure).
The first time I worked outside the USA I was also pointed to that wikipedia article, but it in no way reflects reality on the streets.
Here's the wikipedia article, indicating that this is a common confusion: https://en.wikipedia.org/wiki/12-hour_clock#Confusion_at_noo...
Yes.
- Has the lack of this extreme and unheard-of level of "maintenance" downtime lead to prior problems large enough to make this a legitimate action rather than another disgusting attempt to sabotage the American healthcare system by making it harder for Americans to sign up?
I'm not sure how much churches have done in past years though. Some information about how the site has been used during the announced downtimes in previous years would be a courageous act of journalism. Maybe some hero will tweet it so that all the online journalists can toss something up.
>As part of its mission to translate the false narrative that the Affordable Care Act is failing into reality, Tom Price’s Health and Human Services Department has already announced massive cuts to the navigator programs that fund Obamacare outreach, as well as the budget for ads that alert Americans to the existence of the open-enrollment period.
It just seems like a strange coincidence they need to shutdown the site during open enrollment.
See https://www.theatlantic.com/technology/archive/2015/07/the-s...
given the administration’s previous efforts to sabotage the insurance markets, it’s not paranoid to view this maintenance schedule as a plan to keep down enrollment
We're all technical folks here. What possible reason could there be to close the site every Sunday morning for 12 hours? In what situation would that ever be the best answer for users?
Database compaction?
There's no excuse for doing slow, blocking maintenance operations on a live user-facing database.
You can't judge a government system against the best-of-class standards of the private sector (note that I did not say the typical standards, just the top-of-line).
Government systems are rarely built or staffed by the top talent, often outsourced to lowest-bid contractors, and may not even have people on call for issues. It's not unusual for government sites to be scheduled to function only during standard business hours.
So yeah, it sucks that this site is going to be down during some of the least likely times for people to use it (midnight to noon on Sunday), but that's not at all unusual for government sites.
Except that's not what happened, the Obama administration set up The US Digital Service to fix Healthcare.gov and specifically staffed it with top talent.
https://www.usds.gov/
Anyway, you're are making general platitudes about "government systems" without regard for the level, department, or any other factor that would influence the quality. And you think, "not unusual for government sites to be scheduled to function only during standard business hours". It would be a genuine shock to anyone knowledgeable about these sort of things if this were truly technically necessary.
You can, but you couldn't be further from the truth.
> Anyway, you're are making general platitudes about "government systems" without regard for the level, department, or any other factor that would influence the quality.
I guess, but that's not really relevant to my point, which is to refute the statement "There's no excuse for doing slow, blocking maintenance operations on a live user-facing database.". There is an excuse in government systems for this. You may not like it, and you may think it's an excuse that they should avoid, but that's one reason, and these sorts of issues are pretty common in government systems.
You're correct that healthcare.gov is actually one of the better-functioning sites in that regard, but even if some aspects of it mitigate the shortcomings, that doesn't mean it's not subject to some of the other same limitations.
> It would be a genuine shock to anyone knowledgeable about these sort of things if this were truly technically necessary.
I'm emphasizing that what people on HN think of as "necessary" is different from what the government thinks of as "necessary". The example that I gave was that, if you don't have anyone on-call after hours (which is incredibly common), it makes sense to keep an important system offline during those hours, rather than risk a silent failure.
When I say knowledgeable about these things, I mean these contracted, designed-to-spec, CMMI level 5 projects. Even one of those would be unlikely to require this sort of maintenance. Again, open to correct if you can offer specifics.
Quit lying to yourself and get some ethical standards.
Sometimes the reason isn't always purely technical. Especially in the government, there could be bureaucratic reasons that cause really un-intuitive behavior that's pretty much unheard of in the commercial world.
They could be exercising migrating to their COOP site and when asked "how long does it take for the site to FULLY come up?" the contractor said 12 hours to be on the safe side. If you account for things like acceptance testing done by government employees, 12 hours is unfortunate but not unreasonable for the government.
"Expected (planned) maintenance" outside of normal working hours, and during expected high volumes of usage = WTF.
From the article:
> No explanation was given for the outages, but given the administration’s previous efforts to sabotage the insurance markets, it’s not paranoid to view this maintenance schedule as a plan to keep down enrollment. At a minimum, it displays an indifference to the needs of the population it is supposed to be serving.
How does this administration hate the American people?
Adding variables like time-frame, varying definitions of "common person" only make it more complicated.
People have beliefs about what is good for society at scale that no not align with yours. The current administration thinks that deregulation of industries and trade policies that protect domestic industry are the right thing to do. That belief may be misguided but it's what they believe.
Whether they're right or not is still to be determined. You never really get a good picture of that until the next administration takes office anyway.
In what possible way? The free market has utterly failed healthcare. Costs are higher than ever, insurance is more expensive and covers less than ever, the #1 route to bankruptcy is a medical emergency, our doctors are offing themselves at record rates, and we have boatloads of people addicted to painkillers. Exactly what way has the market done anyone any good, other than the people on the top?
Frankly, Obamacare is a bit of a failure for some - I have friends who have "insurance plans" that don't actually cover anything without a deductible that they can't afford. Care for ongoing conditions is essentially not covered as a result.
The thing is... they'd have no healthcare at all under the previous system either, so going back to that isn't a solution for them.
The only real solution is single-payer healthcare paid out of taxes - unless you believe that somebody's right to live depends on how much money they make.
See Medicare Advantage for a US based example of this that is fairly popular with consumers.
Medicare Advantage doesn't dictate prices. And indirectly, this is the entire reason that it it's popular (the service is substantially better).
http://content.healthaffairs.org/content/34/8/1289.abstract
Parts A and B are Original Medicare. If you're on Medicare Advantage, it replaces Part A/B coverage.
(It's also a bit more complicated than that, because Medicare Advantage plans are provided by the same private insurers who cover non-Medicare patients, and they'll structure their agreements in such a way that the extra payments are hidden. For example, "for every Medicare Advantage patient of ours you see, we'll reimburse an additional X% for non-Medicare patient" - it's not literally like that or that explicit, but that's the shape of how it shakes out).
Perhaps, but at that point, you usually have a system where providers see their only way of making a profit as cutting costs - except those cuts are never passed onto the taxpayer. It's the worst of both worlds. NHS England has been forced into selling off parts of its operation in a similar manner to what you're suggesting for ideological reasons, and it's only cost us more money for worse service.
Perhaps it works where there's actually competition - e.g. where there's a choice of half a dozen providers owned by entirely separate entities that you could reasonably go to, which might be the case in some cities - but it doesn't work where there's few options for service users.
I'm a big believer in Single Payer, I mean come on, we have huge rolls of the public already on Medicare for a variety of reasons. Why not just put everyone on it and be done already?
1) Medicare isn't single-payer; it's a multipayer system
2) The government-run portion of Medicare underperforms the private payers in cost, medical outcomes, and patient satisfaction (the three primary axes on which success is measured).
If you want to point to data that suggests that a single-payer system would be successful in the US, Medicare is the last example you want to pick, because it suggests the exact opposite.
Would you be willing to cede that this is likely because their patient base grows constantly and yet their funding is cut at every turn? I'm not saying it's their only issue, but constantly being drained of resources while getting additional work to do makes it somewhat incredible that they operate as well as they do.
Medicare Advantage is growing faster than Medicare and operates on the same budget, so no, that's not an explanation for Original Medicare's shortcomings.
When was there a free market? Colonial times?
Are you thinking in context or just being contrary?
Some stakeholder somewhere probably refuses to go down during the business week for whatever reason. Maybe that stakeholder isn't even in government. Stuff like this is not uncommon in the world of enterprise.
When you combine this with calls by Trump to refuse to pay insurance companies (see https://www.reuters.com/article/us-usa-healthcare-trump/trum...), it sure looks malicious.
It absolutely shocks me the ignorance (literal use) of so many people when it comes to politics, particularly at the Federal level. Trump isn't the first time we've seen this, but it certainly feels like the most severe manifestation.
If healthcare.gov goes down, though...