That's a pretty hefty fine, but I wonder what the original billings were for developing the system in the first place?
I've heard reports that they spent >AUD$5Million on just the load testing aspect, but I hope someone can verify that. And I trust that they have improved comprehensive testing methodologies that actually work to improve reliability under duress. (<sarcasm>Tip: Perhaps don't build it using Lotus Notes??</sarcasm>)
I guess we can be thankful that is was 'merely' a census and not something like electronic voting!
In the US, Census data is used for redistricting purposes, which CAN influence election outcomes in the future - so the implications of a census may be wider than you think. I don't know exactly the way it works in Australia, but some googling has led me to believe they use a system that is similar.
Absolutely, I didn't mean to downplay the importance of a national census. It is just that the results are used for longer term project planning and policy making, so the effects aren't as immediate as a whole government inadvertently being kicked out or brought into power as a result of an election day debacle.
Another major problem with the census was the Australian Bureau of Statistics holding onto the data and changing their privacy policies which meant that in an act of protest a ton of people entered false information or ommitted details. A really poor census overall.
Australia - or more specifically - already have a known-vulnerable electronic voting system they used in an election, so that horse has already bolted.
All our votes are carried out on paper. If whatever scheme they're using for collation is compromised/suspect a recount could be done entirely by hand.
Your counterpart is probably thinking of NSW's iVote, intended for people who cannot otherwise vote at a normal polling booth, which was trialled in 2015.
It was embarrassingly insecure.
(Doesn't change the annoying niggle of conflating a state with an entire country, but I guess Americans are all Alabamans in that regard)
I'm not sure it is a hefty fine for the situation. Its simply paying the remaining costs for IBM's failures.
I don't see how IBM thought they could pull if off for $9.6 million [0], especially with the ABS requirements for information handling during the census.
"Key measures to safeguard information include strong encryption of data, restricted access on a need-to-know basis and monitoring of all staff, including regular audits. After data collection and processing, the ABS removes names and addresses from other personal and household information. Names and addresses will be stored securely and separate from one another. No one working with Census data will be able to view your personal information (name or address) at the same time as your other Census responses (such as age, sex, occupation, level of education or income). Stored separately and securely, individuals names will also be substituted with a linkage key, a computer generated code, completely anonymising the personal information. Only these anonymous linkage keys will be used by the ABS to bring data sets together." [1]
Currently, it seems IBM is blaming their subcontractors, "While IBM issued a mea culpa, they were then quick to lay the blame at the feet of Australian communications company NextGen Networks and its upstream provider Vocus, blaming them for failing to implement a geoblocking service called Island Australia to prevent traffic reaching the site from outside Australia." [2]
And considering IBM's has a revenue stream of $20.24 billion this quarter [3], the fine won't really impact them.
As to their reputation, IBM doesn't have a brilliant name already:
* Failure to meet deadlines [4]
* In Queensland they were overbudget, overtime, and under scope [5] resulting in their permanent ban from working with the Queensland government.
* They have a great habit of building things that they call incredibly fast, but never perform up to spec in the real world [6].
Anecdotally, National Mutual contracted out to IBM for some "supercomputers" for their COBOL applications, to speed up processing time. The machines were half the speed IBM claimed. IBM's solution? Double the number of machines delivered, and ignore the extra costs of power, maintenance and network load.
Stories of IBM versus state or national governments come up fairly regularly. This one is unusual that IBM was actually found guilty/culpable/whatever. Usually they weasel out on some contractual loop hole.
Availability and load balancing aside, did any other Australian citizen (who actually managed to complete the census at all) notice that the UX was quite ordinary? Reminded me of a web form circa 1990.
For instance, IIRC the census asked for my and my wife's country of birth and nationality, but later on when filling in the info for our children, it asked for the nationality of the both parents again. Simple logic checking back through the data should have shown that our (the parent's) information was already supplied, and this information for our children could be pre-filled or skipped and auto filled in the database.
Just a few small things that would have made the filling out of the forms a lot smoother, faster, and less complicated for lay people who were already nervous about doing the data entry.
Except you may have step-children, adopted children, foster children, children in the house not part of your family (friends of your child, children of your friends), etc. Keep in mind that the census needs to include everyone who was in the house at the time it was filled out, not everyone who is part of your regular abode/family situation.
You might argue that it would work for the "most common scenario". In which case why not pre-fill all the data to the statistically most common values? (Hint: the census's purpose is to collect those statistics.)
Also, pre-filling might cause confusion as to which fields have already been filled out by the user; if a value is present, they might wrongly assume no action is required on their part.
(Having said all that, I haven't seen the online census.)
UX wise, a radio button that hides that section labelled "same as John Doe" might have given that flexibility without the pain of rewriting a dozen things again.
Sure. I did consider the scenario if the kids had a friend sleeping over that night or something, or if the respondents had adoptive or foster children etc., but I think when I traced back the line of questioning in my case there seemed to be a logical thread link there that could/should have negated asking for the same information again.
I am relying on my memory here from few months ago, so I could be wrong, but I did feel at several points during the session that quite a few questions could quite possibly be chained together in better fashion. Perhaps not so much a failing of the actual site UX designer, but the census analysts who designed the questions in the first place?
Not responsible for this census, but I've done work with it in the past. Two observations:
The questions are largely derivative and the same each year: cost to develop and test while maintaining statistical validity and capture of all possible scenarios, while still being interpreted correctly by the population, is enormous.
Secondly, the number of permutations and combinations of answers is almost universally underestimated by everyone I've ever met outside of this process. Nationality is not the same as birthplace is not the same as parent is not the same as adults in the household. The amount of diversity is staggering.
Additionally, although those two are the primary reasons for the question designs (and I assure you, people will be doing deductions and logical inferences on the back-end once the data is collected), being exhaustive at the point of data entry rather than deductive allows for further meta analysis on the data entry process and survey responding experience itself, whereas if you decide to attempt to deduce, you miss out on a dimension of typo, misskey, misinterpretation, and other analysis data that you would forgo in the former case.
Thanks for your viewpoint from the 'sausage factory', so to speak. I am beginning to understand the enormity of trying to combine ease of use, with the effort to gather statistically valid data.
I thought the presentation was very good. I was pleasantly surprised that the design was professional and modern looking.
It seemed to be a single page application because once you got started it was fast to get through the process. I suspect the SPA nature was a deliberate decision to offload workload from the servers. Imagine how much more traffic would been needed if every page/section on the form required another call to the server.
See my other comment further down, but I just thought of another issue again:
Design consideration has to be given to how/what people will be using to access said questions.
If it looks like a web form from the 1990s , ask yourself what kind of technology some Australians might be using to try to fill out their census form, and how standard we have to try to keep that interface/experience.
We requested paper forms long before census night, because we were pretty sure this was going to happen. Didn't put our names or any other personally identifying info on them, either. (Never received a fine.)
Before census day they were predicting two thirds of the 10 million Australian households to complete the census online. They stated their site had been tested with half a million concurrent users and was able to sustain double that.
Unfortunately no one seems to have considered that the majority of users would do it after dinner (say 6-10pm). Take the 6.6 million users divide it by the time frame and it was never going to work with a 1 million per hour capacity.
What is interesting is with IBM paying out they must have failed to deliver the required capacity. I thought it more likely the government had not tendered it properly and requested a lower capacity than needed but obviously IBM just tried to cut costs and under resource the operation.
I noticed the exact same thing. Read about their load testing in the paper, did a quick back of the envelope, and concluded before the census that they hadn't texted big enough.
Which brings us back to another question: was there even a DOS attack? I've seen any little technical details on the attacks, and I know myself, like a lot of other Australians logged on behind vpns. When the site wouldn't load, of course we hit refresh...
There likely was no DDoS. This was an early guess based on the symptoms at the time, but after investigating IBM never claimed DDoS in the days following census night.
No there was no DDoS. Multiple Network Ops groups at the time said they saw no malicious traffic the whole night, either domestically or from international sources.
The report calls the DoS an "event" rather than an "attack", carefully worded so that tech journalists would pick up on the wrong thing, but clear that it was not malicious intentional traffic.
The DoS was the Australian public attempting to use an online government service.
I often wonder why companies and governments even trust large organisations like IBM over individuals they've interviewed themselves. I mean, it's not like IBM's consultant's competence is any secret. IBM is also trying to make it worse, or at least, not making the competence factor any priority. Rather, it's cost over everything:
I mean how do you get away with hiring IBM (or accenture, PwC, didata, deloitte or, may God help us, Tata or something like that, where cost of paycheck is literally the only factor in hiring).
Such individuals are usually unable to detect real signal in the thing they're hiring for, so interviewing in and of itself doesn't provide them any qualitative information.
They also know the people judging them are in the same boat.
Thus they fall back to the only thing understandable to themselves and people who will judge them: big brand and prestige/shininess. If the project succeeds, you take credit. If it fails, you can't be blamed for "buying the biggest and the best" (even if the biggest and the best are just a bunch of stary-eyed ignorant grads with a salesman/project manager at the head).
/that's the good version. The bad version points out how many of these firms gets business through networks, personal connections and "you scratch my back I'll scratch yours" type dealings.
//it's not a rant if it's true.
Liability. If the project fails, it is not your fault.
I cynically suspect that whoever drafts these contracts might even believe that by agreeing on an excessively expensive failure clause, failure will thus not occur — after all, it is legally covered.
We see this in The Netherlands as well (and I suspect it is a global phenomenon). Government needs software, big-name consultancy firm gets the job, builds stuff that won't work or barely functions, project implodes or is delivered in a half-baked state. Repeat ad nauseam. These companies (Cap Gemini comes to mind) thrive by these projects, even though most seem to fail or end up underwhelming. There appears to be some degree of corruption there on the side of these contractors — investigative journalism has uncovered several questionable practices.
Some part of my wonders if having a well-funded government department dedicated to these projects that creates and maintains the (free!) software needed is not a more tenable (and cheaper) approach. On the other hand, large government-driven software projects may just be impossible to deliver at this point; we simply lack the know-how to prevent the inevitable implosion of such complex projects.
My only disappointment is that they didn't have to pay more for their incompetence. IBM has been on a mission to replace their workforce with cheap labour from India for quite a while. I'm glad the work they deliver to their clients is finally being exposed for what it is - junk code with no quality control.
Just wait until IBM's Phoenix Pay system and the scandal here in Canada comes to a head. With 300,000 civil servants waiting six months to be paid and it's still not fixed.
There must be a big penalty looming for such a massive level of incompetence.
Phoenix seems like a rather awkward name for an application.
>Though many government employees work in 9-to-5 office jobs, the public payroll also includes a wide array of people like Mr. Ryan with complicated work schedules and pay rules.
My father used to be in the Coast Guard he worked 28 days on and had 28 days off. If you were injured that meant the days you were off were really two days?? Try explain that to some bureaucrat in Ottawa who can thinks they are speaking English (really French), they're 2,000 miles away and doesn't understand the system either.
As someone who has negotiated similar contracts with providers like IBM, I'm willing to bet that someone on the customer side gave into the pressure to rush the lawyers while IBM did a nice job of generating loophole-filled statements of work, and I wouldn't hold my breath for that penalty.
Oh God, that sounds an awful lot like two more Australian systems: the Queensland Health payroll debacle, where a bunch of health workers were paid incorrectly or not at all. Or the National Disability Insurance Scheme debacle, where healthcare providers haven't been paid, resulting in small business owners having to take out huge amounts of personal debt just to pay their employees, and sole traders unable to provide services their clients need to live day to day while still keeping a roof over their own heads. (They're both IBM projects too.)
You have to understand that what StackOverflow gets away with in terms of hardware is not at all reflective of what other shops will be able to do.
SO can work on such a small amount of hardware because they've got people who either know the entire infrastructure from ground up, or have the skillset and interest to figure it out. Plus they have very tight coordination between the folks doing the development, and running it day to day (possibly because it's the same people).
Your average developer working on a government contract through $BIGNAMESERVICESPROVIDER is almost certainly not going to have that skillset or motivation to learn it. Even in the unlikely scenario that they did, they're not going to be in a situation where they can act on it. It'll be a case of developing to specification, throwing stuff over a wall to some sort of QA, and then throwing the result of that over to someone to run it.
The whole thing will be run by project managers who have opex budgets to meet, on infrastructure that was specified well before the thing was ever built.
43 comments
[ 5.4 ms ] story [ 123 ms ] threadI've heard reports that they spent >AUD$5Million on just the load testing aspect, but I hope someone can verify that. And I trust that they have improved comprehensive testing methodologies that actually work to improve reliability under duress. (<sarcasm>Tip: Perhaps don't build it using Lotus Notes??</sarcasm>)
I guess we can be thankful that is was 'merely' a census and not something like electronic voting!
See: http://www.smh.com.au/business/ibms-reputation-at-risk-in-wa...
For as much as I've heard about this census issue, I thought it was a bigger (monetary) deal.
https://en.wikipedia.org/wiki/Redistribution_(Australia)
It was embarrassingly insecure.
(Doesn't change the annoying niggle of conflating a state with an entire country, but I guess Americans are all Alabamans in that regard)
I don't see how IBM thought they could pull if off for $9.6 million [0], especially with the ABS requirements for information handling during the census.
"Key measures to safeguard information include strong encryption of data, restricted access on a need-to-know basis and monitoring of all staff, including regular audits. After data collection and processing, the ABS removes names and addresses from other personal and household information. Names and addresses will be stored securely and separate from one another. No one working with Census data will be able to view your personal information (name or address) at the same time as your other Census responses (such as age, sex, occupation, level of education or income). Stored separately and securely, individuals names will also be substituted with a linkage key, a computer generated code, completely anonymising the personal information. Only these anonymous linkage keys will be used by the ABS to bring data sets together." [1]
Currently, it seems IBM is blaming their subcontractors, "While IBM issued a mea culpa, they were then quick to lay the blame at the feet of Australian communications company NextGen Networks and its upstream provider Vocus, blaming them for failing to implement a geoblocking service called Island Australia to prevent traffic reaching the site from outside Australia." [2]
And considering IBM's has a revenue stream of $20.24 billion this quarter [3], the fine won't really impact them.
As to their reputation, IBM doesn't have a brilliant name already:
* Failure to meet deadlines [4]
* In Queensland they were overbudget, overtime, and under scope [5] resulting in their permanent ban from working with the Queensland government.
* They have a great habit of building things that they call incredibly fast, but never perform up to spec in the real world [6].
Anecdotally, National Mutual contracted out to IBM for some "supercomputers" for their COBOL applications, to speed up processing time. The machines were half the speed IBM claimed. IBM's solution? Double the number of machines delivered, and ignore the extra costs of power, maintenance and network load.
[0] http://www.smh.com.au/business/ibms-reputation-at-risk-in-wa...
[1] http://www.abs.gov.au/websitedbs/censushome.nsf/home/privacy
[2] http://www.news.com.au/technology/online/security/ibm-faces-...
[3] http://www.businessinsider.com.au/ibm-surprises-with-q2-2016...
[4] http://www.abc.net.au/news/2016-09-27/ibm-unlikely-to-hit-cu...
[5] http://www.healthpayrollinquiry.qld.gov.au/__data/assets/pdf...
[6] https://en.wikipedia.org...
For instance, IIRC the census asked for my and my wife's country of birth and nationality, but later on when filling in the info for our children, it asked for the nationality of the both parents again. Simple logic checking back through the data should have shown that our (the parent's) information was already supplied, and this information for our children could be pre-filled or skipped and auto filled in the database.
Just a few small things that would have made the filling out of the forms a lot smoother, faster, and less complicated for lay people who were already nervous about doing the data entry.
You might argue that it would work for the "most common scenario". In which case why not pre-fill all the data to the statistically most common values? (Hint: the census's purpose is to collect those statistics.)
Also, pre-filling might cause confusion as to which fields have already been filled out by the user; if a value is present, they might wrongly assume no action is required on their part.
(Having said all that, I haven't seen the online census.)
I am relying on my memory here from few months ago, so I could be wrong, but I did feel at several points during the session that quite a few questions could quite possibly be chained together in better fashion. Perhaps not so much a failing of the actual site UX designer, but the census analysts who designed the questions in the first place?
The questions are largely derivative and the same each year: cost to develop and test while maintaining statistical validity and capture of all possible scenarios, while still being interpreted correctly by the population, is enormous.
Secondly, the number of permutations and combinations of answers is almost universally underestimated by everyone I've ever met outside of this process. Nationality is not the same as birthplace is not the same as parent is not the same as adults in the household. The amount of diversity is staggering.
Additionally, although those two are the primary reasons for the question designs (and I assure you, people will be doing deductions and logical inferences on the back-end once the data is collected), being exhaustive at the point of data entry rather than deductive allows for further meta analysis on the data entry process and survey responding experience itself, whereas if you decide to attempt to deduce, you miss out on a dimension of typo, misskey, misinterpretation, and other analysis data that you would forgo in the former case.
It seemed to be a single page application because once you got started it was fast to get through the process. I suspect the SPA nature was a deliberate decision to offload workload from the servers. Imagine how much more traffic would been needed if every page/section on the form required another call to the server.
Design consideration has to be given to how/what people will be using to access said questions.
If it looks like a web form from the 1990s , ask yourself what kind of technology some Australians might be using to try to fill out their census form, and how standard we have to try to keep that interface/experience.
Because it's bloody hard to DDoS paper forms. Widely distributed atoms are curiously disinterested in IP packets.
Unfortunately no one seems to have considered that the majority of users would do it after dinner (say 6-10pm). Take the 6.6 million users divide it by the time frame and it was never going to work with a 1 million per hour capacity.
What is interesting is with IBM paying out they must have failed to deliver the required capacity. I thought it more likely the government had not tendered it properly and requested a lower capacity than needed but obviously IBM just tried to cut costs and under resource the operation.
Which brings us back to another question: was there even a DOS attack? I've seen any little technical details on the attacks, and I know myself, like a lot of other Australians logged on behind vpns. When the site wouldn't load, of course we hit refresh...
The report calls the DoS an "event" rather than an "attack", carefully worded so that tech journalists would pick up on the wrong thing, but clear that it was not malicious intentional traffic.
The DoS was the Australian public attempting to use an online government service.
https://www.righttoknow.org.au/request/abs_preparedness_of_2...
http://www.businessinsider.com.au/ibm-layoffs-1-month-severa...
I mean how do you get away with hiring IBM (or accenture, PwC, didata, deloitte or, may God help us, Tata or something like that, where cost of paycheck is literally the only factor in hiring).
They also know the people judging them are in the same boat.
Thus they fall back to the only thing understandable to themselves and people who will judge them: big brand and prestige/shininess. If the project succeeds, you take credit. If it fails, you can't be blamed for "buying the biggest and the best" (even if the biggest and the best are just a bunch of stary-eyed ignorant grads with a salesman/project manager at the head).
/that's the good version. The bad version points out how many of these firms gets business through networks, personal connections and "you scratch my back I'll scratch yours" type dealings. //it's not a rant if it's true.
I cynically suspect that whoever drafts these contracts might even believe that by agreeing on an excessively expensive failure clause, failure will thus not occur — after all, it is legally covered.
We see this in The Netherlands as well (and I suspect it is a global phenomenon). Government needs software, big-name consultancy firm gets the job, builds stuff that won't work or barely functions, project implodes or is delivered in a half-baked state. Repeat ad nauseam. These companies (Cap Gemini comes to mind) thrive by these projects, even though most seem to fail or end up underwhelming. There appears to be some degree of corruption there on the side of these contractors — investigative journalism has uncovered several questionable practices.
Some part of my wonders if having a well-funded government department dedicated to these projects that creates and maintains the (free!) software needed is not a more tenable (and cheaper) approach. On the other hand, large government-driven software projects may just be impossible to deliver at this point; we simply lack the know-how to prevent the inevitable implosion of such complex projects.
There must be a big penalty looming for such a massive level of incompetence.
Phoenix seems like a rather awkward name for an application.
http://www.nytimes.com/2016/11/18/world/canada/government-wo...
>Though many government employees work in 9-to-5 office jobs, the public payroll also includes a wide array of people like Mr. Ryan with complicated work schedules and pay rules.
My father used to be in the Coast Guard he worked 28 days on and had 28 days off. If you were injured that meant the days you were off were really two days?? Try explain that to some bureaucrat in Ottawa who can thinks they are speaking English (really French), they're 2,000 miles away and doesn't understand the system either.
Based on non-cloud setups like http://nickcraver.com/blog/2016/02/17/stack-overflow-the-arc...
A rack or two would have been able to handle the load.
SO can work on such a small amount of hardware because they've got people who either know the entire infrastructure from ground up, or have the skillset and interest to figure it out. Plus they have very tight coordination between the folks doing the development, and running it day to day (possibly because it's the same people).
Your average developer working on a government contract through $BIGNAMESERVICESPROVIDER is almost certainly not going to have that skillset or motivation to learn it. Even in the unlikely scenario that they did, they're not going to be in a situation where they can act on it. It'll be a case of developing to specification, throwing stuff over a wall to some sort of QA, and then throwing the result of that over to someone to run it. The whole thing will be run by project managers who have opex budgets to meet, on infrastructure that was specified well before the thing was ever built.
However it was all non-cloud self-hosted hardware.
It says the damage was estimated at 30 million. It says IBM paid something. It didn't explicitly say IBM paid $30 million.
"Computer giant IBM will pay more than $30 million in compensation for its role in the bungled census"