I don't know how much, but that barely seems like enough to pay for the hardware to deal with 10+ million visitors in one evening. (Apparently they aren't allowed to use cloud infrastructure for data safety / privacy reasons.)
> that barely seems like enough to pay for the hardware to deal with 10+ million visitors in one evening.
Really?
The specs: The online census form was a pretty simple client side app. I'm going to say, I could whack something together in much less than 200k of JS.
For each household the site needs to:
1. Serve the client side app. The app had no images beside the ABS header. I'll be generous and say 200k of static assets including the client side JS.
2. Accept & validate the code households received on their postal forms
3. Save the final result (a single record) back to the databases. A single append operation. Being generous, 20k of JSON data per household.
When I filled in the census on Thursday night I noticed it made no effort to normalize addresses or anything else. Umless I'm missing something (or IBM turned off half the features after the DOS) the site was that simple.
1. 200k of static assets * 10 million households = 2TB of traffic. AWS would serve 2TB of traffic out of cloudfront for a whopping $200. If you don't trust 'the public cloud' your self-run data center could serve a tiny payload which downloaded the JS from cloudfront and verified its hash before running it. Or just serve all 2TB of traffic directly.
2. Accepting the unique code on the posted form is easy - just have a few servers which store the database of all posted codes in RAM and respond with 'yes' or 'no'. The codes for 10M households will fit in less than a gig of ram.
2. Saving the final result is pretty trivial. You need to accept 200 gigs of JSON data in an append-only journal. The peak write load would be around 4 million requests / hour, or cumulative ~1000 writes per second, or ~20MB/second of data. For an append-only store backed by an SSD this isn't much load at all. If the numbers weren't so low I'd suggest using a hand-written journal, but honestly most off the shelf databases could handle this write load without breaking a sweat. And you could easily shard it if you wanted to, and just aggregate back the data after its been collected. (You might want to actively geo-replicate the data immediately too, while its being collected just in case something happens).
In total, I'd feel comfortable running this entire site out of a few dedicated machines in just about any data center in the country. If you're feeling paranoid geo-replicate it, and actively stream received census submissions to a couple offsite backup locations. Total whiteboarding time: about 20 minutes. Might make a good interview question.
Of course IBM was also paid for consultancy time, and engineering time to design & implement the site itself, but even factoring in healthy margins I can't help but feel our government was ripped off by a factor of ~10-20x.
Even if it supported saving and coming back later, that barely complicates things. The form itself would need the ability to redisplay with fields pre-populated, which you'd want anyway if you've got any server-side validation or to handle submit errors with an option to retry.
For the database, you'd want to use caching. When the form is submitted you write the data locally and queue it for replication to a central server. To redisplay you'd try to read it locally and if you don't find it you'd read from the central server. You'd probably want to use the household's code for sharding so that you're going to the same local database each time.
I can't imagine it'd be necessary to spread the load out geographically. I'd have a single datacenter / cloud equivalent set up with load-balanced machines for the front end and the database tier, a full fail-over backup in case of critical failure of the main datacenter, and a separate database server for the master which the others replicate to and clone from.
This is a six-figure project, for sure, except maybe for time spent dealing with governmental red-tape.
To me it feels like you've left out an awful lot of stuff.
First off is project mgmt and coordination. On the government side there's going to be a small host of people involved, not just a single "customer". Meetings and more meetings. Somewhere between 50% and 100% of someone's time is going to be spent just dealing with them. And that's on top of the normal PM effort (so this could easily be 2 full time people).
You've also left off the generation of the mail outs. This means a data load, an id assignment, and a "print file" for the printer/mailer. High coordination costs (PM now dealing with the gov't and the print shop).
Security. Ouch. Need to make some effort to ensure the mail out codes can't be guessed easily... but are still small enough/readable enough that folks can use them (i.e. a GUID would be perfect except for the low usability). Further there ought to be some sort of post processing that attempts to ensure the data is good (i.e. no one played silly buggers and guessed codes).
On the application side "just in case something happens" isn't good enough. Once the incoming census data is accepted it must not be lost. The statistician would go ballistic (what do you mean you don't know how much data you lost?) and there's really no way to recover lost data without sending out new mailings.
On the back end is the final "give the stats guy the data" step. Conceptually just a giant CSV file... but how likely is that really? Also somewhere/somehow someone needs to build a list of households that have not completed the survey (and reporting on how many/what percentage and eventually producing lists of households for census staff to visit... which is probably handled internally by the existing systems... but it's another entire interface).
And all the design back and forth of how the website should look (and it has to be properly accessible). Not really hard to do on IBM's side... but the gov't side is going to be a nightmare of micromanaging UX specialists.
Oh and general security... it's a web app so there's all that. Hopefully the testing would be handled by a separate organization... which means another organization to coordinate with (PM is gonna be busy).
Plus... there's the upfront costs of the actual bidding process. Can't bill for it... but this does mean that every successful bid by IBM needs to be higher to cover these sunk costs (i.e. if IBM wins 20% of all its bids then the costs for 80% of the failed bids need to be covered by the winning 20%... if you see what I mean).
And now IBM needs to pay for some PR/damage control.
In the end I'm not so sure that $9 million is so very out of whack.
And damn it all if I didn't forget to include something for planing/supporting defending against denial of service attacks...
I agree. The price seems very low to me also, especially since they can't use a cloud provider. I'm assuming they are running this in some already established data center that they own and control.
Having to be in a DC alone makes them much more vulnerable to a DDoS (I remember our provider would just cut us off first and then ask questions later is they spotted bad traffic).
The funniest thing about the census debacle is that enough people mistook census for Sensis (a moderately well known brand that publishes our Yellow Pages) in large enough quantities that sensis.com.au completely fell over.
In my mind, this kind of fuck up seems consistent with IBM's (and other mega vendor) reputation when it comes to this kind of contract.
It's more reflective of poor procurement discipline than anything else -- whomever put out the bid let lobbyists or high level people do their thing without appropriate governance or penalty. It's a relatively small contract, and without clear penalties for poor performance this type of large contractor will do as little as possible.
I'd wager there were other complexities purely from the govt side. When healthcare.gov launched and crashed, it was uncovered that the final definition of the laws and rules were still coming weeks before the launch and had changed dozens of times throughout.
From a purely technical perspective, there's no way to know if something "works" if it hasn't been defined or - worse! - is constantly being redefined on the fly.
>I'd wager there were other complexities purely from the govt side.
That is true for oh so many government IT projects. The part that I still struggle to comprehend is "Why on earth aren't the IT companies, like IBM, telling governments that what they're asking isn't actually possible?"
The Danish government have an insane amount of failed IT projects. Most of them fails do to the complexity of laws, rules and requirements. We even know that they'll fail before they even start. Most good developers are able to predict this with great accuracy. So why aren't IBM and other telling governments that what they're asking isn't possible?
My best guess: Because their lawyers will ensure that they're paid anyway, and because they have absolutely no pride in their work.
You can't hurt the reputation of IBM, or at least it won't matter, because their competitors have equally poor reputation, and no one else is going to bid on the jobs they take on.
We must have had a dozen of these in the 1980s. An internal IBM one was I've Been Moved. Others I can still remember include Install Bigger Memories, It's Better Manually, Itty Bitty Machines, and Idiotic Bumbling Morons....
Until eventually their past customers get smart and stop working with them. Because of lowest bid contracts, the gov't will be an exception unfortunately.
> Because of lowest bid contracts, the gov't will be an exception unfortunately.
The problem is not lowest bid (which few government contracts are any more, AFAIK), but the fact that government contracting is its own domain of expertise (actually, often a separate domain for each state plus the federal government, and even many local governments) separate from the kind of work actually being contracted for, and there are in any field a very small number of players (compared to the field as a whole) that have developed expertise in the "winning government contracts" domain (and, in IT, IBM is one of them, at essentially every level of government.)
As someone who has worked with the US insurance industry, you cannot imagine the amount of crap they have to put up with from Congress.
Last minute changes to a law with a looming implementation deadline so that someone can score political points = everyone is non-compliant for some period
You can't rewrite 30-year old COBOL-backed business processes overnight.
""If you think about something like what goes on in Nauru, if a child is injured it's not the government that's responsible, it's Serco, Transfield or whoever else," she said, referring to the private contractors that run Australia's offshore detention centres."
It's all part of the 'Pacific Solution'[0], the main component of Australia's policy of nobody reaching the country without a visa (including asylum seekers).
Australia essentially pays Nauru to host an immigration detention centre, and sends asylum seekers who tried to reach Australia by boat there, regardless of age or legitimacy of claims or anything else.
FWIW, pretty much every country in the world has a policy of only allowing entry to people with visas. Where and how you acquire that visa varies, naturally, but requiring one is the default position.
Yes, plenty of countries let you turn up at the airport and obtain a visa on arrival - but if you look like you're moving in (no onward tickets, etc) you're at the very least going to have an interesting discussion, and maybe get a free flight back.
This particular situation is undeniably complex.
> ... sends asylum seekers who tried to reach Australia by
> boat there, regardless of age or legitimacy of claims
> or anything else.
Obviously all or most governments would rather determine legitimacy of claims prior to people arriving in-country. There are, clearly, massive and deplorable issues with the implementation of this system, but again, the principle is similar to how most nation states operate.
I think a big part of the problem is language. Like the life/choice descriptors used by people with strong opinions on abortion in the US -- hey, who doesn't love life and choice?
In AU one side will call these people asylum seekers, or refugees - the other will call them illegal immigrants. In many cases, both appellations are technically correct, but I think in almost all cases both appellations are non-helpful.
It is _not_ technically correct to call asylum seekers 'illegal immigrants'.
The 1951 UN Refugee Convention (Australia is a signatory) recognises that refugees have a lawful right to enter a country for the purposes of seeking asylum, regardless of how they arrive and whether or not they hold valid travel or identity documents.
A few years ago I worked on a project where my company was contracted to build a small piece of a much larger system by IBM. IBM was managing many different contractors. We needed a way to "securely" show a picture, aka not using the pre-installed image showing app. So IBM spent tens of thousands of dollars to build a new Android app that displayed a picture passed to it via an intent. I told our product guy that I could build it in an hour for free, but he said it wasn't our problem. In our twice-weekly status calls, we would ask about when the "image showing app" would arrive, and we heard all about the minutiae of building that app, from getting a PM, to finding a partner in India, to QA, to issues with the app, back to the contractor, and finally to us, where it was buggy as hell and never worked right. Completely mind boggling, and very enlightening about how IBM (and I'm sure other big vendors) operate.
Presumably that kind of process rigor is necessary in massive, high-scope projects, and the fallout we see is chiefly from the inability to apply a more lightweight process for tiny projects?
It's counter productive for big projects as well. The Too Big To Fail attitudes to process cover all manor of sins and corruptions to the point the projects is guaranteed to fail. See the Joint Strike Fighter for the worlds single most expensive example of this.
I'd buy that logic if it produced low defect rates and high uptime, but it doesn't. When we used SilverPop earlier this year it was down about 5% of the time and wrong 5% of the time, and even the basic way the interfaces we had to integrate with worked was a mess.
I don't think their process is heavy because it's rigorous, I think it's heavy because it's a reflection of their corporate structure (which is large).
This is hardly limited to IBM or software development. Almost any business that relies on human to human "sales" is at risk of devolving in to a deceptive mess of complexity things that have nothing to do with delivering the desired product or service. That is dangerous because eventually competitors will be lapping you at a tiny fraction of the price.
I don't disagree with the description "process theater", but its a common part of the overhead of enterprise contracting (often as a result of the demands of the customer, and often central to the compensation structure), and even a bigger part of the overhead of enterprise contracting that involves subcontracting.
"Securing revenue through enterprise development contracts" is a different discipline than "software development", and optimizing an organization to perform on the former can involve compromising on the latter. (The entities that understand software development well enough to align incentives of contractors well with effective delivery of value are usually the ones that don't need to contract development out in the first place.)
Process meant to dissect problem to the right size for predictable execution. Right size does not translate to always the same dissection for vastly different sized problems.
For a tool as simple as the parent post describes, what IBM does is not process rigor.
The problem is I've worked with companies with a lot of "process rigor" (as you phrase it), which in theory should result in bug-free quality software. But it doesn't.
The process rigor encourages checking stuff off of lists, but discourages the little ad-hoc fixes that make a product good. Nobody's going to fix the minor bug in the date entry field if it takes 4 hours of "process" to write the 4 minutes of code.
The goal of it is not to result in bug-free software, the goal is to result in completely traceable responsibility. It's to protect the company.
I work for IBM and believe me, the process and paperwork gets super tedious, but I understand why. The number of times I've had a client threaten to sue for breach of contract is staggering. If we didn't have a ton of process and paperwork showing a trail of evidence and every communication that led us to the final result, we'd be out of business so damn fast. Unfortunately some people really like to take unfair advantage of their vendors.
Let me give you an example. Last year I had a client who asked us to implement a solution and configure it according to their specifications. We did so, but they then realized that what they asked for was not what they actually wanted. The real event was more complicated, but for the sake of anonymity, let's say they wanted a weekly report as a bar graph, and when we delivered, they realized a pie chart made more sense. Now, being the good consultant I am, I switched the bar graph to a pie chart. It wasn't in the statement of work, but it only took a couple of hours and I like to keep my clients happy. After they got the pie chart, they said they liked the format but they wanted it switched from a weekly to a monthly report. Sure, it only takes a couple of hours. And then they wanted the monthly report, but the data resolution broken out into weeks instead of days. Only a couple of hours. And then they wanted the name of the report changed. A couple more hours. Bear in mind, this is all 100% custom development, from scratch.
Eventually we ran over budget and over time doing these minor tasks, because what the client asked for wasn't what they actually wanted. They demanded we keep going until they were happy. They hadn't signed off on this work being completed, so they claimed it wasn't done according to the contract.
A few weeks later we got a letter from their legal department, and in return we sent them the statement of work with everything checked off, and all of the emails we had sent back and forth between us and the client, showing where they acknowledged the scope was changing and we had said we would try our best, but couldn't guarantee anything other than what was in the SOW. We also showed where they refused to sign an extension of the project timeline and refused to sign a SOW change to formally add the new scope to the project. And that was the end of that story.
Every step of the way when something changed, even if we discussed it on the phone, we had a follow-up email to confirm. Everything we did got documented, every change was clearly spelled out as to what we changed and what was the original agreed-upon scope. Every scope change was approved by the PM and sometimes, one of my managers. And at the end of the project, we deliver all of this (sometimes hundreds of pages) in a nice, neat report detailing to the client what they hired us for and what we actually did. It takes forever and it doesn't guarantee the end result will be perfect. What it guarantees is that if we get sued, we have all the paperwork proving we did exactly what was agreed to in the contract.
> Last year I had a client who asked us to implement a solution and configure it according to their specifications. We did so, but they then realized that what they asked for was not what they actually wanted.
This is essentially always the pain point for "enterprise" software, and essentially always it's because the actual programmers communicate with the actual users through seven layers of middle management at both client and consultancy. Because middle management has to justify their existence, and are mortally afraid of being cut out of the information loop.
It's a fair point. I've certainly seen the sales team promise something that, as an engineer, I have no idea how I'm going to implement. Some days I wish I could join the sales folks when they're giving their pitch just to make sure expectations are kept in line (from both sides).
That's why I always give a little leeway if I can. If they want their report changed from monthly to weekly, I'll do it. But sometimes it becomes clear they're just trying to take advantage of me and get extra work done for free. After all, why would they pay for a bar graph AND a pie chart, with both of them being shown as monthly and weekly reports, when they could just pay for one and then say "oh that's not what we meant, can you change that?"
I certainly will change that, no problem. But only if I have hours left after I complete everything listed in the original SOW.
if you worked in india for next to nothing, would you see your clients as people who you wanted to help, or people who wanted to hurt others (others who charge fair prices for quality goods, and pay local taxes) to exploit you? exploiting them back is fair moral play. sell them shit and they'll turn around to hire you to fix the shit. fix the shit with more shit and you've got yourself a career.
I'm not defending IBM here, because god knows I've had my experiences with their juggernaut, but in this case you are not considering the big picture. If IBM had let you donate this component, there would be:
1) financial implications (you were paid to develop a solution. we, the customer, have discovered that you were donated time & code. did you disclose your potential status as receiver-of-charity during your initial bid? do we need to hold a recompete because other bidders did not know to use free labor in their bid? how much code was donated? does this reduce the average wage of developers on the project below minimum acceptable standards? can you prove it?)
2) product liability implications (this free code you used is not working properly. people are dying or becoming impoverished as a result. we are going to the donor-developer for doing this, his employer for allowing it, and you for failing to provide sufficiently rigorous integration testing.)
3) support implications (your free code broke, and nobody can get ahold of the rando who wrote it, and the company he worked for at the time disavows responsibility. everyone is going to get sued.)
4) due diligence implications (did you use this code because it was free or because it was correct? can you prove it? was this a cost-cutting measure that may have deprived me, the customer, of a superior product? can you prove it? don't answer me; answer my legal team.)
I know it sounds extreme, but I've seen each of these things occur, at differing scales, over the years, and the reason companies like IBM exist is to deliver defensible solutions. This is not startup-land; it is not sufficient (or even, often, necessary) to deliver a working product -- the product must be able to withstand a Congressional investigation. You don't have to prove it correct, but you do have to prove you adhered to legislative guidelines intended to control the process.
This is also the reason behind the headline regarding IBM's "reputation." Nobody at IBM gives a shit what any of us think about their technological prowess or even their project management skills. IBM's reputation is an ability to operate at a massive scale without getting nailed to the wall by some government or another. If they get their ears pinned back in this census situation, the takeaway is not "IBM tech sucks" or "IBM PM sucks" -- the takeaway is "IBM is not a safe company for politicians to hire"... and for an IBM, that is fatal.
Again, I agree the whole system by which IBM operates is disgusting and wasteful, but to compete in this space you must become this space, as it were.
I don't think you're wrong, it's just amusing that something so simple became such a gigantic mess when put through the IBM process. It's food for thought.
The big problem is that it's not beneficial for IBM/Oracle/Accenture/$consultancy to quickly deliver a stable solution that works well. They make much more money by having seven layers of middle management committees vote together a crappy solution that will need five years of support/upgrades and then a replacement.
In other industries, like construction, the same fundamental problem exists, but there human lives are at stake, standards bodies have been formed to prevent it, and non-software engineers aren't outsorced to people in India who have as little knowledge as they have pride in their workmanship.
If a contractor built apartment buildings the way IBM and friends build software, they'd be sued out of existence. I mean, have you ever heard of a building project going 3x over budget and then not actually delivering a building at all? It happens relatively often with software consultancies, usually on projects so trivial (like public transport ticketing systems) that it's hard to not attribute it to malice.
> usually on projects so trivial (like public transport ticketing systems) that it's hard to not attribute it to malice
Also something to consider in a circumstance like public transport stuff is that the multiple approval levels and meddling happens on the Government side as well. Like a perfect storm of bureaucratic bullshit that ends up making an enormous bloated mess that does everything but what was originally intended.
But rather than split hairs: I interpreted the underlying assertion being that construction is more predictable and reliable than software. I've heard this in other contexts as well. And that didn't pass the gut check for me, because I can think anecdotally of many construction projects (and home renovations) that were way over budget and took longer than expected.
An interesting thing to note that is that Australia selected IBM to run the digital Census (which, surprise, failed horribly) whilst one of their states, Queensland, has an explicit IBM ban due to an ongoing $1.2 billion dollar contract debacle[1].
As an Australian, I hope we can move far away from such contractors. The amount of waste is insane. It reminds me strongly of the initial Healthcare.gov contracting failures.
I think the Healthcare.gov comparison is interesting, because to my knowledge it's the exact opposite situation.
On Healthcare.gov, there was no primary contractor who had authority to manage the other contractors building various pieces. Solution: hire an IBM.
On the Australian census in the article, it seems like they only contracted IBM to both manage and build the project.
My takeaway is that you can get a decent result in large software projects, but (1) hire a party whose sole role it is to manage / conduct ongoing audits of work, (2) set the contract such that the managing party has a mildly adversarial relationship with the implementing parties, & (3) cover off cost overruns in the initial contract.
I may be wrong but from what I read the solution for Healthcare.gov didn't involve a primary contractor, it involved changing policies to reflect start-ups[1].
The Healthcare.gov debacle and resulting "start-up culture" resulted in the US Digital Service, which terms themselves as "a startup at the White House".
IBM's reputation is utterly nonsensical. I've worked for a company bought by IBM and for companies that used IBM's products/services and they've never been anything other than mediocre, sometimes outright bad, with costs that are astronomical.
My recommendation is that if you ever think about using an IBM product, hire a developer instead. You'll pay less money for a developer than you would pay for an IBM product, and after they're done implementing a the product better whatever unreliable enterprise clusterfuck IBM would shoehorn into your "solution", you have a developer who can work on other things.
From personal experience, CP Optimizer beats the crap out of other constraint solvers. Its API is... well, a bit rough, but it's incredibly fast and the support is top notch.
I can't really speak for other IBM products, but I suspect they generally align with your perception here.
I reckon. I've set up the Rational Suite and FileNet. FileNet alone is worse than a raw Oracle DB. You'd write FileNet SQL, only to work around bugs they have in transforming their SQL to Oracle SQL. Any manager buying IBM software should be in jail for corruption and breach of fiduciary duty.
I've dealt with a couple of customers of ours in Australia that have used IBM services for their IT. It's been a complete disaster. I don't know whether they are all that bad, or Australia gets the C team, but, wow.
And don't get me started on TCS. Fucking shitshow.
Where is Watson when you really need it. I would be much more impressed by "Watson successfully saves an IT project screwed up by IBM management" than an article about how Watson diagnoses cancer.
I started working at IBM ~3 months ago, as a DevOps Engineer. Joining IBM was probably the worst mistake that I have ever made. My group (IBM is a big company) is a complete cluster-f. No leadership. Managers have no knowledge of the technology that they are in charge of. Worst of all, we are forced to use IBM's proprietary, shitty tools (Notes, SameTime, Rational Team Concert, UrbanCode Deploy, IBM-specific JRE, Retain, ect, ect.). I started looking for a new job after two weeks. I would not buy anything from us.
"The detail of exactly what happened has not yet been independently reviewed. And as we saw in other stuff-ups, such as Queensland Health, when it was independently reviewed the blame was shared around quite a lot," Mr Noonan said. The attack was either a foreign or locally planned denial of service, or just a "large load" that appeared to be a denial of service but was in fact people trying to fill out the Census, he said. It "beggars belief" IBM's data centre could not handle a denial of service attack, he added."
There's been a bunch of discussion in Australia that there likely had been no "attack" and that it was simply just terrible planning around expected loads the night of the census.
90 comments
[ 2.9 ms ] story [ 169 ms ] threadReally?
The specs: The online census form was a pretty simple client side app. I'm going to say, I could whack something together in much less than 200k of JS.
For each household the site needs to:
1. Serve the client side app. The app had no images beside the ABS header. I'll be generous and say 200k of static assets including the client side JS.
2. Accept & validate the code households received on their postal forms
3. Save the final result (a single record) back to the databases. A single append operation. Being generous, 20k of JSON data per household.
When I filled in the census on Thursday night I noticed it made no effort to normalize addresses or anything else. Umless I'm missing something (or IBM turned off half the features after the DOS) the site was that simple.
1. 200k of static assets * 10 million households = 2TB of traffic. AWS would serve 2TB of traffic out of cloudfront for a whopping $200. If you don't trust 'the public cloud' your self-run data center could serve a tiny payload which downloaded the JS from cloudfront and verified its hash before running it. Or just serve all 2TB of traffic directly.
2. Accepting the unique code on the posted form is easy - just have a few servers which store the database of all posted codes in RAM and respond with 'yes' or 'no'. The codes for 10M households will fit in less than a gig of ram.
2. Saving the final result is pretty trivial. You need to accept 200 gigs of JSON data in an append-only journal. The peak write load would be around 4 million requests / hour, or cumulative ~1000 writes per second, or ~20MB/second of data. For an append-only store backed by an SSD this isn't much load at all. If the numbers weren't so low I'd suggest using a hand-written journal, but honestly most off the shelf databases could handle this write load without breaking a sweat. And you could easily shard it if you wanted to, and just aggregate back the data after its been collected. (You might want to actively geo-replicate the data immediately too, while its being collected just in case something happens).
In total, I'd feel comfortable running this entire site out of a few dedicated machines in just about any data center in the country. If you're feeling paranoid geo-replicate it, and actively stream received census submissions to a couple offsite backup locations. Total whiteboarding time: about 20 minutes. Might make a good interview question.
Of course IBM was also paid for consultancy time, and engineering time to design & implement the site itself, but even factoring in healthy margins I can't help but feel our government was ripped off by a factor of ~10-20x.
For the database, you'd want to use caching. When the form is submitted you write the data locally and queue it for replication to a central server. To redisplay you'd try to read it locally and if you don't find it you'd read from the central server. You'd probably want to use the household's code for sharding so that you're going to the same local database each time.
I can't imagine it'd be necessary to spread the load out geographically. I'd have a single datacenter / cloud equivalent set up with load-balanced machines for the front end and the database tier, a full fail-over backup in case of critical failure of the main datacenter, and a separate database server for the master which the others replicate to and clone from.
This is a six-figure project, for sure, except maybe for time spent dealing with governmental red-tape.
First off is project mgmt and coordination. On the government side there's going to be a small host of people involved, not just a single "customer". Meetings and more meetings. Somewhere between 50% and 100% of someone's time is going to be spent just dealing with them. And that's on top of the normal PM effort (so this could easily be 2 full time people).
You've also left off the generation of the mail outs. This means a data load, an id assignment, and a "print file" for the printer/mailer. High coordination costs (PM now dealing with the gov't and the print shop).
Security. Ouch. Need to make some effort to ensure the mail out codes can't be guessed easily... but are still small enough/readable enough that folks can use them (i.e. a GUID would be perfect except for the low usability). Further there ought to be some sort of post processing that attempts to ensure the data is good (i.e. no one played silly buggers and guessed codes).
On the application side "just in case something happens" isn't good enough. Once the incoming census data is accepted it must not be lost. The statistician would go ballistic (what do you mean you don't know how much data you lost?) and there's really no way to recover lost data without sending out new mailings.
On the back end is the final "give the stats guy the data" step. Conceptually just a giant CSV file... but how likely is that really? Also somewhere/somehow someone needs to build a list of households that have not completed the survey (and reporting on how many/what percentage and eventually producing lists of households for census staff to visit... which is probably handled internally by the existing systems... but it's another entire interface).
And all the design back and forth of how the website should look (and it has to be properly accessible). Not really hard to do on IBM's side... but the gov't side is going to be a nightmare of micromanaging UX specialists.
Oh and general security... it's a web app so there's all that. Hopefully the testing would be handled by a separate organization... which means another organization to coordinate with (PM is gonna be busy).
And of course all the QA you can stand.
Plus... there's legal costs (that contract ain't gonna review itself).
Plus... there's the upfront costs of the actual bidding process. Can't bill for it... but this does mean that every successful bid by IBM needs to be higher to cover these sunk costs (i.e. if IBM wins 20% of all its bids then the costs for 80% of the failed bids need to be covered by the winning 20%... if you see what I mean).
And now IBM needs to pay for some PR/damage control.
In the end I'm not so sure that $9 million is so very out of whack.
And damn it all if I didn't forget to include something for planing/supporting defending against denial of service attacks...
Having to be in a DC alone makes them much more vulnerable to a DDoS (I remember our provider would just cut us off first and then ask questions later is they spotted bad traffic).
https://en.wikipedia.org/wiki/SoftLayer
This happened in canada. https://www.google.ca/webhp?sourceid=chrome-instant&ion=1&es...
It's more reflective of poor procurement discipline than anything else -- whomever put out the bid let lobbyists or high level people do their thing without appropriate governance or penalty. It's a relatively small contract, and without clear penalties for poor performance this type of large contractor will do as little as possible.
From a purely technical perspective, there's no way to know if something "works" if it hasn't been defined or - worse! - is constantly being redefined on the fly.
That is true for oh so many government IT projects. The part that I still struggle to comprehend is "Why on earth aren't the IT companies, like IBM, telling governments that what they're asking isn't actually possible?"
The Danish government have an insane amount of failed IT projects. Most of them fails do to the complexity of laws, rules and requirements. We even know that they'll fail before they even start. Most good developers are able to predict this with great accuracy. So why aren't IBM and other telling governments that what they're asking isn't possible?
My best guess: Because their lawyers will ensure that they're paid anyway, and because they have absolutely no pride in their work.
You can't hurt the reputation of IBM, or at least it won't matter, because their competitors have equally poor reputation, and no one else is going to bid on the jobs they take on.
The problem is not lowest bid (which few government contracts are any more, AFAIK), but the fact that government contracting is its own domain of expertise (actually, often a separate domain for each state plus the federal government, and even many local governments) separate from the kind of work actually being contracted for, and there are in any field a very small number of players (compared to the field as a whole) that have developed expertise in the "winning government contracts" domain (and, in IT, IBM is one of them, at essentially every level of government.)
Last minute changes to a law with a looming implementation deadline so that someone can score political points = everyone is non-compliant for some period
You can't rewrite 30-year old COBOL-backed business processes overnight.
""If you think about something like what goes on in Nauru, if a child is injured it's not the government that's responsible, it's Serco, Transfield or whoever else," she said, referring to the private contractors that run Australia's offshore detention centres."
Private offshore detention centers for children?
What's going on down there?
http://www.abc.net.au/news/2016-07-25/four-corners-evidence-...
https://www.theguardian.com/australia-news/2016/jul/25/teena...
https://www.theguardian.com/australia-news/2016/jul/26/abu-g...
https://www.theguardian.com/australia-news/2016/aug/10/the-n...
http://www.npr.org/2016/08/11/489584342/claims-probed-of-bru...
https://en.wikipedia.org/wiki/Immigration_detention_in_Austr...
https://www.theguardian.com/world/2016/aug/10/a-short-histor...
Australia essentially pays Nauru to host an immigration detention centre, and sends asylum seekers who tried to reach Australia by boat there, regardless of age or legitimacy of claims or anything else.
[0] https://en.m.wikipedia.org/wiki/Pacific_Solution
Yes, plenty of countries let you turn up at the airport and obtain a visa on arrival - but if you look like you're moving in (no onward tickets, etc) you're at the very least going to have an interesting discussion, and maybe get a free flight back.
This particular situation is undeniably complex.
Obviously all or most governments would rather determine legitimacy of claims prior to people arriving in-country. There are, clearly, massive and deplorable issues with the implementation of this system, but again, the principle is similar to how most nation states operate.I think a big part of the problem is language. Like the life/choice descriptors used by people with strong opinions on abortion in the US -- hey, who doesn't love life and choice?
In AU one side will call these people asylum seekers, or refugees - the other will call them illegal immigrants. In many cases, both appellations are technically correct, but I think in almost all cases both appellations are non-helpful.
The 1951 UN Refugee Convention (Australia is a signatory) recognises that refugees have a lawful right to enter a country for the purposes of seeking asylum, regardless of how they arrive and whether or not they hold valid travel or identity documents.
http://www.asrc.org.au/wp-content/uploads/2013/07/it-not-ill...
I don't think their process is heavy because it's rigorous, I think it's heavy because it's a reflection of their corporate structure (which is large).
"Securing revenue through enterprise development contracts" is a different discipline than "software development", and optimizing an organization to perform on the former can involve compromising on the latter. (The entities that understand software development well enough to align incentives of contractors well with effective delivery of value are usually the ones that don't need to contract development out in the first place.)
Process meant to dissect problem to the right size for predictable execution. Right size does not translate to always the same dissection for vastly different sized problems.
For a tool as simple as the parent post describes, what IBM does is not process rigor.
The process rigor encourages checking stuff off of lists, but discourages the little ad-hoc fixes that make a product good. Nobody's going to fix the minor bug in the date entry field if it takes 4 hours of "process" to write the 4 minutes of code.
I work for IBM and believe me, the process and paperwork gets super tedious, but I understand why. The number of times I've had a client threaten to sue for breach of contract is staggering. If we didn't have a ton of process and paperwork showing a trail of evidence and every communication that led us to the final result, we'd be out of business so damn fast. Unfortunately some people really like to take unfair advantage of their vendors.
Let me give you an example. Last year I had a client who asked us to implement a solution and configure it according to their specifications. We did so, but they then realized that what they asked for was not what they actually wanted. The real event was more complicated, but for the sake of anonymity, let's say they wanted a weekly report as a bar graph, and when we delivered, they realized a pie chart made more sense. Now, being the good consultant I am, I switched the bar graph to a pie chart. It wasn't in the statement of work, but it only took a couple of hours and I like to keep my clients happy. After they got the pie chart, they said they liked the format but they wanted it switched from a weekly to a monthly report. Sure, it only takes a couple of hours. And then they wanted the monthly report, but the data resolution broken out into weeks instead of days. Only a couple of hours. And then they wanted the name of the report changed. A couple more hours. Bear in mind, this is all 100% custom development, from scratch.
Eventually we ran over budget and over time doing these minor tasks, because what the client asked for wasn't what they actually wanted. They demanded we keep going until they were happy. They hadn't signed off on this work being completed, so they claimed it wasn't done according to the contract.
A few weeks later we got a letter from their legal department, and in return we sent them the statement of work with everything checked off, and all of the emails we had sent back and forth between us and the client, showing where they acknowledged the scope was changing and we had said we would try our best, but couldn't guarantee anything other than what was in the SOW. We also showed where they refused to sign an extension of the project timeline and refused to sign a SOW change to formally add the new scope to the project. And that was the end of that story.
Every step of the way when something changed, even if we discussed it on the phone, we had a follow-up email to confirm. Everything we did got documented, every change was clearly spelled out as to what we changed and what was the original agreed-upon scope. Every scope change was approved by the PM and sometimes, one of my managers. And at the end of the project, we deliver all of this (sometimes hundreds of pages) in a nice, neat report detailing to the client what they hired us for and what we actually did. It takes forever and it doesn't guarantee the end result will be perfect. What it guarantees is that if we get sued, we have all the paperwork proving we did exactly what was agreed to in the contract.
This is essentially always the pain point for "enterprise" software, and essentially always it's because the actual programmers communicate with the actual users through seven layers of middle management at both client and consultancy. Because middle management has to justify their existence, and are mortally afraid of being cut out of the information loop.
That's why I always give a little leeway if I can. If they want their report changed from monthly to weekly, I'll do it. But sometimes it becomes clear they're just trying to take advantage of me and get extra work done for free. After all, why would they pay for a bar graph AND a pie chart, with both of them being shown as monthly and weekly reports, when they could just pay for one and then say "oh that's not what we meant, can you change that?"
I certainly will change that, no problem. But only if I have hours left after I complete everything listed in the original SOW.
if you worked in india for next to nothing, would you see your clients as people who you wanted to help, or people who wanted to hurt others (others who charge fair prices for quality goods, and pay local taxes) to exploit you? exploiting them back is fair moral play. sell them shit and they'll turn around to hire you to fix the shit. fix the shit with more shit and you've got yourself a career.
you're all idiots.
1) financial implications (you were paid to develop a solution. we, the customer, have discovered that you were donated time & code. did you disclose your potential status as receiver-of-charity during your initial bid? do we need to hold a recompete because other bidders did not know to use free labor in their bid? how much code was donated? does this reduce the average wage of developers on the project below minimum acceptable standards? can you prove it?)
2) product liability implications (this free code you used is not working properly. people are dying or becoming impoverished as a result. we are going to the donor-developer for doing this, his employer for allowing it, and you for failing to provide sufficiently rigorous integration testing.)
3) support implications (your free code broke, and nobody can get ahold of the rando who wrote it, and the company he worked for at the time disavows responsibility. everyone is going to get sued.)
4) due diligence implications (did you use this code because it was free or because it was correct? can you prove it? was this a cost-cutting measure that may have deprived me, the customer, of a superior product? can you prove it? don't answer me; answer my legal team.)
I know it sounds extreme, but I've seen each of these things occur, at differing scales, over the years, and the reason companies like IBM exist is to deliver defensible solutions. This is not startup-land; it is not sufficient (or even, often, necessary) to deliver a working product -- the product must be able to withstand a Congressional investigation. You don't have to prove it correct, but you do have to prove you adhered to legislative guidelines intended to control the process.
This is also the reason behind the headline regarding IBM's "reputation." Nobody at IBM gives a shit what any of us think about their technological prowess or even their project management skills. IBM's reputation is an ability to operate at a massive scale without getting nailed to the wall by some government or another. If they get their ears pinned back in this census situation, the takeaway is not "IBM tech sucks" or "IBM PM sucks" -- the takeaway is "IBM is not a safe company for politicians to hire"... and for an IBM, that is fatal.
Again, I agree the whole system by which IBM operates is disgusting and wasteful, but to compete in this space you must become this space, as it were.
The big problem is that it's not beneficial for IBM/Oracle/Accenture/$consultancy to quickly deliver a stable solution that works well. They make much more money by having seven layers of middle management committees vote together a crappy solution that will need five years of support/upgrades and then a replacement.
In other industries, like construction, the same fundamental problem exists, but there human lives are at stake, standards bodies have been formed to prevent it, and non-software engineers aren't outsorced to people in India who have as little knowledge as they have pride in their workmanship.
If a contractor built apartment buildings the way IBM and friends build software, they'd be sued out of existence. I mean, have you ever heard of a building project going 3x over budget and then not actually delivering a building at all? It happens relatively often with software consultancies, usually on projects so trivial (like public transport ticketing systems) that it's hard to not attribute it to malice.
Also something to consider in a circumstance like public transport stuff is that the multiple approval levels and meddling happens on the Government side as well. Like a perfect storm of bureaucratic bullshit that ends up making an enormous bloated mess that does everything but what was originally intended.
Construction projects go over budget and are late all the time. Remember the big dig? https://en.wikipedia.org/wiki/Big_Dig
The Sydney Opera house was 10 years late and 1475% over budget: https://en.wikipedia.org/wiki/Sydney_Opera_House#Completion_...
Anyone who's had renovations done to their house can also probably sympathize.
Private construction? Not so much, certainly nothing like 3x the estimated cost.
https://en.wikipedia.org/wiki/Plaza_Rakyat
http://gizmodo.com/the-sad-fate-of-the-worlds-six-tallest-un...
But rather than split hairs: I interpreted the underlying assertion being that construction is more predictable and reliable than software. I've heard this in other contexts as well. And that didn't pass the gut check for me, because I can think anecdotally of many construction projects (and home renovations) that were way over budget and took longer than expected.
As an Australian, I hope we can move far away from such contractors. The amount of waste is insane. It reminds me strongly of the initial Healthcare.gov contracting failures.
[1]: http://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-4...
On Healthcare.gov, there was no primary contractor who had authority to manage the other contractors building various pieces. Solution: hire an IBM.
On the Australian census in the article, it seems like they only contracted IBM to both manage and build the project.
My takeaway is that you can get a decent result in large software projects, but (1) hire a party whose sole role it is to manage / conduct ongoing audits of work, (2) set the contract such that the managing party has a mildly adversarial relationship with the implementing parties, & (3) cover off cost overruns in the initial contract.
The Healthcare.gov debacle and resulting "start-up culture" resulted in the US Digital Service, which terms themselves as "a startup at the White House".
http://www.theatlantic.com/technology/archive/2015/07/the-se...
http://www.ibmandtheholocaust.com/
My recommendation is that if you ever think about using an IBM product, hire a developer instead. You'll pay less money for a developer than you would pay for an IBM product, and after they're done implementing a the product better whatever unreliable enterprise clusterfuck IBM would shoehorn into your "solution", you have a developer who can work on other things.
[0]: https://www.youtube.com/watch?v=lx0z9FjxP-Y
The only "problem" with developers is they usually don't throw golf days for their best customers :)
I can't really speak for other IBM products, but I suspect they generally align with your perception here.
Whoever suggests using it should be fired on the spot.
If I see any mention of it on a person's CV (because I only mention what I am ok with), is enough for me to second-guess this person's ability
I'm so glad git is replacing it.
I found this headline kind of amazing because I was like, "reputation at risk? This has been my impression of their reputation for decades."
Corp sector in Australia is on the tail end ridding themselves of IBM nonsense after a decade of utter incompetence.
Unfortunately the gov sector catches on slowly.
Sadly a lot of places don't learn though and just replace IBM with TCS/TechM which is somehow even worse.
And don't get me started on TCS. Fucking shitshow.
Too soon?
I miss this IBM.
There's been a bunch of discussion in Australia that there likely had been no "attack" and that it was simply just terrible planning around expected loads the night of the census.