Seems like a baseless article that is pulling fake facts out of thin air to supplement a couple of real ones like the cost basis and failures of large government projects.
It's not uncommon for large government projects to fail with all the bureaucracy and politics that is in play. I am not saying IBM is great, but this article is just not worth it.
I've used two services from IBM: Notes and node DB2 package. Both of them were pretty low quality, the db2 package was so unstable that it constantly segfaulted the entire node process. It's not fixed for years since the issue was reported with reproducible code. Needless to say I wouldn't recommend IBM even to my enemies.
Can anyone share a good IBM experience? I'm honestly interested.
They bought SoftLayer a few years ago and haven't ruined it. Not sure if that really counts or not, but I was really worried. I've been there for quite a few years, and nothing went downhill after they got bought.
Not the person you are replying to but I'd say no.
As the supplier you should back yourself as to whether its likely that your changes will cause an issue or not. If so, issue a warning, otherwise don't.
The reason I say this is not for your customer's benefit but for yours. I've seen people get all knotted up with process because someone had the bright idea that the customer should be notified when absolutely anything changes behind the scenes - just in case. This abundance of caution will make your life a misery, and stops any rapid movement. It also desensitizes the customer, who gets used to your notifications and eventually decides they can ignore them.
Those appear to be products - which is different than hiring IBM to develop something for you. IBM has had a storied history in tech from developing/popularizing the PC to OS/2 to their thinkpad laptops, to designing systems for the space shuttle, and natural language projects like the Watson. I don't think IBM lacks talent.
Its almost seems like when you can get away with milking the system, nobody seems to have any moral qualms in selling the government a $200 screwdriver.
>Unfortunately IBMs history doesn't really provide any evidence of their current competency.
I gave current examples of competency. Its trivial to find others. In any case, my point was merely to differenciate services from products, so I'm not sure what you were replying to anyway.
I'll share another bad experience. AstraZeneca pulled out of a $1.4 billion/7-year deal with IBM a couple of years early, because of dissatisfaction with IBM's service.
I remember hearing about it then, but I cannot now find details of the problems.
> Can anyone share a good IBM experience? I'm honestly interested.
The IBM pc we bought in the 90ies was fantastic.
It had working suspend resume many years ahead of other computers and came with 365/24 free phone support for a year (and the next years was about 50USD). This being was very useful since I didn't have Internet access.
The support team had unlimited time, patience and knowledge and would walk me (15-16 y.o) through any procedure including if necessary: backup, reconfigure disk with fdisk, restore os from backup etc etc. I learned a whole lot from them both technical stuff as well as how to deal with customers ;-)
(And yes, old IBM ThinkPads were very popular IIRC.)
All of the big consultancy firms are bodyshops in some sense. But in my experience IBM is truly one of the worst. And people still keep falling for it.
Does anyone else remember the days when "No one ever got fired for buying IBM" was true? You might pay a little extra, but it was never a total fail to use IBM products.
I'm not sure when that changed, but I feel like these days, IBM is closer to Oracle than AWS.
IBM has sold off most of their divisions - hardware to Lenovo, storage to Hitachi, etc. They no longer actually produce anything. So 'buying' IBM is no longer a real concept - they're pure consultants now, and consultants never 'sell' a product, just help you implement someone else's. A lovely legal distinction that they seem all too happy to abuse.
This is not true. IBM produces a ton of software (spanning a range from terrible to excellent, quality-wise). This is where much of their profit comes from now. They still produce a bunch of hardware, mostly at the higher (and legacy) end.
Clearly too many people remember. Shows the power of branding. IBM has largely moved out of the consumer space, and occasionally their research labs come out with some cool new tech (Watson and such). In the common consciousness they were the Apple of their day who then ascended out of the market to bigger, cooler things. As a result most people, including non-technical business execs who make purchasing decisions, don't have a good current frame of reference. IBM rode that facade for a while, it's only recently that their blunders have started hurting their reputation within the industry.
I do, but it's important to remember that the saying "no one ever got fired for buying IBM" was not intended as a testament to the quality of IBM's work. It was intended as a testament to the irrelevance of the quality of IBM's work.
"No one ever got fired for buying IBM" because IBM was the safe, established choice. If you bought IBM and the project ended up failing, nobody would blame the failure on your decision, because buying IBM was what you were expected to do. If you bought from some other vendor and the project ended up failing, however, people would rush to claim that the failure was your fault for not doing the safe thing and buying IBM. In other words, you bought IBM to protect yourself from appearing to have taken a risk, not because you expected them to do a particularly good job.
If TFA is accurate, it would appear that "no one ever got fired for buying IBM" is as true today as it ever was. If the Canadian government had hired the Freshbooks guy to build their new payroll system, and the project had failed, the person who had made that decision would have had a whole lot of explaining to do. That person bought IBM, though, so everyone will just shrug their shoulders and assume the fault for the failure lies elsewhere.
> Does anyone else remember the days when "No one ever got fired for buying IBM" was true? You might pay a little extra, but it was never a total fail to use IBM products.
I never though of it that way but rather as a CYA move.
Late 1980's. I was there, working for a skunkworks fragment in a Big Blue shoppe. To express doubt about IBM was to get no bonus. Then they started to haemorrhage cash to the tune of 5B$/quarter.
Really it's that Oracle is closer, these days, to what IBM was 30 years ago.
I will copy my comment from the last thread on this project:
"IBM is a multifaceted company.
These kinds of projects fall under what used to be called IBM Global Services https://en.wikipedia.org/wiki/IBM_Global_Services. It could be compared somewhat to EDS (HP), Accenture, Perot Systems (Dell) etc. I've never heard of over-delivery from any of these kinds of outsourcing arrangements, they always seem so obviously destined for boondoggle.
IBM proper, the one that makes mainframes and POWER and DB2 and a ton of operating systems and storage etc is very much a technology company. Some of their best products have the worst sales and marketing efforts. I'm working directly with the senior leadership of the POWER group right now and there are no salesmen in sight.. the technology will either sell itself or not. When we met in person the first time the GM told me "we can build any kind of computer you want" - meaning microarchitecture changes, SERDES configuration, new board layout, sheet metal, OS, application tweaks. Not a lot of companies can do that. There is hubris, less technology, and lack of technical value at FANG or most startups or whatever your benchmark is in comparison.
IBM has been accumulating a persona among my cadre (developers who consult for companies that are bootstrapping a new tech division on top of a profitable business).
The problem is that having a boondoggle division reflects on the whole company. When I'm making decisions for a client, I never even think about what IBM solutions I could apply to a problem.
I've only had a few projects using IBM products, admittedly most very old legacy systems -- but those introductions failed at just basic dev hygiene things. Documentation behind paywalls, couldn't get things to run on my dev machine without very experimental projects made by individuals - last updated in 2003. Just frustrating. I don't like advocating complete rewrites, I've happily updated a company's internal tool that was written in as3. Could have squeezed more hours out of them if they wanted to go with html5, but doing so didn't open up any new possibilities they wanted so I didn't.
If your company is running on RPG, and it's just a CRUD app -- I'm going to save you money by recommending a rewrite under the two circumstances I've encountered in the wild.
Honestly, this isn't their fault. It's awesome that their products have lasted so long for companies. I don't expect a company to keep up to date documentation / interoperable interpreters for 30 year deprecated tech. IBM's just so old that they've accumulated many layers of fossilized products, and those things have caused an unfair bias in my mind (and probably the minds of others)
I'm sure that there are problems IBM research is uniquely qualified to solve, my perception is that they're the wrong choice for most problems. I'm pretty thankful for companies like them. (IBM, Siemens, SAS etc) Their expensive ERP systems have enabled me to raise my rate over the years off the information asynchrony that forms the foundation of their success.
I worked there in the late 1990's. I was in research division (writing Notes applications) and they were doing better after the downturn in the early 90s. They spent billions on research (some hardware research like chips/ some software research).
The little nub that think pads use was invented in that building.
Global Services (the consulting arm) was the rising division.
The talk inside was the new CEO (Gerstner[1]) was leveraging IBM's size rather than the previous plan to break it into pieces and was somewhat successful at it.
I left to go back to grad school. And I think the new management went back to the "this is too big and unmanageable, its worth more broken up" mindset.
I interpreted that remark as a commentary on humans preference for failing in a conventional manner being perfectly acceptable, versus possibly failing in an unconventional one.
(I also grew up professionally in the 2000s, well after IBM was widely considered elite by the tech nerd community.)
Then again, it's Oracle. I'm sure their licensing department is attempting to track how many neurons in the brain are actively engaged when using their products and billing the customer accordingly.
Yeah, I don't know much about Slovakia but the author's broad, unsupported generalizations of the country (Slovaks are very talented at mid-level service jobs: responsible and polite if not particularly fast-moving... Slovaks, particularly in the service industry, are astonishingly honest) don't inspire a lot of confidence. I was also surprised to see the author's use of "the Ukraine". I still remember someone at my college newspaper using that archaic phrase in an otherwise innocuous article and getting a very angry letter-to-the-editor about the history/politics of Russia and Ukraine, and that was more than a decade ago.
Is there any proof that the technical work for IBM is sent out to Indian boiler rooms with very high turnover? Or is this just based on some belief that any software in India must automatically be shitty?
A while back i was in a hostel in England. I met someone who claimed they were an IBM consultant. After testing his technical prowess verbally there were a few holes but he was very apt. He seemed legit or he was able to fool me. I always wondered what an IBM consultant is doing in a hostel unless he is from India and he wanted to keep his cost down. England is pretty expensive. That is what i was doing. This doesn't mean he is ineffective at his job. It just means there was an Indian person claiming to be an IBM consultant while staying in a hostel. :)
I worked for a firm that sold IBM products. We had a horde of developers in India. Is that the proof you want? Some teams there was particularly great. Other's not so much. Turnover though was EXTREMELY high.
I wouldn't agree with the high turnover part. Its as much as in other similar sized companies.
Its just that most of my colleagues hate the software made by IBM, its slow, a pain to use and has a tendency to make things very very complicated. You cannot build great stuff with mediocre tools.
The IT industry has moved on to leaner and simpler tools and platforms. IBM, atleast the GBS group, is basically a sales company.
Note : I worked in IBM GBS for a year before I left for more interesting and rewarding jobs. This was in 2008 though.
Well, this news that came out last year is somewhat relevant. MD Anderson Cancer Center's IBM Watson project failed, with $62 million paid to IBM and PwC for essentially no results. However, I don't blame IBM, I blame people who dove in without clearly thinking it through. https://www.healthnewsreview.org/2017/02/md-anderson-cancer-...
We have all seen a project go wonky with requirements. However, I haven't heard of a successful product that IBM built/managed in the last decade.
On the other hand, talking with friends in the government and government contracting, they argue that IBM and the major contractors have perfected the art of exploiting government into maximizing billable hours at the expense of results.
It's a bit like the metaphorical malicious genie interpreting your request. It isn't possible to write specifications that can't be interpreted malevolently.
IBM is likely the single largest government contractor in North America. Most governments at every level have contracted IBM for something, at some point. You don't hear about the successes because that's not news. That's just a government and contractor functioning as they should. Only large, expensive failures make the news.
It's not just IBM. Dutch government IT projects also have a tendency to fail big, and those are handled by other companies. It seems governments are easy marks for mediocre IT companies.
From my experience it's mainly that governments don't have a lot of experts on staff so a lot of decision making gets outsourced too. I worked on a gov contract a long time ago and it was pretty obvious that there were several higher ups in government who added their own conflicting requirements to the list. Everybody in the trenches saw that they made things incredibly complex for no good reason but nobody pushed back. The main contractor was happy to sell many, many more hours and inside government there was nobody who had the ability to screen requirements for consistency and feasibility before giving them to the contractor.
It's like dealing with any other service business like contracts, lawyers, wedding photographers or others. If you don't give them clear guidance the result will most likely be an expensive disaster.
Another problem with governments is that they're made up of dozens or hundreds of different organisations, and they all make these mistakes on their own. Even if one organisation gets it right, other parts of the government don't benefit from that expertise.
I think governments should have a special organisation that just maintains the expertise for handling these sort of big projects all across the government.
That organisation would have as its only job to make sure all other government organisations get this right. It should be able to work when that's their only responsibility.
These stories of failure, but skew the reader simply because there are very few companies that can bid on and attempt to manage projects of this scale. This article could easily be written to focus on the failure of sweeping, enormous, poorly-thought out projects that suffer changing visions and scope, rotating project managers, and evolving systems ... that IBM happens to bid on and execute poorly.
Small projects are better focused, cost less, are easier to understand, and therefore succeed more frequently (or fail more silently). It's big projects, frequently proposed and conceived by governments or enormous industrial conglomerates that are poorly thought-out, improperly managed, and suffer the worst of project management incompetence or hubris/excess. But who bids on that kind of project? Big companies like IBM. Let's be fair, they don't just bid on them, they also coax them into being, but my point stands: if you want to fail big, you've got to dream big. This isn't a defense of IBM, who deserves to own the shame of talking big but being unable to actually deliver. But it is a reminder that the project designers get equal blame for these sweeping, grandiose, visionary catastrophes.
Sure. It's both to blame. I know plenty of small companies in other industries that refuse to do work that they expect to fail because they don't want to hurt their reputations. Those guys all probably didn't bid or dropped out when they saw it would fail.
There's a dimension left out here on the Phoenix disaster: It wasn't really IBM's fault at all.
The Canadian public service is really complex. There are multiple unions with multiple overlapping collective bargaining agreements, where the public service is allocated to different classes. These classes are paid specific rates, with retroactive pay being common for changing classes. The majority of the problems with Phoenix have been employees moving from their classes and being paid the correct amount. It has also adversely affected non-unionized positions.
My understanding is that the Harper government (Prime Minister until 2015), who was responsible for the negotiation and for laying out the requirements, was trying to save a money and not responibly create a pay system. Two major factors that jump out at me:
1) Requirements did not call for training. The system was implemented and IBM was not required to train any operators on how the system functions, which is important because all new staff were hired to run the system and,
2) Due to the need for cost saving measures (the government was trying really hard to balance the budget, as the election was coming up), the previous payroll staff were terminated and a new payroll centre was opened in Miramichi, which is a small town in the middle of nowhere.
So, on top of new software, the government lost all of its institutional knowledge regarding payroll and how things are supposed to work. It's actually hard to say how much of this is IBM's fault and how much is the governments, because the government doesn't know how to fix it. No one knows how Phoenix works and no one knows how it is supposed to work. It's just a big mess with no end in sight.
Could IBM have done a better job? Probably, but garbage in, garbage out.
As someone who works in same space, I can say this is happening because unrealistic expectations of customers are not challenged by bidding Vendors because they fear losing the project. This in turns happens because customer tries to outsmart vendors by pressing them for reducing the timelines to absolute impossible. If any vendor tries to play by the rules, they are thrown out. Smart vendors respond by crafting the contract smarty. Skipping training is one such example. It was crucial but ....
In the end project is either canceled or goes over budged or barely manage to deliver anything.
Yup, this kind of contracts game is what really needs modern innovation. I think the space is kind of living in an extensions of 70's assumptions with many decades of beauracratic rules added on to try to fix up fundamental problem. For another factor, it would be lower risk to allocate into many smaller projects and sign contracts in phases and sub-systems - but then the funding of the total project becomes more at risk politically, and then there is a higher technical load on the contracting dept to architect the boundaries.
They are barely able to state the outcome of project so expecting them to do the necessary legwork to break projects into logical subprojects is too much to ask.
No disrespect but we have come to a point where IT have started appearing magical to pointy hairy bosses who think juse because it's software everything is infinitely malleable with no impact on quality, cost, or time.
If you don't have an organization that can break it down to a smaller projects then there is a question if doing it as one large project has a higher or lower chance of failure.
Shouldn't that depend upon complexity of change but who cares. CIO have a short tenure so collecting the bonuses and leaving before house of card collapses works.
It's all about managing the personal bonus and career, who cares for the success 9f project?
It depends on complexity, but I think it's more of a situation akin to refactoring or rewriting a complex code base. A total rewrite is always an attractive thought, but a real focus on continuous refactoring in smaller portion at a time is often lower risk, lower cost, and faster in the end - and I suspect it's the same with large bureaucratic processes too. It's also a way to sharpen the skills of the organization in small steps with lower risks for failure (and improving future project steps...).
Customers love to be lied to; in a recent set of presentations I was favoring the vendor who was presenting reality but the winner chosen by the business evaluators was the vendor who brought in the glossiest presentation and glossed over the complexities and the frightening part expressed a comfort with ever changing requirements. They were put off by the vendors who boasted about zero change requests and making their timelines but liked the vendor who proposed a fantasy timeline with no promises.
A responsible vendor then declines to take the project (smart decision) or executes anyway. We've been in the situation a few times where what started as an attractive project with a reasonable budget turned out to be an underestimate and we barely earned back our salary expenditures on the project.
Those companies did continue to do business with us though and we've normally earned back our losses and more on future projects.
That's what honest vendors do. That's not what IBM did.
They most likely did raise this, but when asked to provide a more "competitive" bid, they axed training and who knows what else. This is common in large enterprise sales deals. For professional services deployment engagements, there is always the possibility of an amendment to the services scope (often referred to as a change order or change request).
If the project is estimated to take over a year to deploy, by then you're in a new budget cycle and the vendor has come to an agreement with the procurement department to charge for additional services as required in subsequent phases.
Another nice thing about governments is that they do not deal in monetary damages but in media fallout- so something loud (collapse of a billion dollar project) is worser to them, then silently going over budget because additional points appear on the list.
So biding yourself into a project to low gives you enough leverage when the point of no return is reached to still make a big plus.
This is why it's not accurate to claim that "it's not IBM's fault." They are fully culpable for making a bid that didn't include necessary components of the project priced in.
Exactly. Based on their track record, IBM clearly knew what was going to happen and even anticipated the extra income as the project spiralled around the drain.
I concur. What I'm hearing is that the Canadian Federal payroll is extremely complex (like all government payrolls, I suppose). The Harper gov't justified the system by laying off everyone who knew the old payroll system.
The software developed by IBM was so complex, that it required the expertise of the people who got laid off to operate it. The novices who took over didn't know how to make the software work, and chaos ensued.
The Canadian government is as much at fault as the developers on this one. (I speak as one who is paying for this fiasco :o).
IBM has failed to build these state level systems successfully around the world (Canada, Australia, Pennsylvania, Slovenia, New Zealand). IBM should know how to go about building them right or IBM should not be in this business.
The first $200 million of cost overruns I'm willing to pin on the Government of Canada. The subsequent billion dollars and the failure of the system to work at all is the vendor's fault.
If the Government of Canada is entirely impossible to work with, the vendor should have shut the project down at 200% of budget rather than run it to six times budget and still fail.
basically, it's a political game... the same jobs they "lost" in miramichi because of the firearms registry legal change, they tried to get back. Same pork barrel politics that congressmen in the US play with trying to bring jobs to rural districts.
> There's a dimension left out here on the Phoenix disaster: It wasn't really IBM's fault at all.
I respectfully disagree. IBM management either knew what they were contracting was a recipe for disaster, or they were incompetent. In both cases, as they were supposed to be the billion-dollar expert on the matter, they bear great responsibility for this failure.
And at an industry level, companies that promise the impossible push out of the market more honest ones, and they deserve all the bad PR IBM is getting on this one.
you're both right, but you're a bit more right. IBM should have been pointing out that losing the institutional knowledge may cause problems. they possibly assumed some of their milestones and deliverables would be while working with existing staff. if the existing staff is let go, many of the assumptions in the original estimate are just wrong at that point.
The managers needed IBM to tell them about the concept of institutional knowledge? They need to be told that people with experience have learned their jobs? That new employees need training? This is like Management 101. Just how incompetent are these managers?
That's a trick question, there are no competent managers in government service for the most part. They either burn out from managing without the ability to recruit competent staff, or they moved elsewhere to make drastically more money.
I'm sitting with a 4x multiple on my old public service job.
They're the ones spending the money. It's not "oh, we're getting paid? Caveat emptor." Everyone involved has a social responsibility, but the ones who are getting paid to be there have more responsibility.
Haven't read the docs, so I don't know what the contracts and proposals states, but regardless of how obvious some things are, putting that in writing is a bit of a CYA.
"We're estimating based on having access and participation from your senior/experiences payroll employees. Substantial changes in personnel will impact the viability of the project."
So incompetent they hired someone else to write software only they could possibly know how to write.
There is a degree of management bankruptcy implicit in a large proportion of outsourcing projects. Hiring a vendor to handle billing for your public records dept makes sense, because you can’t afford to maintain a stable of experts on billing.
But a lot of the skill brought in by good contracing firms is in extracting and negotiating requirements. Because your management team has no idea how the work actually gets done.
Think about that. It is cheaper to have some other company come in and learn your problem domain than it is for your company to write down clearly what it is they do and figure out a set of steps to attract employees to do the work for them.
Personally, I think that has Management 101 written all over it in large block letters.
Yeah, with that price tag IBM could have rolled out multiple federated payroll systems and then iterated on the theme until they had a unified system, but they would not have been able to farm out the development to off-site contractors.
Any vendor will do exactly what is asked, no matter what. IBM, Dell, Accenture, Deloitte, McKinsey, etc. There is no reason why a customer should allow a contractor like this to run roughshod over them -- thats incompetence and bad governance.
I've run big projects with vendors like this. You write good RFPs and hold feet to the fire and they will deliver.
> You write good RFPs and hold feet to the fire and they will deliver.
That's the key bit though. Its not easy to write good RFPs. Unless done to an excruciating level of precision (which they very seldom are), you leave wiggle room which the vendor will naturally take advantage of when things get tough.
I've written RFP responses in the telecom industry. "Excruciating" is about right. Thing is, if you are not highly detailed you'll fall between the stools of waterfall and agile. It sounds like this project went astray that way. Not sufficiently spec'ed, everybody gets everything they want, and not agile. You cannot make a sensible kanban board out of change orders.
Not to mention government procurement processes do everything in their power to make it difficult to write a good RFP and the major players are highly skilled at responding to RFPs for the purpose of winning the bid, not necessarily to deliver on the RFP's requirements.
I've worked in government and commercial sectors. Big procurement organizations are a pain in any situation. Government adds some wackiness because of transparency and personal liability that procurement officers are living with.
But end of the day, if the people writing technical requirements have the time and know WTF they are talking about, it works. If you hire <vendor a> to write an RFP that <vendor b-y> responds to and <vendor z> evaluates, how could the result not be a shitshow? (Btw, that happened in a Fortune 500 org!)
In the construction business, there’s no need to state that nails get hammered or screws turned, unless there is a very specific application. But they tend to structure projects in a way where there is an incentive for success and punishment for failure. That's not to say that it's perfect, but there are far more $10M+ failed IT projects than failed construction projects.
If you work in this industry, you've seen your share of IT vendors delivering barely literate offshore workers through 2-5 layers of contracting pimps. That simply doesn't happen with pipefitters and others.
In IT, the tenure of the CIO other senior leadership is often is lower than the vendor and the project. The incentive for the CIO is to listen to <insert vendor here> because they may communicate what is happening better AND they may want a job with the vendor later. Without the threat of reprisal, many organizations will make poor decisions (both vendor and customer).
I've seen that pattern several times, often enough that it's a bit of a red flag as it seems management know their own weakness but don't want to be exposed to it.
Exactly. Based on IBM's track record around the world (I know of IBM payroll projects for Canada, Australia, Pennsylvania which have failed while New Zealand and Slovenia have been mentioned in comments), I'd say failing on low ball government payroll projects and then taking the government to the cleaners is strategic at IBM.
IBM have made billions doing so. Bad publicity is just a good start.
The really interesting point is that if this business plan is shown to be strategic (if I were Canada, Australia or Pennsylvania I'd be looking at full discovery of all email records), IBM would then be vulnerable to civil suit for fraud.
I agree that buyers typically deceives themselves due to their limited understanding of highly complex large systems, and that their incompetence is amplified by a sub-optimal procurement process.
It's obvious that there was zero actual knowledge involved related to implementing large systems - just by the fact that someone thought it was a good idea to take a behemoth system excreted out of Oracle, and then pass it through the corporate body of IBM, before inserting it into a huge government organisation, while at the same time getting rid of any process expertise and business knowledge they might actually have on-board.
However - I don't think it's probable that any amount of training could save the situation to any measurable degree whatsoever.
Is IBM to blame? Well, I think they knew what would happen.
I think if Harper had stayed in office, he would have blamed this on the complicated payroll requirements. He likely would have leaned on PSAC/CUPE to simplify it in the next round of contract negotiations.
As someone who used to work in/on/underneath websphere. I can tell you I am loathe to even consider anything IBM. I was almost recruited by them and then I recalled how miserable I was on one particular project. The money was good but I said no. I recommend everyone I mean to stay away from IBM and Oracle as best I can. I just don't think the line between open source and enterprise is crystal clear anymore.
As a general rule, if a work task lands on your desk that involves working with a IBM/Oracle/SAP product, preemptively apologize to those regularly around you for your miserable attitude in the near future.
When The Daily WTF was still maintained it reserved an entire section of the site for Oracle Database. It may rarely have catastrophic failures (although it's the second one I've seen most catastrophic failures, just after MySQL), but it is composed of one unreasonable choice after the other.
Having worked at a company that used an Oracle cluster considering of over £1 million in hardware alone (I was never brave enough to ask about the licensing for the actual DB), the complexity of the setup far outweighs anything I could get my head around, and the company employed several full-time DBAs who were constantly tweaking parameters to keep the system performing - it could not simply be left to its own devices, there were humans in the loop around the clock.
Despite this, there was still a semi-major outage every few months that required even more complex failover systems surrounding it (message queues etc.) to deal with the downtime. I was with the company for only a few months but it was long enough for this to occur. Does make me wonder what the company was paying all this money (hardware, software and DBAs) for if the system was still expected to fail regularly.
Yep - this is IBM's standard scam. I guess their branding is strong enough to obscure their history of catastrophic failure, to the VP level exec, at least.
Freshbooks has built from scratch two SAAS which have handled over $60 billion worth of invoices and expenses for over 10 million users.
I'm suggesting that Mike McDerment could build and lead a team who would successfully build a payroll system for the Government of Canada for far less than a billion dollars.
IBM has not built a working payroll system (despite starting with Oracle's Peoplesoft for that money). I suspect my company (or some of our best competitors) could build a working payroll system for about $50 million based on our success on smaller projects. Our specialty however is in publishing systems and online video. I'm specifically recommending Freshbooks as McDerment has a background in creating accounting and billing systems and scaling them.
Additionally, as a Canadian, McDerment is already familiar with many of the variables to do with Canadian tax and employment legislation and geography. Any foreigner would need three to six months to assimilate Canada specific information.
I also don't suggest that IBM could not have built this payroll system satisfactorily. Based on failed projects in Canada, Australia, Pennsylvania and Slovenia, big promises, poor delivery, unlimited billing and zero liability on government projects appears to be strategic policy (links to each of those projects appear in a new preface to my article). Such behaviour has made Big Blue billions.
> Freshbooks has built from scratch two SAAS which have handled over $60 billion worth of invoices and expenses for over 10 million users.
Right but those transactions have all been more or less the same type of vanilla invoicing. Unless something has really changed in Freshbooks, that’s an app that handles very specific relatively simple needs. The government of Canada is a giant entity that is most seriously orders of mangnitude different and complex than any usecase for FreshBooks, and that’s just the reality.
> I'm suggesting that Mike McDerment could build and lead a team who would successfully build a payroll system for the Government of Canada for far less than a billion dollars.
Maybe, but probably not. The real problem is probably that the business needs of the government of Canada are not well defined and the bureaucrats cannot define them. The comparison to FreshBooks is just so off because it’s totally upside down. FreshBooks was built and provides certain functionality. Building a payroll system for The government of Canada is an exercise in reverse engineering the Government of Canada. It’s an exercise in squaring hundreds of not thousands of different circles and making them fit into a box.
Global American consultancies are paying 25-40k USD annually in post-Communist EU countries. The teams are run as little dictatures by the "directors" who are facing the outside world. What are you expecting to extract from such arrangement? BTW There is no place for IBM in Germany, they have their own three-letter named IT moloch.
It's not just IBM: any of the larger enterprise consultancies should, at best, be regarded with suspicion. I well remember the EDS/UK Inland Revenue fiasco of the late nineties/early noughties, and there are plenty of other examples involving other big players.
Well, I'd be lying if I suggested I had an overabundance of respect for large consultancies, but actually I don't think it's so simple in either direction.
Consultancies mis-sell and misrepresent their services, whilst public service/government scopes the project poorly and is more than happy to buy into the fantasy. Every project is different so this is still an oversimplification.
Nevertheless, when an enterprise IT service provider is involved in a project it's often a red flag on successful delivery. Whether that's correlation or causation is up for debate.
The issues on the other side are the opportunity ibm, accenture, oracle etc exploit. They're not just aware of those issues going in, they are 100% counting on them.
They're not even interested unless those issues exist - it's one or two orders of magnitude less profit. In fact you can say that those issues are why these consultancies get hired in the first place. No competent manager would go near these carpet baggers.
You can blame the marks for being marks and say they deserve it (tax payers) but this has nothing to do with IBM being a con. A con they are. If they don't like that reputation they shouldn't do it, again and again. But it's so profitable it pays for the PR to restore reputation up to a point. We've now got that point. No PR can cover the stench.
IBM - only an idiot or the corrupt on kickbacks would engage them to con.
There are very similar stories from Australia, with IBM being sued for being late on major (billion dollar) government projects and releasing products that just don't work.
IBM produced a $1.2 billion payroll project for Queensland heath that just never worked.
This is an interesting narrative. However it cannot be the only one. Let us explore some alternatives and side notes.
Who was responsible for picking IBM?
Are they still working for the CA gov?
Have they passed the hot potato to someone else?
How were their technical skills and soft skills evaluated?
Did they receive any donations for the contract?
Where are the safety clauses in the contract?
Is IBM the only benefactor of this contract?
Any political implications?
Remember that Gov. point person also bears a lot of the responsibility for shopping for IT services in a magazine or trough their business network.
If IBM has been so ineffective at delivering services there would be more cases like this and it would ultimately hurt their bottom line. If this was wide spread practice across their business units. Maybe their business as a whole is insulated by the other better performing parts of it's corporation.
It seems to me that these "Governments" should investigate anyone who touched these contracts.
To the widespread corruption present in Eastern Europe. It does exist. However this article goes to show that corruption is present at a larger scale in some of the "most" developed nations on earth.
Because government contracts have to be transparent, the award process tends to be overly bureaucratic and algorithmic. There's no real mechanism with which the Canadian government could bar IBM from bidding, and they're obliged to take the best bid. Usually this is lowest cost per rated point, scored against a large matrix of must-haves and nice-to-haves that are assigned weights.
Sounds like the system is set up in such a way as to deliver a under-performing product. Especially if it optimizes for "lowest cost per rated point". Quality things require significant investment.
No wonder IBM has to go get developers outside of the US/CA/AU if they were the lowest bid.
Now it all fits together. Whenever i go to my states web services and they look odd. I now know why. That is changing though.
If you don't have an objective metric like that, these bureaucrats don't magically learn how to make good decisions. They just overpay their friends instead. These bidding systems are a symptom of bad decision makers, not their cause.
Governments used to build things in house as well ... I remember the first time I applied for federal student aid — in 1999 — it was a long complicated form but it was far more complex and functional than any web application I’d interacted with up to that date.
Built on in-house knowledge rather than subcontracted out to incompetents with zero stake in product quality or understanding of what they were trying to build ... (aka the outsource everything mentality of the gop that spread everywhere during bush era from which American government managed infrastructure on all levels has never recovered — and probably will never recover)
I used to work as an in house contractor for the US Government on a team with two other developers. Our federal managers were pushing to replace a highly customized HR system with a 3rd party product. It was going to cost millions of dollars and take over a year. Our job was to help with the transition even though we built and maintained the original system. However, our contractor manager pushed for us to upgrade the existing system in house by hiring two additional surge developers. We ended up finishing the entire project in six months well under budget and exactly to spec with great feedback from the users. Funny thing about it is the federal managers took credit for the project even though they initially wanted to use a 3rd party system. Such is politics.
It seems that the logic is: "We know this will go bad, are we capable of taking PR hit when (not if) this goes bad". Many companies failed that risk assessment. IBM concluded - "Yes, let's make a run for the money. We are likely doomed for failure, but this will not damage us.". I hope I'm just overly cynical...
IBM is a horrible vendor, but the author is obviously clueless about the levels of incompetence, irresponsibility and indecisiveness of the stakeholders on client side for this kind of projects. All governmens should simply always pick the cheapest vendor because 90% of their projects fail regardless of who the implementers are
I'm hesitant to just blame IBM and call it the day. What about the U.S. healthcare.gov website? Do we also blame CGI Federal and Accenture?
There's probably blame on both sides, but it seems nowadays that "it needs to be a roaring dumpster fire until we do things the right way". Unfortunately, either no one _in_ government seems competent to know that right way or be empowered enough to fight for it (maybe they've all left).
It's a combined issue: the big players have perfected the "malevolent genie" process: bid low, then take every requirement and find the way to extract the most billable hours and change requests from it.
On the government side, many of them realize they're playing a losing game, but they don't have the resources to change the system, and it's not a priority for people to get elected, so congress doesn't step in. (And would they know enough to do so?)
All things considered, healthcare.gov wasn’t that bad. Having to scale to the size of the entire country on day one is a hard-to-impossible task and the managed to recover impressively quickly.
Didn't they recover by hiring a completely different team to rebuild the entire site from scratch?
"Here is the tl;dr version of their story: Marketplace Lite, or “MPL” as they came to be known, devoted months to rewriting Healthcare.gov functions in full, working as a startup within the government and replacing contractor-made apps with ones costing one-fiftieth of the price."
Historically Slovakia gets lumped with Eastern Europe in popular perception - Slovakia is nothing like Romania, the Ukraine, Albania or even Poland in turning out petty criminals or promiscuous online fraudsters. Not to forget Western Europe, France has a far more unhealthy work culture and fraud at work and as a way of living is far more acceptable than in Slovakia. Germany and Austria tend to value probity far more highly than the Mediterranean countries. Hundreds of years of late Roman Empire corruption left an undying footprint.
This attitude is shot through the whole piece (essentializing not just Europeans but also Indians), and it's weird and creepy.
I think we should do more of it on HN (and everywhere else). People should try harder to decouple their biases from facts especially in their public writing.
I'll play devil's advocate here: I actually greatly prefer when people expose their biases early on rather than try to hide them through weaselly or bland language. I think immediate total aversion to nationally biased language is a specifically American trait (and left-coast American at that) - as the result of contemporary cultural conditioning and general lack of exposure to international identities. Europeans are more comfortable with it as it happens more there, there is much more international exchange, and as a result, people can be given the benefit of the doubt by default, in the sense that "oh, I know that when the author says Slovakian culture, he obviously doesn't mean every single individual Slovakian person, but rather his perception of the traits of their culture as a whole."
If Europeans are more exposed than Americans to folks of different nationalities, wouldn't they be less inclined to broadly generalize them?
No one would accuse the author of meaning to judge "every single individual Slovakian person". It's ironic that you state your dislike of "weaselly or bland language", and then go on to describe the author as having a "perception of the traits of their culture as a whole". Most people would describe that as simply having stereotypes.
> If Europeans are more exposed than Americans to folks of different nationalities, wouldn't they be less inclined to broadly generalize them?
Why would it? It's just more data to pattern-match on, and that can mean confirming bias and prejudice as well. It might also shock you to find out that people who work in retail or food service also develop prejudices as well. It's not a good thing but it happens.
I'm not saying that stereotypes are good (or I suppose, that they are always bad), I'm just saying I don't need to clutch my pearls about it every time I see it. I can just read it and think "Okay, this guy is obviously using stereotypes about other countries here" and then keep reading without it necessarily immediately invalidating else that's been written. This is particularly so when Europeans talk about other European countries, as in this particular case. It is, I think, actually a form of comradery. (We might hate the Germans, but they're _our_ Germans, and we won't have any yanks slagging them off!)
I am sharing my perspective here as a European who immigrated to the United States who has observed the response to this in both cultures.
OK I won't get into an argument of whether stereotypes or prejudices are good/bad and I'm sorry if I implied that. I can't speak for the GP (u/alexk), but I interpreted his assertion of "People should try harder to decouple their biases from facts especially in their public writing" as saying that using stereotypes as factual evidence leads to weak writing.
For example, this graf:
> Yet if these people or their friends were the only ones who had contact with your data, no issues at all. Slovaks, particularly in the service industry, are astonishingly honesty.
> [the next section is about how technical work is outsourced from Slovakia]
The author apparently thinks IBM's off-shoring would be fine (indeed, "no issues at all") if the work outsourced to Slovakia stayed in Slovakia (rather than be outsourced to India), and his supporting argument is that "Slovaks are astonishingly honest". Take away the stereotype, and we see how flimsy the author's assertion is. It's not the stereotype that's wrong, necessarily, it's how the author uses it as evidence.
Yes, I suppose I agree with you that people being explicit about their biases makes it easier to identify weak essays. But it's also worth arguing that no matter your biases, your writing is stronger when you rely on empirical evidence, rather than using generalizations/stereotypes that you assume the audience will agree are true.
> The author apparently thinks IBM's off-shoring would be fine (indeed, "no issues at all") if the work outsourced to Slovakia stayed in Slovakia (rather than be outsourced to India)
Yeah. It's a blog post from a Slovakian offshoring company. Of course he does.
I don't expect that this Slovakian blog post about why Slovakian offerings are better than Indian offerings will be free of bias, and in fact I think it's unreasonable to ask for that.
Mostly, I find it a bit annoying that people (almost always Californians - stereotype!) are tripping over themselves trying to point out - "Hey! This guy might be biased! Let me tell everybody, and congratulate everybody else on that awesome 'call out!' We need more of that here!" - with total contextual and cultural blindness.
I think, no, we need more critical thinking, more benefit of the doubt and less US-centrism.
>> I can just read it and think "Okay, this guy is obviously using stereotypes about other countries here" and then keep reading without it necessarily immediately invalidating else that's been written.
The author's whole point in this article seems to be "IBM offshored its work to Bratislava and India and that's bad". I don't know how you can decouple the stereotyping from an article when it's the main line of reasoning of the article.
>The author's whole point in this article seems to be "IBM offshored its work to Bratislava and India and that's bad". I don't know how you can decouple the stereotyping from an article when it's the main line of reasoning of the article.
And - in passing by and without any particular apparent reason - he manages to bad mouth half of the rest of Europe or nearby countries:
"... nothing like Romania, the Ukraine, Albania or even Poland in turning out petty criminals or promiscuous online fraudsters. Not to forget Western Europe, France has a far more unhealthy work culture ..."
If the above is not (additionally gratuitious in the context) stereotyping, I wonder what it is.
> I think immediate total aversion to nationally biased language is a specifically American trait (and left-coast American at that) - as the result of contemporary cultural conditioning and general lack of exposure to international identities.
As an expatriate who has traveled through dozens of countries and lived long-term in a handful, let me assure you: different national personal characteristics are very real. Put an Englishman, a Norwegian, and an Italian in a room together and it is more than their accents that tells you who's whom. Or a Chinese, a Japanese, and a Korean for that matter.
Pick up your suitcase and get outside North America for a bit. The world's a lot more diverse and interesting than the education system might have you believe.
>> Put an Englishman, a Norwegian, and an Italian in a room together and it is more than their accents that tells you who's whom.
Unfortunately, that's very difficult to know for sure, for instance because you can't just take the accents out of the people and see if you can guess their nationalities from their behaviours.
Also, I don't know how you can tell that the differences between an Englishman, a Norwegian and an Italian are due to their nationality and not just the difference in personalities you'd expect to see between any three people- say any three Englishmen or Italians etc.
I've spent time in countries all over the world and my experience was not like yours at all. There were so many more important factors in guessing a foreigners personality than what their nationality/ethnicity was. What is their education level? What is their occupation? What is their economic background? If you want to stereotype then those were much better questions to ask than "chinese, japanese or korean."
Now if you want to argue that certain countries produce way more of one socio-economic class than another and that you're more likely to be exposed to people raised in that culture then maybe I can agree. But to just say, for example, all Nigerians are loud mouthed tricksters? No I don't agree with that kind of stereotyping.
Glad to see someone already posted this. I don't know how anyone writes something like "this is what Slovaks are like" and doesn't stop to think about their biases.
I originally thought maybe he was from Slovakia (I have occasionally made jokes or generalisations about Irish people, as I am one) -- but then the footnotes make it clear it's not self deprecation. It's just prejudice.
Or maybe just not perfect? Google trends show that "the Ukraine" is about 1/3 as popular as "Ukraine" in common usage. It's more like calling the FSB the KGB - wrong, for sure, but was correct for decades, so not at all surprising that people still say it. http://www.businessinsider.com/why-ukraine-isnt-the-ukraine-...
Can anyone explain what's so complex about a "payroll system"?
Presumably you just have a table with people with salary, and every month you wire the money and record information about the wire or otherwise have a way to record when the person has taken cash.
You might want to have more complex rules for determining schedule and amount of salaries, and support for contractors, unpaid leave, taxes, etc. but that doesn't seem particulary complex either.
How can this possibly take more than 100 person-years (i.e. < $30 million) to make even a truly extravagant version of?
Not to mention that presumably such software already exists and could be used instead of writing a new one.
[obviously my explanation would be corruption, but I wonder if I miss something]
Remembering and tracking all the agreements made with workers and unions going back decades and handling the cases where workers change jobs and union designations (or even move from one union to another).
There are thousands of byzantine rules in the contracts and agreements, some mutually contradictory.
The old code likely contained a lot of business logic that represented an institutional memory of how to handle all these corner cases.
Anyone wishing to reimplement a system like that will have to rediscover these corner cases one at a time in production, while thousands of people complain that their paycheque is short.
Then there are also pensions to keep track of etc...
That why you should start small but cover most common cases first. Then gradually work on policies to simplify byzantine rules. Retrain some people and keep internal dev team.
IBM promised a big bang that would be finished before the election. What could possibly go wrong?
I think that there is a positive aspect. Other governments will learn from mistakes when ordering expensive projects like this.
199 comments
[ 3.2 ms ] story [ 230 ms ] threadIt's not uncommon for large government projects to fail with all the bureaucracy and politics that is in play. I am not saying IBM is great, but this article is just not worth it.
Can anyone share a good IBM experience? I'm honestly interested.
The Model M keyboard is from IBM 40+ years ago.
As the supplier you should back yourself as to whether its likely that your changes will cause an issue or not. If so, issue a warning, otherwise don't.
The reason I say this is not for your customer's benefit but for yours. I've seen people get all knotted up with process because someone had the bright idea that the customer should be notified when absolutely anything changes behind the scenes - just in case. This abundance of caution will make your life a misery, and stops any rapid movement. It also desensitizes the customer, who gets used to your notifications and eventually decides they can ignore them.
Its almost seems like when you can get away with milking the system, nobody seems to have any moral qualms in selling the government a $200 screwdriver.
You really don't have to go far to find stories of IBMs cost cutting, moving staff to the lowest possible cost locations, poor treatment of staff etc.
https://www.theregister.co.uk/2018/02/06/ibm_retracts_agreed... is one recent example.
I gave current examples of competency. Its trivial to find others. In any case, my point was merely to differenciate services from products, so I'm not sure what you were replying to anyway.
I remember hearing about it then, but I cannot now find details of the problems.
The IBM pc we bought in the 90ies was fantastic.
It had working suspend resume many years ahead of other computers and came with 365/24 free phone support for a year (and the next years was about 50USD). This being was very useful since I didn't have Internet access.
The support team had unlimited time, patience and knowledge and would walk me (15-16 y.o) through any procedure including if necessary: backup, reconfigure disk with fdisk, restore os from backup etc etc. I learned a whole lot from them both technical stuff as well as how to deal with customers ;-)
(And yes, old IBM ThinkPads were very popular IIRC.)
Edit: details.
I'm not sure when that changed, but I feel like these days, IBM is closer to Oracle than AWS.
This is not true. IBM produces a ton of software (spanning a range from terrible to excellent, quality-wise). This is where much of their profit comes from now. They still produce a bunch of hardware, mostly at the higher (and legacy) end.
What day was that? Back in the 70's with the Selectric typewriters?
No, seriously, those were fucking fantastic machines!
I'm afraid Watson is very far from a "cool new tech", many companies have described their experience with Watson as a money gouging disaster.
"No one ever got fired for buying IBM" because IBM was the safe, established choice. If you bought IBM and the project ended up failing, nobody would blame the failure on your decision, because buying IBM was what you were expected to do. If you bought from some other vendor and the project ended up failing, however, people would rush to claim that the failure was your fault for not doing the safe thing and buying IBM. In other words, you bought IBM to protect yourself from appearing to have taken a risk, not because you expected them to do a particularly good job.
If TFA is accurate, it would appear that "no one ever got fired for buying IBM" is as true today as it ever was. If the Canadian government had hired the Freshbooks guy to build their new payroll system, and the project had failed, the person who had made that decision would have had a whole lot of explaining to do. That person bought IBM, though, so everyone will just shrug their shoulders and assume the fault for the failure lies elsewhere.
I never though of it that way but rather as a CYA move.
Employed elsewhere (o hai msft) but originated, under that particular acronym, at IBM.
The cconcept itself can be traced back at least as far as Sun Tzu.
But: "All warfare is based on deception."
https://suntzusaid.com/book/1/18
(And following, as below.)
Also on the use of spies.
More:
https://suntzusaid.com/book/11/36
https://suntzusaid.com/book/11/37
Late 1980's. I was there, working for a skunkworks fragment in a Big Blue shoppe. To express doubt about IBM was to get no bonus. Then they started to haemorrhage cash to the tune of 5B$/quarter.
Really it's that Oracle is closer, these days, to what IBM was 30 years ago.
"IBM is a multifaceted company.
These kinds of projects fall under what used to be called IBM Global Services https://en.wikipedia.org/wiki/IBM_Global_Services. It could be compared somewhat to EDS (HP), Accenture, Perot Systems (Dell) etc. I've never heard of over-delivery from any of these kinds of outsourcing arrangements, they always seem so obviously destined for boondoggle.
IBM proper, the one that makes mainframes and POWER and DB2 and a ton of operating systems and storage etc is very much a technology company. Some of their best products have the worst sales and marketing efforts. I'm working directly with the senior leadership of the POWER group right now and there are no salesmen in sight.. the technology will either sell itself or not. When we met in person the first time the GM told me "we can build any kind of computer you want" - meaning microarchitecture changes, SERDES configuration, new board layout, sheet metal, OS, application tweaks. Not a lot of companies can do that. There is hubris, less technology, and lack of technical value at FANG or most startups or whatever your benchmark is in comparison.
IBM Research is one of the only remaining great industrial research organizations https://en.wikipedia.org/wiki/IBM_Research. Their results speak for themselves."
The problem is that having a boondoggle division reflects on the whole company. When I'm making decisions for a client, I never even think about what IBM solutions I could apply to a problem.
I've only had a few projects using IBM products, admittedly most very old legacy systems -- but those introductions failed at just basic dev hygiene things. Documentation behind paywalls, couldn't get things to run on my dev machine without very experimental projects made by individuals - last updated in 2003. Just frustrating. I don't like advocating complete rewrites, I've happily updated a company's internal tool that was written in as3. Could have squeezed more hours out of them if they wanted to go with html5, but doing so didn't open up any new possibilities they wanted so I didn't.
If your company is running on RPG, and it's just a CRUD app -- I'm going to save you money by recommending a rewrite under the two circumstances I've encountered in the wild.
Honestly, this isn't their fault. It's awesome that their products have lasted so long for companies. I don't expect a company to keep up to date documentation / interoperable interpreters for 30 year deprecated tech. IBM's just so old that they've accumulated many layers of fossilized products, and those things have caused an unfair bias in my mind (and probably the minds of others)
I'm sure that there are problems IBM research is uniquely qualified to solve, my perception is that they're the wrong choice for most problems. I'm pretty thankful for companies like them. (IBM, Siemens, SAS etc) Their expensive ERP systems have enabled me to raise my rate over the years off the information asynchrony that forms the foundation of their success.
The talk inside was the new CEO (Gerstner[1]) was leveraging IBM's size rather than the previous plan to break it into pieces and was somewhat successful at it.
I left to go back to grad school. And I think the new management went back to the "this is too big and unmanageable, its worth more broken up" mindset.
[1]https://en.wikipedia.org/wiki/Louis_V._Gerstner_Jr.
https://www.amazon.com/IBM-Redux-Gerstner-Business-Turnaroun...
(I also grew up professionally in the 2000s, well after IBM was widely considered elite by the tech nerd community.)
That's a compliment. IBM is by far more dysfunctional.
It changed in early 2000s, I believe.
> surprised to see the author's use of "the Ukraine"
heh
just look up his linkedin; apparently he has spent his prime years working in Russia as some kind of tv producer there.
They were forced to put some IT guys in our office. Those are 2 kids working as cheap freelancers for a local subcontractor hired by IBM.
Service is horrible. Tickets stacking up and being deleted regularly. Project relevant IT not working, people leaving because of it. It's a mess.
Its just that most of my colleagues hate the software made by IBM, its slow, a pain to use and has a tendency to make things very very complicated. You cannot build great stuff with mediocre tools.
The IT industry has moved on to leaner and simpler tools and platforms. IBM, atleast the GBS group, is basically a sales company.
Note : I worked in IBM GBS for a year before I left for more interesting and rewarding jobs. This was in 2008 though.
On the other hand, talking with friends in the government and government contracting, they argue that IBM and the major contractors have perfected the art of exploiting government into maximizing billable hours at the expense of results.
It's a bit like the metaphorical malicious genie interpreting your request. It isn't possible to write specifications that can't be interpreted malevolently.
It's like dealing with any other service business like contracts, lawyers, wedding photographers or others. If you don't give them clear guidance the result will most likely be an expensive disaster.
I think governments should have a special organisation that just maintains the expertise for handling these sort of big projects all across the government.
Small projects are better focused, cost less, are easier to understand, and therefore succeed more frequently (or fail more silently). It's big projects, frequently proposed and conceived by governments or enormous industrial conglomerates that are poorly thought-out, improperly managed, and suffer the worst of project management incompetence or hubris/excess. But who bids on that kind of project? Big companies like IBM. Let's be fair, they don't just bid on them, they also coax them into being, but my point stands: if you want to fail big, you've got to dream big. This isn't a defense of IBM, who deserves to own the shame of talking big but being unable to actually deliver. But it is a reminder that the project designers get equal blame for these sweeping, grandiose, visionary catastrophes.
The Canadian public service is really complex. There are multiple unions with multiple overlapping collective bargaining agreements, where the public service is allocated to different classes. These classes are paid specific rates, with retroactive pay being common for changing classes. The majority of the problems with Phoenix have been employees moving from their classes and being paid the correct amount. It has also adversely affected non-unionized positions.
My understanding is that the Harper government (Prime Minister until 2015), who was responsible for the negotiation and for laying out the requirements, was trying to save a money and not responibly create a pay system. Two major factors that jump out at me:
1) Requirements did not call for training. The system was implemented and IBM was not required to train any operators on how the system functions, which is important because all new staff were hired to run the system and,
2) Due to the need for cost saving measures (the government was trying really hard to balance the budget, as the election was coming up), the previous payroll staff were terminated and a new payroll centre was opened in Miramichi, which is a small town in the middle of nowhere.
So, on top of new software, the government lost all of its institutional knowledge regarding payroll and how things are supposed to work. It's actually hard to say how much of this is IBM's fault and how much is the governments, because the government doesn't know how to fix it. No one knows how Phoenix works and no one knows how it is supposed to work. It's just a big mess with no end in sight.
Could IBM have done a better job? Probably, but garbage in, garbage out.
For further reading, the Auditor-General's report: http://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_01...
In the end project is either canceled or goes over budged or barely manage to deliver anything.
No disrespect but we have come to a point where IT have started appearing magical to pointy hairy bosses who think juse because it's software everything is infinitely malleable with no impact on quality, cost, or time.
It's all about managing the personal bonus and career, who cares for the success 9f project?
Those companies did continue to do business with us though and we've normally earned back our losses and more on future projects.
That's what honest vendors do. That's not what IBM did.
If the project is estimated to take over a year to deploy, by then you're in a new budget cycle and the vendor has come to an agreement with the procurement department to charge for additional services as required in subsequent phases.
So biding yourself into a project to low gives you enough leverage when the point of no return is reached to still make a big plus.
The software developed by IBM was so complex, that it required the expertise of the people who got laid off to operate it. The novices who took over didn't know how to make the software work, and chaos ensued.
The Canadian government is as much at fault as the developers on this one. (I speak as one who is paying for this fiasco :o).
The first $200 million of cost overruns I'm willing to pin on the Government of Canada. The subsequent billion dollars and the failure of the system to work at all is the vendor's fault.
If the Government of Canada is entirely impossible to work with, the vendor should have shut the project down at 200% of budget rather than run it to six times budget and still fail.
my earlier post: https://news.ycombinator.com/item?id=16495431
basically, it's a political game... the same jobs they "lost" in miramichi because of the firearms registry legal change, they tried to get back. Same pork barrel politics that congressmen in the US play with trying to bring jobs to rural districts.
I respectfully disagree. IBM management either knew what they were contracting was a recipe for disaster, or they were incompetent. In both cases, as they were supposed to be the billion-dollar expert on the matter, they bear great responsibility for this failure.
And at an industry level, companies that promise the impossible push out of the market more honest ones, and they deserve all the bad PR IBM is getting on this one.
I'm sitting with a 4x multiple on my old public service job.
"We're estimating based on having access and participation from your senior/experiences payroll employees. Substantial changes in personnel will impact the viability of the project."
There is a degree of management bankruptcy implicit in a large proportion of outsourcing projects. Hiring a vendor to handle billing for your public records dept makes sense, because you can’t afford to maintain a stable of experts on billing.
But a lot of the skill brought in by good contracing firms is in extracting and negotiating requirements. Because your management team has no idea how the work actually gets done.
Think about that. It is cheaper to have some other company come in and learn your problem domain than it is for your company to write down clearly what it is they do and figure out a set of steps to attract employees to do the work for them.
Personally, I think that has Management 101 written all over it in large block letters.
Any vendor will do exactly what is asked, no matter what. IBM, Dell, Accenture, Deloitte, McKinsey, etc. There is no reason why a customer should allow a contractor like this to run roughshod over them -- thats incompetence and bad governance.
I've run big projects with vendors like this. You write good RFPs and hold feet to the fire and they will deliver.
That's the key bit though. Its not easy to write good RFPs. Unless done to an excruciating level of precision (which they very seldom are), you leave wiggle room which the vendor will naturally take advantage of when things get tough.
But end of the day, if the people writing technical requirements have the time and know WTF they are talking about, it works. If you hire <vendor a> to write an RFP that <vendor b-y> responds to and <vendor z> evaluates, how could the result not be a shitshow? (Btw, that happened in a Fortune 500 org!)
In the construction business, there’s no need to state that nails get hammered or screws turned, unless there is a very specific application. But they tend to structure projects in a way where there is an incentive for success and punishment for failure. That's not to say that it's perfect, but there are far more $10M+ failed IT projects than failed construction projects.
If you work in this industry, you've seen your share of IT vendors delivering barely literate offshore workers through 2-5 layers of contracting pimps. That simply doesn't happen with pipefitters and others.
In IT, the tenure of the CIO other senior leadership is often is lower than the vendor and the project. The incentive for the CIO is to listen to <insert vendor here> because they may communicate what is happening better AND they may want a job with the vendor later. Without the threat of reprisal, many organizations will make poor decisions (both vendor and customer).
Customer even gets angry and asking for removal of such risks. Vendors comply by sneeking it in the contract in clever words.
IBM have made billions doing so. Bad publicity is just a good start.
The really interesting point is that if this business plan is shown to be strategic (if I were Canada, Australia or Pennsylvania I'd be looking at full discovery of all email records), IBM would then be vulnerable to civil suit for fraud.
It's obvious that there was zero actual knowledge involved related to implementing large systems - just by the fact that someone thought it was a good idea to take a behemoth system excreted out of Oracle, and then pass it through the corporate body of IBM, before inserting it into a huge government organisation, while at the same time getting rid of any process expertise and business knowledge they might actually have on-board.
However - I don't think it's probable that any amount of training could save the situation to any measurable degree whatsoever.
Is IBM to blame? Well, I think they knew what would happen.
Despite this, there was still a semi-major outage every few months that required even more complex failover systems surrounding it (message queues etc.) to deal with the downtime. I was with the company for only a few months but it was long enough for this to occur. Does make me wonder what the company was paying all this money (hardware, software and DBAs) for if the system was still expected to fail regularly.
"How IBM Conned My Execs Out Of Millions" http://atdt.freeshell.org/k5/story_2005_9_27_95759_4240.html (cache)
I remember the guy was harassed by IBM lawyers and even lost his job.
Did this article really compare FreshBooks to the payroll needs of the Government of Canada? Whatt??
I'm suggesting that Mike McDerment could build and lead a team who would successfully build a payroll system for the Government of Canada for far less than a billion dollars.
IBM has not built a working payroll system (despite starting with Oracle's Peoplesoft for that money). I suspect my company (or some of our best competitors) could build a working payroll system for about $50 million based on our success on smaller projects. Our specialty however is in publishing systems and online video. I'm specifically recommending Freshbooks as McDerment has a background in creating accounting and billing systems and scaling them.
Additionally, as a Canadian, McDerment is already familiar with many of the variables to do with Canadian tax and employment legislation and geography. Any foreigner would need three to six months to assimilate Canada specific information.
I also don't suggest that IBM could not have built this payroll system satisfactorily. Based on failed projects in Canada, Australia, Pennsylvania and Slovenia, big promises, poor delivery, unlimited billing and zero liability on government projects appears to be strategic policy (links to each of those projects appear in a new preface to my article). Such behaviour has made Big Blue billions.
Right but those transactions have all been more or less the same type of vanilla invoicing. Unless something has really changed in Freshbooks, that’s an app that handles very specific relatively simple needs. The government of Canada is a giant entity that is most seriously orders of mangnitude different and complex than any usecase for FreshBooks, and that’s just the reality.
> I'm suggesting that Mike McDerment could build and lead a team who would successfully build a payroll system for the Government of Canada for far less than a billion dollars.
Maybe, but probably not. The real problem is probably that the business needs of the government of Canada are not well defined and the bureaucrats cannot define them. The comparison to FreshBooks is just so off because it’s totally upside down. FreshBooks was built and provides certain functionality. Building a payroll system for The government of Canada is an exercise in reverse engineering the Government of Canada. It’s an exercise in squaring hundreds of not thousands of different circles and making them fit into a box.
Consultancies mis-sell and misrepresent their services, whilst public service/government scopes the project poorly and is more than happy to buy into the fantasy. Every project is different so this is still an oversimplification.
Nevertheless, when an enterprise IT service provider is involved in a project it's often a red flag on successful delivery. Whether that's correlation or causation is up for debate.
No really.
The issues on the other side are the opportunity ibm, accenture, oracle etc exploit. They're not just aware of those issues going in, they are 100% counting on them.
They're not even interested unless those issues exist - it's one or two orders of magnitude less profit. In fact you can say that those issues are why these consultancies get hired in the first place. No competent manager would go near these carpet baggers.
You can blame the marks for being marks and say they deserve it (tax payers) but this has nothing to do with IBM being a con. A con they are. If they don't like that reputation they shouldn't do it, again and again. But it's so profitable it pays for the PR to restore reputation up to a point. We've now got that point. No PR can cover the stench.
IBM - only an idiot or the corrupt on kickbacks would engage them to con.
IBM produced a $1.2 billion payroll project for Queensland heath that just never worked.
NZ also had a disastrous police database project fail that was managed by IBM.
Who was responsible for picking IBM? Are they still working for the CA gov? Have they passed the hot potato to someone else? How were their technical skills and soft skills evaluated? Did they receive any donations for the contract? Where are the safety clauses in the contract? Is IBM the only benefactor of this contract? Any political implications?
Remember that Gov. point person also bears a lot of the responsibility for shopping for IT services in a magazine or trough their business network.
If IBM has been so ineffective at delivering services there would be more cases like this and it would ultimately hurt their bottom line. If this was wide spread practice across their business units. Maybe their business as a whole is insulated by the other better performing parts of it's corporation.
It seems to me that these "Governments" should investigate anyone who touched these contracts.
To the widespread corruption present in Eastern Europe. It does exist. However this article goes to show that corruption is present at a larger scale in some of the "most" developed nations on earth.
No wonder IBM has to go get developers outside of the US/CA/AU if they were the lowest bid.
Now it all fits together. Whenever i go to my states web services and they look odd. I now know why. That is changing though.
Built on in-house knowledge rather than subcontracted out to incompetents with zero stake in product quality or understanding of what they were trying to build ... (aka the outsource everything mentality of the gop that spread everywhere during bush era from which American government managed infrastructure on all levels has never recovered — and probably will never recover)
http://www.cbc.ca/news/canada/ottawa/phoenix-ibm-contract-un...
Few if any 'integrators' will touch these projects.
There's probably blame on both sides, but it seems nowadays that "it needs to be a roaring dumpster fire until we do things the right way". Unfortunately, either no one _in_ government seems competent to know that right way or be empowered enough to fight for it (maybe they've all left).
On the government side, many of them realize they're playing a losing game, but they don't have the resources to change the system, and it's not a priority for people to get elected, so congress doesn't step in. (And would they know enough to do so?)
"Here is the tl;dr version of their story: Marketplace Lite, or “MPL” as they came to be known, devoted months to rewriting Healthcare.gov functions in full, working as a startup within the government and replacing contractor-made apps with ones costing one-fiftieth of the price."
https://www.theatlantic.com/technology/archive/2015/07/the-s...
This attitude is shot through the whole piece (essentializing not just Europeans but also Indians), and it's weird and creepy.
I think we should do more of it on HN (and everywhere else). People should try harder to decouple their biases from facts especially in their public writing.
No one would accuse the author of meaning to judge "every single individual Slovakian person". It's ironic that you state your dislike of "weaselly or bland language", and then go on to describe the author as having a "perception of the traits of their culture as a whole". Most people would describe that as simply having stereotypes.
This is the exact opposite of what I would expect.
Why would it? It's just more data to pattern-match on, and that can mean confirming bias and prejudice as well. It might also shock you to find out that people who work in retail or food service also develop prejudices as well. It's not a good thing but it happens.
I'm not saying that stereotypes are good (or I suppose, that they are always bad), I'm just saying I don't need to clutch my pearls about it every time I see it. I can just read it and think "Okay, this guy is obviously using stereotypes about other countries here" and then keep reading without it necessarily immediately invalidating else that's been written. This is particularly so when Europeans talk about other European countries, as in this particular case. It is, I think, actually a form of comradery. (We might hate the Germans, but they're _our_ Germans, and we won't have any yanks slagging them off!)
I am sharing my perspective here as a European who immigrated to the United States who has observed the response to this in both cultures.
For example, this graf:
> Yet if these people or their friends were the only ones who had contact with your data, no issues at all. Slovaks, particularly in the service industry, are astonishingly honesty.
> [the next section is about how technical work is outsourced from Slovakia]
The author apparently thinks IBM's off-shoring would be fine (indeed, "no issues at all") if the work outsourced to Slovakia stayed in Slovakia (rather than be outsourced to India), and his supporting argument is that "Slovaks are astonishingly honest". Take away the stereotype, and we see how flimsy the author's assertion is. It's not the stereotype that's wrong, necessarily, it's how the author uses it as evidence.
Yes, I suppose I agree with you that people being explicit about their biases makes it easier to identify weak essays. But it's also worth arguing that no matter your biases, your writing is stronger when you rely on empirical evidence, rather than using generalizations/stereotypes that you assume the audience will agree are true.
Yeah. It's a blog post from a Slovakian offshoring company. Of course he does.
I don't expect that this Slovakian blog post about why Slovakian offerings are better than Indian offerings will be free of bias, and in fact I think it's unreasonable to ask for that.
Mostly, I find it a bit annoying that people (almost always Californians - stereotype!) are tripping over themselves trying to point out - "Hey! This guy might be biased! Let me tell everybody, and congratulate everybody else on that awesome 'call out!' We need more of that here!" - with total contextual and cultural blindness.
I think, no, we need more critical thinking, more benefit of the doubt and less US-centrism.
The author's whole point in this article seems to be "IBM offshored its work to Bratislava and India and that's bad". I don't know how you can decouple the stereotyping from an article when it's the main line of reasoning of the article.
And - in passing by and without any particular apparent reason - he manages to bad mouth half of the rest of Europe or nearby countries:
"... nothing like Romania, the Ukraine, Albania or even Poland in turning out petty criminals or promiscuous online fraudsters. Not to forget Western Europe, France has a far more unhealthy work culture ..."
If the above is not (additionally gratuitious in the context) stereotyping, I wonder what it is.
Given that you appear to be incorrect about this case being such an example, are you still sure that it’s all well meaning comradery?
I am Russian
The work culture in France has been understood for a long time. I don't understand how you think it's bias, when you clearly have never worked there.
Pick up your suitcase and get outside North America for a bit. The world's a lot more diverse and interesting than the education system might have you believe.
Unfortunately, that's very difficult to know for sure, for instance because you can't just take the accents out of the people and see if you can guess their nationalities from their behaviours.
Also, I don't know how you can tell that the differences between an Englishman, a Norwegian and an Italian are due to their nationality and not just the difference in personalities you'd expect to see between any three people- say any three Englishmen or Italians etc.
Now if you want to argue that certain countries produce way more of one socio-economic class than another and that you're more likely to be exposed to people raised in that culture then maybe I can agree. But to just say, for example, all Nigerians are loud mouthed tricksters? No I don't agree with that kind of stereotyping.
I originally thought maybe he was from Slovakia (I have occasionally made jokes or generalisations about Irish people, as I am one) -- but then the footnotes make it clear it's not self deprecation. It's just prejudice.
Referring to Ukraine as "the Ukraine" has been inappropriate since the end of 1991. Just this detail paints the author as being out of touch.
Or maybe just not perfect? Google trends show that "the Ukraine" is about 1/3 as popular as "Ukraine" in common usage. It's more like calling the FSB the KGB - wrong, for sure, but was correct for decades, so not at all surprising that people still say it. http://www.businessinsider.com/why-ukraine-isnt-the-ukraine-...
:(
Presumably you just have a table with people with salary, and every month you wire the money and record information about the wire or otherwise have a way to record when the person has taken cash.
You might want to have more complex rules for determining schedule and amount of salaries, and support for contractors, unpaid leave, taxes, etc. but that doesn't seem particulary complex either.
How can this possibly take more than 100 person-years (i.e. < $30 million) to make even a truly extravagant version of?
Not to mention that presumably such software already exists and could be used instead of writing a new one.
[obviously my explanation would be corruption, but I wonder if I miss something]
There are thousands of byzantine rules in the contracts and agreements, some mutually contradictory.
The old code likely contained a lot of business logic that represented an institutional memory of how to handle all these corner cases.
Anyone wishing to reimplement a system like that will have to rediscover these corner cases one at a time in production, while thousands of people complain that their paycheque is short.
Then there are also pensions to keep track of etc...
IBM promised a big bang that would be finished before the election. What could possibly go wrong?
I think that there is a positive aspect. Other governments will learn from mistakes when ordering expensive projects like this.