These large government software projects end up costing way more than they should pretty much every time. The process is just broken. There's way too much design by committee and tying the hands of the people who are doing the actual work, preventing them from implementing the correct solution and instead making them implement the one the disconnected-from-reality specification writers decided on.
Then they have to go back an iterate the correct solution, which involves more committee meetings at huge expense and rewriting documents and disseminating them to everybody...
The CA government engaged the vendor, IBM, after they had decided on a list of requirements from what they wanted from a solution. In your argument you say that the hands of the people doing the actual work were tied. Well that's the vendor, and so no, their hands weren't tied.
And not sure if you've ever worked at a project like this but the committee meetings aren't expensive. They are basically free. And there often should be far more of them. The real cost comes from major changes in the design too late in the project plan.
Committee meeting take the time of multiple people. They are absolutely not free. People's time is by far the biggest expense in a development project like this.
In the long run they can save money (getting everybody on the same page to avoid conflicts in the future), but you're paying up front for that. In practice the value of the meetings decreases with frequency (in the reducto ad absurdum case 100% of your time is spent in committee meetings and nothing gets done).
Only engaging with the vendor after the requirements were drawn up is what I'm talking about. They contract with IBM and hand over a 2,000 page document describing their requirements, then discover that it's impossible to implement and end up spending many years and millions of dollars on trying to fix it.
> but the committee meetings aren't expensive. They are basically free. And there often should be far more of them. The real cost comes from major changes in the design too late in the project plan.
> The term was coined as a metaphor to illuminate Parkinson’s Law of Triviality. Parkinson observed that a committee whose job is to approve plans for a nuclear power plant may spend the majority of its time on relatively unimportant but easy-to-grasp issues, such as what materials to use for the staff bikeshed, while neglecting the design of the power plant itself, which is far more important but also far more difficult to criticize constructively. It was popularized in the Berkeley Software Distribution community by Poul-Henning Kamp[1] and has spread from there to the software industry at large.
Often multiple smaller/medium-sized systems are better for the particular task than one large. Even more so for taxation and payroll systems where taxation/regulation varies greatly.
To that point, I’m wondering if all IT initiatives codenamed “Phoenix” are doomed from the start, given that the implication is “let’s burn down everything we have that works, and rise anew from the ashes”...
Well, one possibility is that I am the common factor in all of my bad relationships. But another possibility is that projects tend to get named 'Phoenix' for a reason, and that just as in real life, it's pretty hard to successfully be reborn from a pile of ashes.
Not just all IT initiatives - I just heard today that the project that would eventually create the Ford Pinto was also named Phoenix. (from Stuff You Should Know podcast)
The Pinto was a good car, though, even with the one design flaw that was actually the result of both best practices for the time and a lack of regulation. Except for that, the Pinto was a landmark subcompact car.
I can literally do that with my DigitalOcean droplet. I can destroy the instance and run a bash script to provision a new, identical machine in around 3 minutes.
It’s a lot of hard work to get every little detail working automatically. Probably 10x as much as just manually setting things up from scratch. But in the long-term it can pay off.
It wasn't a bad book, but if you've read anything by Eli Goldratt on the theory of constraints (The Goal, for example) then Phoenix Project just looks like a watered down, IT-focused, and nearly plagiarized variation.
The comments made about the government about excluding IBM from making future systems for the government stood out.
The reality is payroll should be executed by payroll specialists. I recently had to research payroll providers and there are players who service 20-30% of the Canadian population already.
This is another sign of folks not knowing how to source, design, oversee or implement software implementations.
From everything I hear about working with those companies—new and suitable replacements would be welcomed wholeheartedly.
That is, if they can maintain the same level of professionalism and ability to handle situations like you mentioned. Reliability is of immense importance, so it's not as easy to just hack away at it and sell people an MVP and iterate.
Ripe for disruption, except that when you look at the laundry list of requirements, the thing that needs disrupting is the need for a billion different custom requirements, not a different payroll system.
I thought that too, until I learnt how all the payroll rules are fed into a mainframe for companies like ADP and Ceridian.
A mainframe stood between a client and their need.
Maybe we're after a better mainframe that doesn't need so much workarounds to do what has become payroll today.
I'm choosing to be optimistic that if a Stripe can navigate the payments industry, a payments startup could allow a better mainframe to be built through a rules engine that allows the degree of flexibility and data interoperability required in today's world.
Your comment made me think of two things: Accurate payroll, and auditing of data.
ACCURACY: One of the biggest things I learned is that a correct payroll isn't a correctly executed payroll, but one that the employee is paid correctly. What's the difference? A lot of payrolls have a ton of manual adjustments to fill the gaps, which never easily get codified. If the barrier to codifying rules could be improved, it would pretty easily start to push around mainframes.
AUDITING: Existing platforms like Ceridian and ADP appear to do an okay job with their auditing - any calculation set to be done by a computer, rule, or human is forever recorded on the same leger. For what I saw, it was a healthy paranoia of tracking how everything is calculated for liability's sake. Faving this available, for repairing data and fixing payrolls is more important than reliably executing incorrect rules, or data. It creates a great audit trail for weeding out behavior.
I'm in no way endorsing one product, but I was reasonably surprised by ADP's newish workforce platform. Of course, once we got into the weeds, things broke and and processes needed to be hacked or split up, but it was only because the ancient mainframe underpinning could not be touched. Which sucked.
If Teyve said in Fiddler on the Roof, all traditions were new once. Maybe all new mainframes were new once.
and that's why governments should hire software engineers that can be in the loop and understand not only the technical but the policy side. Hiring an army of contractors earning 300k/year to deliver "something" in the waterfall model is a recipe for disaster.
Have you actually tried what you are saying ? Because it's a nice idea just completely unrealistic.
Software engineers who are not just technical but SMEs as well as having excellent skills in stakeholder engagement are incredibly rare. And most of them know how good they are and are contracting at, you guessed it, around 300k/year. Talented engineers are usually quite savvy when it comes to money.
Also for most projects like this it is Waterfall for the overall project (Requirements, Design, Development, UAT, Production) and Agile for the Design/Development parts. It's really the only way since the client does Requirements/UAT/Production and the vendor does Design/Development.
> Have you actually tried what you are saying ? Because it's a nice idea just completely unrealistic.
I've been in places where it's done. In fact, I've been involved for a long time in a public sector environment with three key systems, one of which is a central claims adjudication system, and two of which are accounting and tracking systems that interface with it, performing similar functions to each other but for two separate programs that share the adjudication system. Those three systems, for historical reasons, have three separate models:
The adjudication is state-owned (that is, the code belongs to the state) and operated, developed almost entirely by a team of contractors with some involvement by state programming staff and a state development lead and also tight integration state-staff IT team that does requirements analysis and acceptance testing and manages most of the interface with state business users.
One of the accounting/tracking systems is, and has been from the beginning, developed and maintained almost entirely in-house by state staff (and almost solely with programming staff.) Because of subsequent organizational consolidation, the state team working on this system overlaps the one working on the prior system.
The other is a vendor-maintained MOTS system, with state IT oversight.
Each has it's strengths and weaknesses, but the MOTS system is consistently the one that is the roadblock to adapting to changing business requirements, and both before and after consolidation the all-internal one was the one with the quickest best-case requirementd to delivery speed, and by far the least expensive for the value delivered.
This is what the US Digital Service is doing. Recruiting these folks /is/ damn hard, but one thing that helps is being able to offer the opportunity to make truly meaningful large-scale impacts. It's a pretty unique environment with some interesting challenges...
One of the things people grossly underestimate when dealing with government is that nobody ever really has enough power to make unilateral decisions.
I'm not even putting the bar at a correct decision, simply A decision.
Just try terminating a person or a contract with the government. Good luck getting the 12 signatures you need.
Even IF IBM screwed up this badly, who with sufficient power in the government is going to sign off on the order to fire IBM? It's a small world, and if you piss off somebody more powerful than you, your position is toast.
The cardinal rule of bureaucracy is to never sign off on anything unless you absolutely have to.
I hope to someday read a comprehensive post-mortem but I doubt there's anyone involved who can/would write one without a lot of bias against the other parties involved. This will go down in an annals of massive software project failures.
If what I've read is accurate, the various departments were already served by smaller systems that fit their specific needs.
The intention with Phoenix was to save money by consolidating them all into one. Oops.
At least part of the problem is that projects like this don't end up being awarded to those who are most capable of completing them successfully. Instead, they end up being awarded to companies and consultancies who are experts at shepherding proposals through the RFP process.
Oracle Peoplesoft is used successfully at a number of organisations so blanket statements aren't helpful.
But I would question whether it is better/cheaper in these cases to just write bespoke software from scratch rather than trying to modify a vendor's existing product.
Everything infects your entire stack. If I hire Java, Ruby, PHP etc people then everything from hardware up to personnel is going to be selected to optimise the delivery and execution of software. Kind of common sense that you would do this.
I don't disagree that Oracle costs a lot and that they inflate margins pretty poorly but to say they have no real utility advantage is a bit ridiculous. Most of the world's enterprises are running core business systems off Oracle and if they move to the cloud it's usually to Oracle instances in the cloud not any other system.
Software implementations usually fail because of the implementors failing to understand and map the business needs prior to beginning in detail, vs launching with whatever and making twice as much money fixing it after the fact "as you go".
I have lead a few full cycle ERP implementations, and replaced vendors along the way if they were not performing.
The challenge remains is software vendors are not sufficiently motivated enough to learn the business in detail before implementing software to help run the process.
It's a mix, I've lead many enterprise implementations, erp, crm, etc. It's not just the implementors, it's all of the above, from the lack of the customer even being able to communicate the business needs.
It'd be lovely if you were right, because the problem is then fixable, but you can take the most motivated, best team of SMEs into a project, and be unsuccessful given the client.
It's all a mess trying to get things done, and every single company is different. Going onsite that first few weeks is an exploration into in what exact ways the mess is going to manifest itself, and how we're going to fight our way through it.
I agree. To clarify, I'd like to believe any client in the position to spend money on an ERP size solution (including payroll) has earned the right to do so - be they got large enough to have a problem by luck, manually inefficient processes, outgrowing existing systems, etc.
There is still a responsibility in my mind on the implementors part to be experts who can assess the maturity in place at a clients operation, as well as the degree and potential depth of unknowns, as well as their bench's ability to be experts of closing the unknown.
Too often, implementors are focused on landing clients, more than passing on work beyond their pay scale due to trivializing or outright ignoring things.
Every single company is totally different - a desire for implementors to want to templatize as much as they can may be part of the issue.
I'm being a bit harsh on implementors despite being one, I think it boils down to if you don't understand a business in detail, including it's data, before beginning, one should probably not touch it, whether its a client or the other side.
I'm certain one of the reasons is that all kinds of things were selected, before getting to the meet of the requirements. Which is typical of large government IT projects, where the order is often: choose some of the technology stack, write an contract opportunity (RFO, etc.) around the chosen technology and a vague (even if stated in terms that seem precise) concept of the requirements, select a contractor, who then pre-determines more of the tech stack based on their own offerings, business relationships, or competencies, then starts gathering real implementable requirements and building the customer-specific elements of the software (and even the last two are sometimes in the reverse order.)
So, what makes a payroll system at this scale so difficult? On the face of it, there are salary employees and there are hourly employees. How many taxing bodies does Canada have? Federal, provincial, and municipal? Is that a few dozen or a few hundred? Is Canadian tax code crazy complicated that doesn't allow for a rules based system?
I'm honestly curious as to how a "simple" payroll system can go so far off the rails.
FTA: ".. had yet to be encoded with many of the payroll complexities faced by a massive and diverse public service, such as unique shift rules for Coast Guard officers or prison guards."
I remember thinking “Why does this simple state Jury Duty site need to be down from 9PM to 9AM”
“I could write a site that did this job with 99.9% reliability in a weekend”
Turns out disability laws require a real person be available to assist with the site. Imagine bidding on a website project and accidentally promising 24/7 live support.
There are multiple rules based on collective bargaining (union).
"The software platform itself, a heavily modified version of Oracle’s PeopleSoft program, had yet to be encoded with many of the payroll complexities faced by a massive and diverse public service, such as unique shift rules for Coast Guard officers or prison guards."
In the end, it seems a bit like "why is $social_network hard" I mean, why is it hard to show my my social graph?
And the answer is: it isn't. The issue is building a system that deals with everybody's social graph. Because you have in your social feed one person that has few friends (easy), that other person that is followed by several people and they leave lots of comments (and you are loading their information when the post is shown - and of course there's no way of sharding that so your info can be read from one shard exclusively) and everybody has their list of blocked persons, other permissions than on top of that you have things like "algorithmic reordering", prioritization, paid promotions, etc
That article doesn't seem to exist at the moment. For the sake of discussion, let's imagine there was some DB schema somewhere that kept two separate WOMEN and MEN tables. Can we not agree that was always wrong? Gay marriage cannot possibly be the first situation in which that sort of decision caused extra work or inconvenience.
> So, what makes a payroll system at this scale so difficult?
Complex pre-existing business rules of a large, sui generis employer (governments, especially sovereign ones rather than localities, often are because the laws that apply to other employees don't apply to them, and the rules that do apply often aren't coherent across-the-board policies determined by a central org focussed on HR, but often are set in special case laws applied to individual subunits of the government by a legislature then operationalized by the HR staff in that subunit) trying to be put in a COTS system make it hard, but probably what makes it most out of control is that the contractor is optimized in maximizing value extracted not efficient delivery and that governments generally suck at overseeing IT contracts (sometimes this is corruption, but it's often just lacking the knowledge and skills to do it effectively at the direct contract management staff level, often complicated by mandated IT contracting frameworks that are based on the underlying assumption that software engineering is basically structurally the same as civil engineering.)
It seems like lots of the payroll was being carried out by humans in local units. Humans are very good at holding lots of exceptions in our heads and, in the absence of clear rules, doing reasonable things. Computers are shit at both of the above.
So things like Bob negotiated with his boss to leave early on Thursday to cover childcare and, in exchange, is coming in early on Tuesdays... but got asked to come in Saturday and now has overtime and what does he get paid based on province or local rules? Does OT end at midnight or not? Do employees get paid non-required OT as a courtesy? And frankly, lots of payroll probably sometimes violates the rules in minor ways, but as long as the employee is happy and the org is happy no-one is upset. Computers are not good at letting small things like that go. And that's a pretty trivial case.
What about eg janitorial that is paid by multiple government orgs on some cost sharing arrangement because the orgs share office space, but is covering an event for a 3rd org in the 2nd org's space? And that janitor is on a particular contract, one of perhaps 5 or 6? If that has to get encoded into a central system, have fun with that. God only knows how many exceptions there are across a quarter of a million employees.
> Humans are very good at holding lots of exceptions in our heads and, in the absence of clear rules, doing reasonable things. Computers are shit at both of the above.
Which is a remarkably concise summary of my thoughts regarding self driving cars.
When did you form those thoughts? The last 10 years has had pretty monotonic progress in the opposite direction. It seems unlikely for the technology to suddenly stop and certainly not go backwards.
It's that 80/20 rule... I'm sure Canada saw monotonic progress in their payroll system over the last 10 years, too. But with cars, the 20 can kill, not just screw up paychecks.
The fundamental problem is when people want software to prevent people from ever messing up instead of simply enabling people to do their jobs better. I heard someone say the other day that they wouldn’t use a simple workflow tool becuase it didn’t provide read only access for other company employees. Look, other people could walk over to your desk and throw all of your papers in the trash too. You still use paper. Why must software prevent malicious actions perfectly?
> does he get paid based on province or local rules?
I believe the answer is "neither". Federal employees are exempted from both. That case would be decided based on federal law and the union contract. It's made more complex by the fact that the union contract is often applied retroactively to events several years in the past due to how long negotiations take.
Corruption is often brought up, and I'm not saying it doesn't happen, but any time I've been consulting, it's always general issues, personal issues, technical issues, refusing to change, all sorts of standard stuff. Not corruption.
It's hard to implement even a minor thing in a large company.
That vendor that has lots of technical issues and the manager that is refusing to change. Well, they know each other and the vendor is giving kickbacks for dragging it out as long as possible.
Most of the corruption in government IT contracting is more subtle; its things like government managers developing relationships with contractors and favoring them subtly without kickbacks, simply because they are comfortable and feel better working with them. The people doing it probably mostly feel like they are actually doing a public good and working around red tape, because they genuinely believe that intangibles that wouldn't be weighed properly in the formal process do weigh in favor of the contractor being a better value.
OTOH, yes, the kind with actual kickbacks absolutely exists, too.
I don't even know if I call that corruption. Humans are the key part of business, and relationships with contractors are just a reflection of that. That absolutely happens, near constantly. I don't think of that as corruption, as much as in a world of very difficult success, when someone proves they can be successful, and make you feel confident, it makes logical sense to stick with them.
Decades of different payment rules are anything but simple.
Imagine people hired over the last 50 years. Each at a different rate, pay scale, union, non-union, some people in a position have certain perks, some don't, and it all varies from region to region, province to province, department to department of the government.
I recently oversaw a payroll transformation project in Canada for an energy co at a smaller, but most complex scale according to the payroll vendor.
A few key problem makers add to the payroll mess;
- Accountants and their spreadsheets who think adding one little tweak here or there isn't a big deal. They create so much technical debt. It's not realized, until 10 or 20 simple tweaks on top of each other, and then have to interpret them relative to labor or union rules of each province. Accountants are not a vision of innovation. You will hear many just talk about having to do "manipulations" of data to get it how it should be and quickly losing vision that computers are good calculators and followers of instruction.
- HR / Payroll practitioners do not have enough digital skills on average. Either they can't do the excel work and feel threatened, or do have the excel skills and feel threatened. Either way, change management is a constant threat.
- No one knows how to successfully procure, implement and roll out software as a non-technical manager. Whether it's accounting, hr, payroll, sales, every position is becoming digital first, and the powers that be are not. It's a good case as to why CTO's are needed reporting to CEO's who get the big picture.
Payroll othherwise, is a neat problem on it's own. In my case, the project ended up implementing a larger vendor of payroll that did a decent job that was saved by preparations we made at the CTO level.
Before selecting a vendor, a payroll script capturing the rules for all employees in all regions to calculate everything was developed in-house. It took months of hardening at a few payrolls per month. Using this in-house payroll tool, and not wishing to update it, it was used to hold the screws to the payroll vendor if their output didn't match. Hilarity inevitably ensued.
If someone built the stripe of payroll, it would be something.
Absolutely. You reminded me of the pain I blocked out from the project - at the end client decided to go international and now needed to pay people everywhere through separate entities after swearing they never would at the start.
Sadly the "global" payroll vendor did not have a global, multi-entity solution, and required individual accounts to be setup in each country and reconciled.
Creating a new multi-tenant, multi-entity, multi-country compatible mainframe is maybe something that needs further exploration.
For example, in the private sector, if my boss goes away on a 2 week vacation, usually the boss might delegate someone to answer questions and possibly run meetings... but that delegated person doesn't get paid more.
However, the public sector unions have negotiated that the delegated person get a pay bump for those 2 weeks only while the boss is away.
> How many taxing bodies does Canada have? Federal, provincial, and municipal?
For payroll tax, only federal and provincial. Municipalities can only collect property tax, not sales or payroll tax. So, all in all, 14 tax jurisdictions (1 federal, 10 provincial, 3 territorial).
If they were also handling payroll for diplomats and other Canadian employees in other countries, this number could be significantly higher.
My understanding is its not taxing, its shift work and the thousands of edge cases across the various gov departments that have evolved their pay systems, contracts/union agreements separately
The headline is misleading. According to the content of the article, the project has been going for 10 years and has spend $460 million. The government has pledged another $431 million in the future to deal with problems going forward.
$460 million over 10 years is $46 million a year, which is still a lot of money, but if you break it down it's not nearly so mystifying. If all the money is being spent by contractors, we could assume that the loaded labour rate of the contractors is about $200K per year. It's been more than a decade since I was in Canada, but I'm guessing this is not an outrageous rate. That breaks down to 230 people working on the project. In truth, the payroll system seems to have been operational for the last 2 years and the system employs 500 employees (it was supposed to replace 1200 employees, thus saving money, but there is no indication if those 1200 were, indeed, replaced). I'm guessing that the $460 million includes those 500 employees for at least 2 years and may also include capital costs to house them (the reporter may not be interested in depreciating capital assets in their report).
So it's hard to say exactly how many contractors worked on this, but it's probably safe to say that there were more than 100. A government project with more than 100 contractors == not possible to succeed in my experience. Just the communication issues alone would be insurmountable. As time wore on and pressure to put something into use grew, the mistakes would have piled up causing absolute mayhem.
Unfortunately this is an all too common occurrence. If you go to any meetup in Ottawa and talk to government contractors I'm sure there will be a couple wiping their brow and saying, "This is a stroke of luck because it will take some of the heat off our our project that is burning even more money".
Not having ever worked on anything like this, the death knell for these projects must be the inability to ever reduce functionality into a reasonable core set. Basically the 80/20 rule but where there are practically an infinite number of edge cases that stretch the project on indefinitely?
You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞
Also there's a good chance that the decision to go with IBM as a vendor, and the conduct thereafter, was corrupt from the start. That would go a long way toward explaining why so little due diligence was observed in assessing whether or not to continue the project with them at any of the stages it could've been.
Yes, people do make huge blunders in powering forward with overpowered, unnecessary solutions to problems they largely do not have; but blunders this big usually don't take this long to suss out.
Idunno, that doesn't really seem like such a big deal. The whole idea was rather pointless (in the face of all the bigger fish they have to fry) to begin with. With such headed, small minded planning, it may have been a blessing if they never bothered to implement it.
At the same time, "Nobody got fired for choosing IBM" is a statement for a reason. If you hire someone else, people will say, "Why didn't you hire the best?"
If you hire IBM, you're avoiding that reasoning, even if on any given project IBM or other large enterprise vendors may not be the best.
>You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞
this is where crucial distinction and choice comes - enterprise software which supposedly have enough flexibility and configurability to cover all the cases or the software which allows and requires a bunch of custom code. The former is a monster and the latter requires good tech chops. And choosing the software with the right balance between what can be configured and what would have to be developed requires deep tech wisdom and experience on the part of the leadership which is usually a very tall order...
Having worked on similar things on a smaller scale but higher up the chain, I can tell you the list of possible issues is a mile long. Everything from lack of leadership and conflicting goals to ever-changing scopes and insufficient resources is a candidate for the big blame. A lot of people, organizations, and systems have to work well together to succeed.
As techies, we think the technical solution is the most superior solution but that is not always the case. Here is an example of something that just happened to me 2 hours ago today. User called and said osTicket system I manage treats '0' as blank. So when they fill out a form with question 'How many X happened today?' and they enter '0', the system just treats the field as unfilled and does not print '0'. That's a big problem for auditing because they absolutely need to be able to show that it was '0'. I started digging into the source code to see where '0' int was treated as a blank. osTicket being PHP where 0, '', null, and false can all be treated as FALSE, it could've been 8+ hours worth of work digging into the right piece of code to figure out the fix.
So I called the user up and we discussed the best way to handle this. In the end, I added a new question above this one: 'Did one or more X happen today? YES/NO'. If it's 'NO', it doesn't matter the 'How many X happened today?' with '0' doesn't get printed.
To make a major implementation work, this kind of discussion needs to happen 10000x a day among 1000+ people, many of whom may not have the authority to just decide things on the fly.
And, often times that discussion leads to a decision maker saying - that shouldn't be the case, it's your problem, not ours that you can't handle that.
So you end up assigning someone to 8+ hours of work to do it, you also have to end up having 2-4 meetings on the topic, and on and on.
You're right, but at the same time, it's a totally unhelpful statement in a project.
Everything is bad design everywhere, at least when you're a consultant. You're generally not brought into projects that have good design, and everything's roses.
I've been on a few projects with amazing design, and it was a very different experience. I didn't even argue for more work for them, because they had it all under control.
They essentially, wanted to be confirmed how great they were. I don't know if the CIO was curious if it was true, or if they all just wanted pats on the back, but we taught them a few really detailed things they didn't know, helped on a few really tricky edge cases, and said you're golden, keep killing it.
I would have gone for the 8 hrs option. Polluting the ui with redundant information and flow on impact to reporting potentially. The impact of saving some hrs might end up amounting to even more. Also, testing and lifecycle of this approach could be worse. And a sign of how we sometimes get carried away by tge tech definition. Why call it 1 or more tickets rather than any tickets?
That's the point I was trying to make. This was something where we could've bikeshedded for weeks with numerous interdepartmental meetings and spent way more hours than necessary to try to come to the optimal solution. Instead since both the user and I had the power to come to the final decision together, we picked this option.
If it works, then we spent less time on the business problem than I've spent writing my two comments here. If it doesn't work, we go back and decide if 8 hours to fix osTicket is worth it or not. Or maybe we find an osTicket consultant to do it for us. Or we find a hosted, mobile-friendly alternative to replace osTicket. Or we integrate this feature into our existing ERP system.
There are a million ways to do anything. Nobody has the time to do any of them in the real-world when we all have hundreds of things to do on a daily basis. The right/optimal/correct solution isn't often the most appropriate solution and vice-versa. People willing to take risks and defend their stances on difficult decisions can get work done. Otherwise you get 'Nobody Ever Got Fired for Buying IBM'.
> You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞
I read something in a trade journal probably probably about 15 years ago, that really stuck with me: When businesses buy big software solutions (I think the article was about ERP systems), they should prepare to adjust and change their processes (within reason, of course) and not just insist that the software be modified endlessly to suit every little ossified process and preference. Not just because it's very expensive, but because the resulting system will be brittle and incompatible and taking advantage of further developments.
Too many buyers of software would benefit immensely from taking a good, hard look at those remaining 20% and making some hard decisions about what is actually important. Sure, for a government payroll system, there is limited room for movement, but I'd suspect that the majority of the long tail are people on weird, custom contracts and the correct solution is actually to hire a room full of clerks to handle these in Excel - and/or applying political pressure on the hiring parties to get their contracts adjusted to fit standard payroll schemes.
> Not just because it's very expensive, but because the resulting system will be brittle and incompatible and taking advantage of further developments.
But that's where the lure of the IBMs of this world comes in: they claim no change will be required, it's all free. Of course, we know that eventually it'll be a big fuckup.
Having worked on ERP software for medium businesses, that sounds spot-on to me; in my experience the biggest challenge was not so much technical as getting clients to change their processes so they can be systematized in a sane way.
There is a lot of resistance from employees not willing to change the way they work. Even seemingly simple things like standardizing language / glossary between departments so we can have a single representation in the system can be an uphill battle.
My favorite quote illustrating this is from former GE executive Craig Kipp, quoted in the book Softwar: An Intimate Portrait of Larry Ellison and Oracle[1]:
“So Oracle has this great product that will do eighty percent of what you want just by throwing the right switches in the software—no special code or anything—and it all falls down because someone asks that innocent little question: ‘What else would you like it to do?’” (p. 231).
Except that there are often legal reasons for things to be in a weird way. Or it would require a long negociation with trade unions and 3 strikes. I have some sympathy for the complexity large organisations (public or private) have to deal with (even if “keeping it simple” is still too underrated). I have way less sympathy for the ERP software than can only handle one use case and then it takes a bunch of non technical consultants months to hack a way around the ERP through configuration when really the problem is that the ERP has been designed in a too narrow minded way or that the consultants aren’t the right people for the job.
I doubt they are on medium - probably low. I once met the IT guy onboard his yacht. Told me a funny story about trying to get the superbowl for one of Larry's mates one year.
Captain position ignored for a moment, I have no idea what second or third mate on a giant Russian/saudi oligarch class of yacht might make in yearly gross, but if I were a billionaire I'd be looking for people with many years of offshore experience and a degree from a major maritime University... $175k usd?
Back in 1999, I was in the elevator and I heard the admin. staff say PeopleSoft (the software) doesn't do anything. This was @ Polytechnic Uni. in Brooklyn. This conflicted with the school's public message where (officially) adopting PeopleSoft was a big step in the right direction. I always wondered if PeopleSoft improved since. I guess not?
My employer recently switched from PeopleSoft to Oracle for HR stuff. So now instead of the system itself being slow to respond, it is plenty responsive when I put in for time off. However, if I fill out the form too quickly it rejects the input as invalid. They can't even get form input validation to work correctly.
Stuff like this reminds me of "Star Wars: A New Hope", where people noticed how the storm troopers kept missing Hans Solo despite having so many hi-tech rifles firing at him. "How can they miss him? They are highly trained and so many of them firing at him at once." These IT anecdotes might explain why.
> That stuff is toxic, and I pity any organization that Oracle got its claws into using that junk.
In my experience, no one is really duped by these big companies. The people writing the checks generally start their search with magic quadrant leaders and then further narrow it down to the biggest players because "we can't trust our business to some no-name." It's really a situation that companies willingly put themselves in.
I've yet to see any way to procure custom "Enterprise" software that isn't awful, and sadly I do believe duping is a huge part of it.
The duping starts long before the Oracles/IBMs/Whomever of the world ever reach a customer too - the "analyst" firm role is a huge problem with companies like Gartner/Forester who dream up the "Magic Quadrant" that people purchasing this stuff end up stuck relying on. I've seen in several roles the enormous gulf between what Gartner/Forester will claim about a leading player's capabilities vs the reality, which of course is a state of affairs large Enterprise software companies are all too keen to encourage.
I've never seen a sales consultant answer "no" to a question about capability in an enterprise software sale scenario, ever, across many RFPs. If your job explicitly targets you to bring in x millions of dollars of business a year, you answer "yes, of course" safe in the knowledge that when it does blow up it's someone else's problem and far away from you by that point.
Duplicity is the name of the game in Enterprisey software. Whether you're a startup struggling to add the features that a client requested, that sales said "Of course we can!" to, and that are utterly foreign to your product ("Why the hell did they ask for voice recognition in this vacation hour tracker?"), or a hugecorp like Oracle with a do-nothing platform that is essentially re-written on site to customer specifications, the time-honored practice of "managing customer expectations" is the name of the game in this market.
Falling behind? Ship half a feature, lie about it or explain how it's going to be working in the next release (and when that doesn't happen, blame the customer's requests for churn, or the height of the tide, it doesn't matter). In danger of actually completing a contract? Trod hard on the bugs, you don't want that money pump to stop. Larry Ellison needs a new dock for his yacht? Time to increment the product's version field and let a thousand consultants bloom. Ka-ching, baby.
My experience with the space, both as a developer of Enterprisey software at several start-ups and as a customer, was that this corner of the industry is ethically sick, and I won't have anything to do with it.
My father was accused of being "anti-Microsoft" sometime in the early 90s, because he kept pushing for cheaper alternatives. He was writing Windows books at the time, so he ended up bringing in a book to show them.
A lot of this purchasing is purely "cover your ass" thinking. Going with IBM, Microsoft or Oracle won't get you fired.
I don't have a lot of love for MS today, but you'd be hard-pressed to compare the MS of today to the MS of the early 90s. Find some developer magazines from that time and check out their ads - they were still scrambling to grab market share in areas other than DOS/Windows for client PCs, and had a lot of competition in the form of Borland, Lotus, Novell, and Oracle, as well as smaller fish like Watcom, Gupta Technologies, Fox Software, etc.
SAP is just as bad - at my last place there was a member of staff who worked 4 days a week - SAP could never get the rounding on their annual leave correct, often leaving them with 0.999 or 0.499 days left that they couldn't book. We had to just record it manually.
"The project was meant to save costs by firing 1,200 employees handling payroll at various departments around the country and replacing them with about 500 people in a centralized location using Phoenix to handle most of the government’s payroll needs."
It's not software problems, or not primarily software problems anyway. They fired EVERYONE who knew anything about payroll in the entire Canadian government, all at once, and hired 500 brand new people in one call center to work with the brand new software as it rolled out. The new software doesn't include code for a lot of "special cases" which together represent a lot of cases. The new people have no idea how to solve most pay problems (people problems OR software problems). They get four days of training.
"Phoenix was supposed to cut our work in half. It actually doubled our work. So, you can't get rid of 2,000 people ... and double the work and expect the work to get done," one said."
Overall, even if the software worked perfectly the execution would have been a disaster.
Reminiscent of the state of Indiana's lawsuit against IBM for allegedly botching a project to automate the state's welfare system — lots of finger-pointing on both sides; the trial court's 65-page decision started out with the words, "Neither party deserves to win this case" [0]. The case has been up to the Indiana Supreme Court already [1]; on remand last summer, the trial court found that IBM is liable for USD $128 million [2].
My brother-in-law's uncle is a doctor (specialist) in Queensland. A couple of years ago he told us a story about this. Because of the massive screwups a bunch of private practitioners like himself weren't getting paid by the state government for work they did for the public health system.
It reached the point where a lot of doctors were simply not going to provide nonessential services if they continued to not get paid (this went on for a year or more). So some workers in the state health department intervened to help resolve the situation. I don't remember the specifics but they essentially approved an advancement that sort of looked like a loan as far as the government system was concerned.
So this guy did get paid. Almost a year later he started getting threatening letters from the government for defaulting on his loan and I think they were threatening court action.
The health department who organized this were sympathetic but said because of the system their hands were tied.
This actually reached the point of the government initiating an investigation and potentially having ramifications for his medical license with the health department seemingly powerless to stop these wheels that were put in motion and looked to be headed to court.
So this guy ended up sending a letter to the health minister threatening to not provide any medical services to the state government at all if this wasn't resolved. Now this would actually have left the state in a pickle as for some things he was the only provider in the state, basically.
Sure enough it was resolved within 2 weeks and the health minister apologized.
Anyway, from this one anecdote it certainly seemed like Queensland health was a disaster.
A similar thing happened in New Zealand when the Ministry of Education rolled out their new Novopay payroll software.
Teachers were getting overpaid, and then collection services would be sicked on them to recover the excess pay, often without contact first to simply ask for the money.
Meanwhile, there was no compensation for teachers who were underpaid.
I didn't know about that issue. Although your story also include:
> Since then, Queensland has initiated - and
> comprehensively lost - attempts to recoup damages
> from the failed project. A judge has ordered the
> government to pay all IBM’s legal costs in the
> failed suit.
Ouch. I know nothing about the rights and wrongs of that case, but many of these agencies wounds are self-inflicted.
I heard informally of one case where a federal department demanded a "big bang" change introducing multiple new systems all in one go, even though it was obvious (and IBM advised) that the thing be done incrementally. Of course the budget and timeline ended up blowing out.
I expect that sort of thing is quite common, and companies like IBM will be perfectly happy to take advantage of the blowouts.
I have known two people who worked in the Queensland Health payroll roll-out. Both actually blamed the government for constantly changing specs, and politicians making claims in parliament about deliverables and deadlines that they then had to meet. They also said IBM (or whatever actual contractor they had on the ground) should have managed things better.
I now work with someone who was at Main Roads Queensland who claims the whole of the Queensland government has an implicit ban on IBM for any new work, and are trying to replace all current IBM products and services.
Edit: My previous job used DB2 on an IBM server. During end of financial year time we had to ask them to turn on more server processors on the server. Jeeesus they charge like a wounded bull!
officials set out to fix Indiana's poorly-performing
welfare system by inserting an untested, theoretical
experiment, and substitute personal caseworkers with
computers and phone calls ("remote eligibility"). This
is now admitted to be an error, and there is nothing in
this case, or the Court's power, that can be done to
correct it, or remedy the lost taxpayer money or
personal suffering of needy Hoosiers.
ML based welfare eligibility screening seems a few short steps from Kafka. I'm not sure if that's what they were doing, but it seems like something that somebody would think is a great idea.
The ONLY person who would think all that is a good idea are the engineers who were able to update their linkedin with "implemented ML based solution to replace human resources with computer resources to streamline the welfare system for the state of X"
I don't think any engineer would consider that a good idea. But if somebody tells you that's what you're working on for the next few years, make lemonade I guess.
That's not at all true. The only engineers who would think all that is a good idea are those who could update their LinkedIn. The salespeople and non-technical leadership though? They're going to love the idea of ML (read: magic) based solutions to replace human resources with computer resources to streamline their welfare systems.
In France a few years ago I registered for unemployment benefits (that was quite a quest), I received a letter from the electricity utility telling me that I qualified for a reduced price. Later when I found a job, they put back the normal price.
States were successful in moving Medicaid enrollment to outsourced providers. THis was probably seen as building on that theme.
I think IBM bought an Irish company that did similar work in the UK, so it probably looked achievable, and a potential Cash cow for other states. (Accenture was able to sell child welfare solutions at great profit in the 90s) Its a great business as it generates alotmof complimentary sales and services.
Accenture was able to sell child welfare solutions at great profit in the 90s
In the 90s that would have been Andersen Consulting rather than Accenture. Although having worked with them at the time and found them worse in every way than IBM is described in this thread, I wouldn't be surprised to learn they profited at the expense of poor children.
I've worked in the public sector and in support of a large complex social services system at a very large scale.
It's a frustrating place to be put in charge of for a political appointee charged with reform, usually without alot of knowledge, or with a idealogical bent fueled by conservative dogma in this case.
IGS is a shitshow, but outsourcing stuff like this is ridiculously dumb. IBM was in a position where failure was the only option, and anyone attempting to do this will fail. IBM underestimated the difficulty of the task, probably because the customer had zero clue.
Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company.
But after selling the deal, their workers have to solve a difficult engineering and organizational management problem simultaneously (the project).
They use contractors (devs) which proves they are not an engineering but sales culture. So, you need to bridge the two worlds. That's the PMs.
These are the most incompetent members of the team. The devs very often are very smart.
Draw horizontal lines on the org chart. The sales guys are at the top. They are generally competent. Proof = they sold a massive deal. The devs are at the bottom. They are often competent. They have to be or they wouldn't get hired or find work. You can PROVE someone doesn't know dev work. But, they don't have enough POWER to change things or fix things.
Reasons for failure:
1. The middle. This is the breakdown. Many PMs often have no real skills. ORGANIZING for success, given a complex technical and organizational problem.
2. The projects are too big. Any huge project is from the get-go at an unacceptable level of risk. Decentralization, Deconstruction is powerful. The projects must be broken into smaller pieces to be managed.
> Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company.
An EDS executive's alleged overpromising, in landing a big contract to build a CRM system for British Sky Broadcasting, ended up costing EDS big bucks: USD $460 million, or more than four times the value of the original contract [0].
"After the decision was handed down, Sky announced that it expected the damage award to be at least £200 million. Had it not been for the misrepresentation claims, the pure-contract damages presumably would have been capped at £30 million. The difference works out to about US$270 million."
I agree with some of your points but I don't think that deconstructioning the pieces of the project is a panacea. What frequently occurs when that happens is that you get distributed work along with distributed responsibility. Then whenever anything goes wrong or needs to change, there is no one who can make it happen.
You get team A who needs team B to make a new api, but they won't do it till they get a new device because their kpis don't improve for any work they do for team A, so HR needs to be involved the do hiring but they need to talk to accounting to approve the budget and on and on and on.
If you've ever seen Rick and Morty, the episode where aliens have accidentally pulled another character in and the leader is trying to find out why, when every department blames another and he says "oh, so it's nobodies fault" is a perfect example of most large projects
Not a panacea, but if you imagine - every effort is begun with a built in base probability of failure (battle won or lost before it's fought kind of thing) - that a smaller effort reduces likelihood of failure cause you have less unknown variables.
Breaking things up, not necessarily with the teams, but with time. Milestones, separate the projects over time. Also, the shorter deadlines I've found help keep focused. Longer than a couple months - things can languish.
Early in my career I had a brief stint as an engineer at a consulting company in the ERP industry. Our projects always had at least one functional consultant whose expertise was _using_ the ERP systems and understanding how they integrated with other systems and procedures. This was a different position than PM, who was often an employee of the client company who drew the short straw.
The functional consultants had varying degrees of technical experience (some were highly technical, including having CS degrees), but in general they were people who in a previous life became really good at managing and hacking their company's ERP system, became the goto person to deal with crap, and figured out they their domain knowledge was highly valuable.
The competence of our technical consultants was questionable. On my first project the senior tech consultant told me that I was the first person he worked with (himself included) who structured his code into modules. He was like, "it makes it so easy to use your stuff!" :) But the functional expertise was excellent and the reason the company had an excellent project success rate.
The company basically imploded after some senior technical engineers and managers bought into the Java and XML fad. Their plans failed horribly because the technology was too complex for the technical consultants (not to mention too immature), and it left little room for leveraging the expertise of the functional consultants as pivoting to Java and XML effectively required reinventing everything. Chaos ensued and, after being bought by a major ERP systems vendor (ironically for their strong project success rate), effectively disbanded.
A functional consultant on our team who is great. Connected to their functional consultant who was wildly differing in skill level, from expert to can barely use a computer.
Technical Consultants, which involved 1 Senior who knew everything about one segment of topics. Another Senior who knew everything about a different segment of topics. A junior to learn things. And the PM which was also the sales lead looking for more work.
1 of those Seniors needed to be able to have social skills and diplomacy, the other didn't. You could hide the 2nd through preplanning.
The junior just needed to know to keep his mouth shut.
Then you'd have a variety of other seniors who you could call in on a particular topic, but you'd try not to use, as they are on projects.
And in our case, we had a GM who was more functional than all of us, almost as technical as all of us, and the best in front of customers. He could be brought in to deal with any special scenarios, and to gut check our plans.
So we were really, ridiculously successful with that model.
Basically, really well paid, SMEs, no cruft except a younger guy to learn on the job. Such a good structure.
Amusing to me that people consider a ton of modules to be a sign of high code quality.
What was your role in the project? Did you swoop in and do a copy and paste refactor into a bunch of tiny files and declare your victory?
I’m going out on a limb here but I’m going to say you’re a little bitter at the value of the consultant vs. your own textbook, indignant idea of value.
You obviously never used Allaire ColdFusion, before it got fancy features like functions.
IIRC, a "tag" was the closest equivalent to a function, but it was basically a parameterized textual include. A "module" was a "tag" that didn't need to be installed into a special directory. During development the only easy way to not have the source code for your entire application in a single logical source file (split across unparameterized includes) was to use modules. Using tags was a PITA because of the need to install them in a special location, but for [reasons] people never bothered using modules at the time.
So to better understand the context, s/modules/functions/.
The fact that I uniquely used modules says less about my competency (or the competency of any particular consultant) and more about the general technical competency of the organization. As I said, some of the technical consultants had CS degrees so they fully understood the concept of functions, as well as more difficult concepts. The senior consultant I mentioned had a CS degree. I didn't mean to imply that I thought I was more competent than he was; quite the opposite. I remember what he said to me precisely because he did have a CS degree (which I lacked), was one of the most experienced consultants at the company who had worked with most every other technical consultant, and was someone I generally looked up to.
And you conveniently skipped my larger point which was that the organization was successful without strong technical competency. All things being equal you want strong technical competency, but especially in the world of ERP systems where everything is highly customized and a tremendous amount of code is ad hoc business logic, functional competency is incomparably more important for achieving project success.
> The sales guys are at the top. They are generally competent. Proof = they sold a massive deal.
I do understand what you meant here, and it's mostly right. But there are also obvious exceptions.
You can often make a sale by promising more than your competitors. If your competitors' bids are calibrated by what's actually possible, and yours isn't, then you'll win the bid... and then, years later, be unable to deliver what was specified, and have to renegotiate. But hey, you won the bid!
I guess that, if the market doesn't keep track of these renegotiations and failures-to-deliver, this is the optimum strategy over the short- and even medium-term. But it gets your company known to devs as a company that chews up and spits out talent. Devs that get stuck on projects trying to do the (literally) impossible, slogging forward each day with the knowledge that all of this is going to be ripped out when the deadlines pass and the renegotiation hits, don't tend to recommend to their friends the companies where they had to do that. So, over the long-term, this is a recipe for a talent shortage.
But, like you said, there's always contractors: fresh pools of devs who never signed up to work for something like IBM, headed by either unscrupulous or just plain naive management willing to take on such jobs with literally-impossible specs.
These kinds of projects fall under what used to be called IBM Global Services https://en.wikipedia.org/wiki/IBM_Global_Services. It could be compared somewhat to EDS (HP), Accenture, Perot Systems (Dell) etc. I've never heard of over-delivery from any of these kinds of outsourcing arrangements, they always seem so obviously destined for boondoggle.
IBM proper, the one that makes mainframes and POWER and DB2 and a ton of operating systems and storage etc is very much a technology company. Some of their best products have the worst sales and marketing efforts. I'm working directly with the senior leadership of the POWER group right now and there are no salesmen in sight.. the technology will either sell itself or not. When we met in person the first time the GM told me "we can build any kind of computer you want" - meaning microarchitecture changes, SERDES configuration, new board layout, sheet metal, OS, application tweaks. Not a lot of companies can do that. There is hubris, less technology, and lack of technical value at FANG or most startups or whatever your benchmark is in comparison.
I’m surprised the Stratix FPGA platform isn’t getting more marketing. Half a TiB/s memory bandwidth [0] (white paper) when GPU’s memory transfer is the bulk of the overhead could help Intel make up for lost gains in SIMD application marketshare.
HP has GenZ, IBM is going to move from DDR or DDR buffers to CAPI attached RAM -- expect to see HBM2 attached to CPUs in 2019. I'd be happy to discuss that kind of thing in email.
Certainly misplaced! It was late, and somehow I scrambled IBM Research and Intel. I was thinking about how Intel's marketing is generally quite bad, whether it's unfair benchmarks or not knowing which products to actually push.
There isn't more buzz around the Stratix 10 MX because it's a phantom chip. Intel's December press release clearly states that the chips are available but you can not buy one of these chips today. A blogger did some research and came to the conclusion that the press release claims were simply not true:
If only people knew how often employees of companies like this scrambled to build something in a mad panic because some exec made a promise to a client about software that wasn’t even designed yet.
> Sales and Engineering - very different competencies. Companies like IBM are NOT technology companies. They are sales-culture oriented, and their product HAPPENS TO BE technology. In fact you could argue, the only way to win big enterprise contracts in the first place, is to be a sales-culture company.
Companies as large as IBM don't have a single unified culture. The sales groups are sales oriented, and the engineering groups are engineering/ tech oriented. Most of the time the two aren't in the same building, and often not even the same city.
> Draw horizontal lines on the org chart. The sales guys are at the top. They are generally competent. Proof = they sold a massive deal. The devs are at the bottom. They are often competent. They have to be or they wouldn't get hired or find work. You can PROVE someone doesn't know dev work. But, they don't have enough POWER to change things or fix things.
> Reasons for failure:
> 1. The middle. This is the breakdown. Many PMs often have no real skills. ORGANIZING for success, given a complex technical and organizational problem. 2. The projects are too big. Any huge project is from the get-go at an unacceptable level of risk. Decentralization, Deconstruction is powerful. The projects must be broken into smaller pieces to be managed.
It seems like you're stretching to blame PMs, and I don't think it's deserved.
The sales guy's job isn't just to sell the biggest deal - it's to sell the biggest deal that the company can actually execute and deliver.
And it's no secret that there at least as many incompetent devs as there are smart, amazing devs. There's a reason "fizzbuzz" is a weed out question.
Likewise, there are good and bad PMs.
Without all of the information it's impossible to place blame on a single group of employees.
I have been on the bad side of IBM Global Services both times I’ve encountered them and was warned off by someone with existing beef.
They will build the most complicated thing that could possibly solve the problem. And then you can’t get rid of them because holy shit nobody else wants to deal with their code.
If Rube Goldberg and H P Lovecraft had a child it would weep in despair knowing that in all its life it would never create something as sinister and complex as the stuff IGS makes before breakfast.
They were trying to charge $80k a year for a proxy server to handle XML RPC. For a system that was basically ftp with code signing. Our team took care of the code signing, soup to nuts. To this day I don’t know what they were doing with all that money for a tiny part of the system. Except try to take over. They didn’t expect the wall of competence they encountered.
That seems a little over-analyzed. Sales is seen as a profit center, engineering is seen as a cost center.
This is no different than a good salesman selling some shipping contracts and then the company failing to deliver because some penny pinching nit-wit "saved" some money by skipping oil changes and tire maintenance causing the trucks to break down before the job was done.
> The sales guys are at the top. They are generally competent. Proof = they sold a massive deal.
Seems very flawed logic. Anyone can make a sale if you promise everything, charge a low price (that can't sustain your organisation/solution), and have no responsibility for actually being able to deliver within the timeframe they promised.
I fail to see how that shows the sales person is competent.
Pretend you're a customer. Would you buy from someone who promises everything at a price that cannot sustain his organisation and accepts no responsibility?
Still, I can confirm from personal experience this is how the world works for many people.
Been in multiple companies that went bankrupt because the sales people showed this behavior.
The project sizes are fine. The team sizes are too large.
There's too much empire building and career-minded politicking going on in companies of this size, which gets in the way of actually working on the product. Managers increase scope to increase their budget, then do busy make-work to justify the budget so they can get more next round. The engineers need to look busy even though things aren't defined, and optimize to internal-facing metrics as opposed to product-oriented productivity. Anybody who steps back and mentions that progress isn't being made towards actually getting the customer a working product is reprimanded as undermining the unquestionably accepted established processes (which usually aren't working anyway).
The actual deliverable gets lost in the shuffle, as the lumbering hulk of the company and career paths within it overshadow the customer.
This is the correct answer. I saw it first hand how PM and engineers don't care about the product, they just care about their careers and company internal metrics.
A friend of mine has had an illustrious career in I.T. with many successes in the last decade. Right before he struck out on his own to build his brand he was unhappy with his then job and decided to work with a recruiter to find a new role.
The recruiter got him a job with a large bank going through a transition from a major platform from a third party that had been long neglected. The platform was on what we will call version 5 of the software while everyone else was running what we will call version 9 of the software.
My friend was hired as a technical project manager. He worked there for less than a month before he struck out on his own and got to realize his full potential.
In that brief period of time he learned the following:
* The bank had not started to put together the requirements for the new software to be integrated.
* The software vendor had no upgrade path from the very old version to the new version.
* The bank needed the migration done in under ninety days or they would start getting fined millions of dollars by regulators.
* The project had already committed to spending $3M / month on a five year commitment with a hosting partner but they didn't have any developers working with them yet.
What I took away is that when big companies do stupid things they are big stupid things. I'm not surprised that government being as big as it is would also do stupid things on a larger scale.
I wholly disagree with your criticism of the middle. The original article points to the cause of failure which was dismissal of SMEs prior to deployment. Almost all projects of this size are doomed to complexity overload but that is surmountable, but loss of Product Owners and SMEs is not.
It wasn't sales who decided not to have a SME on the project. They already got paid; why would they worry about personnel? That is a classic middle management decision.
With Oracle involved, you've also got the strong possibility that the salesperson who got the commission check on the deal is long gone by the time the product is delivered. Turnover is unreal on some of those teams. I have a friend who no longer works there that had 100% turnover on their team in six months.
The whole big 4 IT consulting industry needs to die a swift death. It's basically institutionalized corruption. Overbill the client, under deliver with shoddy products made overseas, get that vendor lock in and watch the money flow in. The cost savings are largely an illusion sold by false promises of a better world.
You are mistaken, I work for one of these. I have 12 years experience and I have only one failure to deliver and it was stopped in the design phase because it was too complex and the client wasn't ready.
I worked for one too and I can't say the same thing. Entire projects were poorly thought out, designed and improperly executed. The people working for the company were incompetent and their "hiring bar" was if you had a pulse. And this was a big well known firm.
Same experience with me. I have mostly only consulted with large companies. There are some companies that insist on solid code and design. There are others that just live from quarter to quarter and the whims of the CEO and just throw out whatever is done at the end of the quarter. Having a great lawyer team chomping at the bit for litigation helps their confidence in living from quarter to quarter.
Do some research into government contracting and you will realize that companies like Oracle are tame compared to many of the straight up scams being pulled.
You will see countless examples of companies with no assets, no employees, virtual offices, etc, pulling down multi-million contracts.
I agree. I worked for the Fed for 6 years. I used software that often didn't work. I'd look in the HTML comments, and oftentimes they were not in English.
These big IT firms sell a premium on top of what the very best US-based development shops would charge, and then outsource it to the cheapest bidder overseas. It's a scam. I don't blame the overseas teams. They are getting paid a fraction of what they should be. So there's probably a lot of junior developers working on this stuff.
For historical perspective on another (in)famous payroll system project I encourage developers to read about the Chrysler C3 project. It ultimately failed, and yet provided the genesis of some agile development best practices that most of us use today.
Canadian here. This has become a huge political football. People are blaming the current Liberal government, when the project was started by the Harper Conservatives. The full extent of just how fucked up it is has just now become public knowledge.
It is also a political pork barrel project.
> The project was meant to save costs by firing 1,200 employees handling payroll at various departments around the country and replacing them with about 500 people in a centralized location using Phoenix to handle most of the government’s payroll needs.
What's not written there is that the decommissioned long gun registry was located in a rural part of New Brunswick. When it was killed by the conservatives, they had a big political issue on their hands because suddenly hundreds of federal government clerical workers were out of a job in this area. So what was the solution? Re-hire the same people and put them on the Phoenix payroll project. Never mind that not one of them had any experience with IBM, Oracle, PeopleSoft, payroll, finances, or anything else related. They wanted butts in chairs in front of keyboards and to keep those people on the federal workforce payroll.
early 1990s, Chretien Liberals: we need a gun registry
liberal party: okay we passed a gun registry law
liberal party: we need a pork barrel project to make these people in small town new brunswick happy, let's hire 500 people in miramichi
liberal party: this gun registry has now cost 400% of what was originally budgeted
liberal party, paul martin government loses an election to stephen harper and the conservative party
harper conservatives: we promise to scrap the long gun registry
harper conservatives: we've removed the long gun registry but now all these people are out of work and pissed off
harper conservatives: we need to overhaul our old payroll system that runs on a bunch of old minicomputers, let's hire ibm
harper conservatives: let's re-hire those people we fired and put the payroll center in small town new brunswick!
harper conservatives lose the 2015 elections
trudeau liberals: wtf is going on with this contract the previous government signed. wtf is going on with this clusterfuck of oracle, ibm, peoplesoft and porkbarrel politics of employing people who are not payroll specialists.
trudeau liberals, later on: seriously this is broken beyond repair
You're missing at least two pieces of incompetence:
1: the long gun registry was also a huge failed massively over budget project, so using the same people for Phoenix...
2: the long gun registry project was relocated from Ottawa to rural New Brunswick. Any competent civil servants switched to other departments to avoid the move. Only the civil servants without other options made the move...
This isn't really a political issue. Implementation was done by the civil service and was and is managed by non-political deputy ministers. There was a similar disaster with the centralization of web servers in the federal civil service a couple of years ago.
The Canadian government attempted to build a gun registry, originally predicting it to cost $12M, then $85M, then $1000M, then $2000M, and then it was scrapped. While it's easy to blame vendors like IBM, I have to imagine feature-creep and "what about"-isms lead to such monstrosities.
quick correction, Canada still has a gun registry for handguns. In one of the several elections that they won, the Harper Conservatives campaigned on removing the long gun registry (rifles and shotguns). Which they successfully did, putting hundreds of federal employees out of work in a small town part of New Brunswick. The political outcry from these unemployed people and the economic impact sent them on a new trip back to the pork barrel, centralizing all of the Phoenix payroll system clerical operations in... you guessed it... the same small town part of New Brunswick.
There are quite a lot, actually, though canada has a greater proportion of traditional type long guns like bird hunting shotguns, bolt action rifles, etc. Less people who want to buy a consumer-legal version of a HK416.
New Zealand in the 90's was trying to have an integrated policing system designed by IBM which burnt a lot of money (in the order of 100 million) and was abandoned [0].
I was working with one of the companies that came along to replace one part of the system with a much smaller component that just handled that individual piece and vividly remember the meeting with the IBM team. They had no code to show me, but they handed my a massive folder - probably verging on 10cm thick of use case diagrams. So many little stick figures staring up at me I was awestruck - but not in a positive way!
I remember interviewing a BA (business analyst) who had just come off of the INCIS project. She said their entity model was enormous, with tens of thousands of entities.
I couldn't understand how any project could have so many entities, let alone how anyone would ever hope to code up such a thing.
She said the commandment to the analysts was to model everything so they did, just like they learnt in data modelling 101.
So (INCIS was a project for the NZ Police), e.g.:
- A police station has zero or more cells
- A cell has one or more doors
- A door has one and only one key
- A door has many bars
Another data point in INCIS - my sister at the time worked in the police. Her workplace ran out of pencils (they did a lot of note taking with pencils and paper). INCIS had so thoroughly depleted the police's budget that there was nothing left for...say.. computers with email and the like.
The software itself, before fixing it, cost $182mm and covered payroll for 90,000 teachers. That's over $2000 per teacher. It cost another $45mm to fix it after this.
As a Canadian who has worked at IBM, I'm not surprised that this didn't find success. In IBMs defense, I don't believe our public sector has the experience or aptitude required to act as a supporting interface for a job of this scale.
With that being said, IBM isn't a successful technology company with a proven record building good software products. They were an unwise choice from the get-go. IBM is a successful financial engineering and sales firm that sells things, then scrambles to hire/acquires to get things done to an ever-evolving, always justifiably rough spec. They quack and then start the hack-show.
It seems kind of simple in a way, if IBM was hired to build X and failed to do so, then there will be a law suit or some form of arbitration and some money will be recovered or something. It's pretty standard breach of contract type stuff. You can audit everything too, maybe IBM was billing for stuff they shouldn't have; I kind of doubt it but maybe there was something really amiss.
On the other hand if IBM was hired to build X, then it changed in to X' and then X'' and 10,000 changes later it turns our, Canada really wanted Y from the start, isn't IBM guiding them along their process to figure that out? It's an insanely expensive way to build and buy stuff.
Is there supposed to be some sort of internal customer's advocate that looks at the project and says "hmm, this is a payroll system, it shouldn't cost a $billion typically, or this was a $100m contract that is already up to $300m, something must be wrong..." Don't get me wrong, in most sales driven organizations, they're high-fiving whoever inked that contract and trying to get invited to the parties he throws on his yacht. The catastrophic failure sucks for everybody, it's hard to see someone else finishing the job more inexpensively that getting IBM to do it though.
What if X wasn't defined to begin, besides a high level of saying "one way or another we'll get you a working result".
Any reasonable firm is hired to map out X to Y as best as possible, or at least be sure they did everything to uncover the possibilities, as much as they may, or may not occur. I have a hard time believing IBM has only experienced this once. I'm not saying IBM would do this, but maybe it happened that certain risks or unknowns didn't get the full attention or support from either side.
Contingencies in contracts are often for when things go to hell, taking up any action or arbitration is still not getting the client the result on top of taking resources away.
Saying we tried when we took a lot of your money isn't exactly a satisfying experience and use of taxpayer dollar. If this had been in the private world, would it have happened as easily?
Your point about competency in procuring digital solutions is a neat one - I don't know of many organizations, be it commercial, government, or academic that are good at this.
I guarantee that "X" wasn't defined when the contract was signed. Not defined as something that could be implemented. It was a problem solving mission.
I also want to clarify, I'm not saying IBM plotted X to Y in the most efficient way or that they did it when others could not in my last sentence. I was saying that now that Canada is a $billion deep. They have 3 choices: 1) kill it all and be $billion poorer and have no solution. 2) Somehow beat IBM in to a better deal and get a solution for like $1.2billion, I think news like this is part of the beating. or 3) Probably spend more than like $1.2b to get someone else to get it across the finish line or start over from scratch. Presumably they need something and at this point IBM is probably the closest to the finish line; change horses midstream and you usually get wet. Hey and let's throw out the other condition, IBM might own the product so switching vendors might be a defacto start from scratch.
In this context, the only history that matters is that which speaks to its current ability to good work. I don't follow IBM closely, but the impression I get is that its current reputation is spotty.
You're right. That was too sweeping, in fairness. I've edited in 'good software products'; that record is categorically abysmal. IBM makes amazing mainframes and physical technical systems, and conducts some fascinating cutting-edge research, and even built really effective punch-card machines for use within Nazi concentration camps. (https://en.wikipedia.org/wiki/IBM_and_the_Holocaust)
IBM has branch prediction (and Out of Order Execution) in their mainframe architecture: the Z series.
I strongly suspect they are subject to a variant of spectre meltdown. Yet, no news were made public on the topic. A tinge of "same cause produces the same effects" makes me think that they are immune to the scrutiny of security research on the topic because there is a tad of security by obfuscation, and secondly university and google don't care about mainframe. It does not mean attackers don't care about the backbone of banking system.
You must not read much technological news. From the top of my head, I can cite Quantum, Photonics, the Neurosynaptic chip, the 5nm chip fab process, POWER and Z, ...
> In IBMs defense, I don't believe our public sector has the experience or aptitude required to act as a supporting interface for a job of this scale.
How in any way is this "In IBMs defense"? We should think it OK that they pursued and committed to a large government contract that in reality they were not qualified to complete? The sentence reads more as a condemnation.
It’s likely more complicated than this and there is probably tons of blame and failure to go around on all sides, particularly on the management and oversight side.
I've worked in consulting on a much smaller scale. Like $1-30MM kinda deals. It was priority 1 to make sure the client knew exactly what we needed from them to get a successful outcome.
Sure, me too. But this doesn't mean the management doesn't change, their requirements don't change, and idiots in consulting will bend over backwards to make them happy.
> Some people were paid too much, others not at all. Those issues snowballed with the deluge of requests to fix incorrect paychecks.
> IBM said it’s “fulfilling its obligations on the Phoenix contract, and the software is functioning as intended.”
Okay, so the cost is coming from incorrect paycheck values, but the software is working correctly?
Does that mean it's human error? Can these issues be traced back to one or a few major instances of human error? Is this a widespread issue caused by a confusing or misleading UX design?
The article is quick to talk about the projects costs and failures as well as the politicians involved, but I can't learn anything or blame anyone until I understand what went wrong.
My guess is that IBM believes the software works according to specification. And maybe it does. Maybe the specifications are wrong. Most likely, it's a combination of bad specs and programmed logic errors that don't always follow the specs that are correct.
Of course, this requires testing and time to hash out. It would be a major mistake to scrap a project in it's final phase of refinement. They should have switched over slowly, department by department, handling and resolving issues along the way. But scrapping the entire project now only means another $1B investment and repeat down the road. Trudeau would join the ranks of the FAA in flushing multi-billion dollar software projects. It's a waste. I don't know about this project, but software engineers died on-the-job working on the FAA's AAS that never launched.
From the anecdotes I've heard, it sounds like the support staff is just wildly insufficient to handle all the issues that crop up with a major change like this. Initially the government switched over a trial group of (iirc) something like 40k employees. There were a bunch of problems with people not being paid or being paid the wrong amounts, but for some incomprehensible reason (perhaps they wanted to ditch the old system to focus on fixing the new one or something crazy like that?) they switched over everyone before fixing those.
Now, I personally know people who've been getting incorrect pay for over a year, who submit a complaint to their HR, who pass it up the chain, where... nothing seems to happen. Presumably because the 500 people or whatever it is tasked to deal with this are literally dealing with tens of thousands of issues, many of which aren't trivial changes (and so block the many that actually are trivial changes, but aren't even being seen).
Basically it was released before it was ready, initial signs of issues were ignored, and now the human side of the system is overwhelmed. It's kind of like when a web server gets overwhelmed with traffic, and it isn't set to reject at a reasonable threshold, so things start getting queued and timing out, people start refreshing pages, and it just snowballs until the whole thing goes down. Which apparently is what has now happened.
Don't quote me on that number; it's from memory. Point is, it was a fraction of the total, and then they released it to everyone without solving the problems in the trial group.
It's been a long time since I've heard about any successful projects coming out of IBM's consulting and services businesses. Mostly it's one headline after another like this- "Billion dollar project scrapped". How much longer can they keep coasting on the reputation of what was, if we're being honest, a completely different business that just happened to have the same name?
You don't hear about successful projects. They don't make news stories.
I've done a ton of projects in consulting, not IBM, but you've never heard of any of them, because they worked, and the customer was happy (although maybe some delays, or other randomness.)
It's self selecting that news is about massive failures.
The software platform itself, a heavily modified version of Oracle’s PeopleSoft program ...
What implementation of Peoplesoft isn't heavily modified?
Adapting business processes to COTS software packages is extremely challenging. Anyone remember the absolute bank that Peoplesoft consultants made back in the day when ERP and Business Process Re-engineering was all the rage?
When I last "dealt with" PeopleSoft, the standard operating procedure was:
1) Pay PeopleSoft/Oracle a metric f--kton of money.
2) A CD arrives in the mail. (This little envelope cost $800,000?)
3) Your PeopleSoft consultants show up, laugh at that CD (tossing it into the microwave) and pull out their own distros with a zillion customizations that they proceed to spray over your servers. You have no idea what they are actually installing.
4) You get out a firehose, fill it with money, and keep showering the consultants with cash until you are sure they won't succeed in delivering anything worthwhile, and you fire them.
I was only installing a local test instance of PeopleSoft (to do some integration work with another product). Somehow the PS consultants in the area got wind that someone in my company was doing an install, and I started getting all kinds of cold calls offering customization and optimization, with some of the most condescending and customer-hostile attitudes I've encountered. PS had pretty a pretty shitty ecosystem 15+ years ago, and it doesn't sound improved. I'd hate for anyone nontechnical to be at their mercy.
I was privvy to discussions surrounding a search for a university president. One of the candidates dropped out when he heard that the university was replacing a perfectly functional home brewed accounting system with PS - it had brought his previous university to its knees and he didn't want to ever deal with it again.
That's so depressing. What really gets me is just how bad the actual Peoplesoft product is: in my experience it's unreliable, poorly designed and unpleasant to interact with.
My company recently switched away from Peoplesoft fortunately.
This reminds me of the part in the Robert Moses biography when he was made responsible over implementing a more meritocratic employee review system for the government. Just like this story, that one also ended in flames, though Moses made it out all right in the end. I wonder who the Moses character in this story was, and if we'll read about it in a biography in a few decades.
332 comments
[ 3.0 ms ] story [ 151 ms ] threadIBM can claim that they have indeed delivered the thing that was ordered, while CA remain disappointed that the effect they had in mind is missing.
In Canada's defense though, it's sometimes hard to find someone who will even take on contracts of this magnitude. Getting good terms, even more so.
Then they have to go back an iterate the correct solution, which involves more committee meetings at huge expense and rewriting documents and disseminating them to everybody...
The CA government engaged the vendor, IBM, after they had decided on a list of requirements from what they wanted from a solution. In your argument you say that the hands of the people doing the actual work were tied. Well that's the vendor, and so no, their hands weren't tied.
And not sure if you've ever worked at a project like this but the committee meetings aren't expensive. They are basically free. And there often should be far more of them. The real cost comes from major changes in the design too late in the project plan.
In the long run they can save money (getting everybody on the same page to avoid conflicts in the future), but you're paying up front for that. In practice the value of the meetings decreases with frequency (in the reducto ad absurdum case 100% of your time is spent in committee meetings and nothing gets done).
Only engaging with the vendor after the requirements were drawn up is what I'm talking about. They contract with IBM and hand over a 2,000 page document describing their requirements, then discover that it's impossible to implement and end up spending many years and millions of dollars on trying to fix it.
https://en.wiktionary.org/wiki/bikeshedding
> The term was coined as a metaphor to illuminate Parkinson’s Law of Triviality. Parkinson observed that a committee whose job is to approve plans for a nuclear power plant may spend the majority of its time on relatively unimportant but easy-to-grasp issues, such as what materials to use for the staff bikeshed, while neglecting the design of the power plant itself, which is far more important but also far more difficult to criticize constructively. It was popularized in the Berkeley Software Distribution community by Poul-Henning Kamp[1] and has spread from there to the software industry at large.
http://bikeshed.com/
To that point, I’m wondering if all IT initiatives codenamed “Phoenix” are doomed from the start, given that the implication is “let’s burn down everything we have that works, and rise anew from the ashes”...
Firefox seems to be doing pretty well :-).
It’s a lot of hard work to get every little detail working automatically. Probably 10x as much as just manually setting things up from scratch. But in the long-term it can pay off.
The reality is payroll should be executed by payroll specialists. I recently had to research payroll providers and there are players who service 20-30% of the Canadian population already.
This is another sign of folks not knowing how to source, design, oversee or implement software implementations.
I take it that's who you're referring to?
Most have their pros and cons but ultimately can handle complex union type payrolls.
Payroll, also is ripe for disruption.
From everything I hear about working with those companies—new and suitable replacements would be welcomed wholeheartedly.
That is, if they can maintain the same level of professionalism and ability to handle situations like you mentioned. Reliability is of immense importance, so it's not as easy to just hack away at it and sell people an MVP and iterate.
A mainframe stood between a client and their need.
Maybe we're after a better mainframe that doesn't need so much workarounds to do what has become payroll today.
I'm choosing to be optimistic that if a Stripe can navigate the payments industry, a payments startup could allow a better mainframe to be built through a rules engine that allows the degree of flexibility and data interoperability required in today's world.
ACCURACY: One of the biggest things I learned is that a correct payroll isn't a correctly executed payroll, but one that the employee is paid correctly. What's the difference? A lot of payrolls have a ton of manual adjustments to fill the gaps, which never easily get codified. If the barrier to codifying rules could be improved, it would pretty easily start to push around mainframes.
AUDITING: Existing platforms like Ceridian and ADP appear to do an okay job with their auditing - any calculation set to be done by a computer, rule, or human is forever recorded on the same leger. For what I saw, it was a healthy paranoia of tracking how everything is calculated for liability's sake. Faving this available, for repairing data and fixing payrolls is more important than reliably executing incorrect rules, or data. It creates a great audit trail for weeding out behavior.
I'm in no way endorsing one product, but I was reasonably surprised by ADP's newish workforce platform. Of course, once we got into the weeds, things broke and and processes needed to be hacked or split up, but it was only because the ancient mainframe underpinning could not be touched. Which sucked.
If Teyve said in Fiddler on the Roof, all traditions were new once. Maybe all new mainframes were new once.
Software engineers who are not just technical but SMEs as well as having excellent skills in stakeholder engagement are incredibly rare. And most of them know how good they are and are contracting at, you guessed it, around 300k/year. Talented engineers are usually quite savvy when it comes to money.
Also for most projects like this it is Waterfall for the overall project (Requirements, Design, Development, UAT, Production) and Agile for the Design/Development parts. It's really the only way since the client does Requirements/UAT/Production and the vendor does Design/Development.
I've been in places where it's done. In fact, I've been involved for a long time in a public sector environment with three key systems, one of which is a central claims adjudication system, and two of which are accounting and tracking systems that interface with it, performing similar functions to each other but for two separate programs that share the adjudication system. Those three systems, for historical reasons, have three separate models:
The adjudication is state-owned (that is, the code belongs to the state) and operated, developed almost entirely by a team of contractors with some involvement by state programming staff and a state development lead and also tight integration state-staff IT team that does requirements analysis and acceptance testing and manages most of the interface with state business users.
One of the accounting/tracking systems is, and has been from the beginning, developed and maintained almost entirely in-house by state staff (and almost solely with programming staff.) Because of subsequent organizational consolidation, the state team working on this system overlaps the one working on the prior system.
The other is a vendor-maintained MOTS system, with state IT oversight.
Each has it's strengths and weaknesses, but the MOTS system is consistently the one that is the roadblock to adapting to changing business requirements, and both before and after consolidation the all-internal one was the one with the quickest best-case requirementd to delivery speed, and by far the least expensive for the value delivered.
I've been on the end of jeopardizing, and on the end of bringing in SMEs... It's a nightmare.
Agile is much better when no one has figured out a way to pay people money
One of the things people grossly underestimate when dealing with government is that nobody ever really has enough power to make unilateral decisions.
I'm not even putting the bar at a correct decision, simply A decision.
Just try terminating a person or a contract with the government. Good luck getting the 12 signatures you need.
Even IF IBM screwed up this badly, who with sufficient power in the government is going to sign off on the order to fire IBM? It's a small world, and if you piss off somebody more powerful than you, your position is toast.
The cardinal rule of bureaucracy is to never sign off on anything unless you absolutely have to.
http://www.cbc.ca/news/canada/ottawa/phoenix-federal-governm...
Maybe if they started with a subset and slowly built up..
Government is way too complex to try to cram all rules into a single monolith like Oracle's PeopleSoft.
The intention with Phoenix was to save money by consolidating them all into one. Oops.
At least part of the problem is that projects like this don't end up being awarded to those who are most capable of completing them successfully. Instead, they end up being awarded to companies and consultancies who are experts at shepherding proposals through the RFP process.
But I would question whether it is better/cheaper in these cases to just write bespoke software from scratch rather than trying to modify a vendor's existing product.
I think proper configuration is what's lacking in this case.
With no real utility advantage, either.
I don't disagree that Oracle costs a lot and that they inflate margins pretty poorly but to say they have no real utility advantage is a bit ridiculous. Most of the world's enterprises are running core business systems off Oracle and if they move to the cloud it's usually to Oracle instances in the cloud not any other system.
Software implementations usually fail because of the implementors failing to understand and map the business needs prior to beginning in detail, vs launching with whatever and making twice as much money fixing it after the fact "as you go".
I have lead a few full cycle ERP implementations, and replaced vendors along the way if they were not performing.
The challenge remains is software vendors are not sufficiently motivated enough to learn the business in detail before implementing software to help run the process.
It'd be lovely if you were right, because the problem is then fixable, but you can take the most motivated, best team of SMEs into a project, and be unsuccessful given the client.
It's all a mess trying to get things done, and every single company is different. Going onsite that first few weeks is an exploration into in what exact ways the mess is going to manifest itself, and how we're going to fight our way through it.
There is still a responsibility in my mind on the implementors part to be experts who can assess the maturity in place at a clients operation, as well as the degree and potential depth of unknowns, as well as their bench's ability to be experts of closing the unknown.
Too often, implementors are focused on landing clients, more than passing on work beyond their pay scale due to trivializing or outright ignoring things.
Every single company is totally different - a desire for implementors to want to templatize as much as they can may be part of the issue.
I'm being a bit harsh on implementors despite being one, I think it boils down to if you don't understand a business in detail, including it's data, before beginning, one should probably not touch it, whether its a client or the other side.
And then you end up with the OP.
I'm honestly curious as to how a "simple" payroll system can go so far off the rails.
“I could write a site that did this job with 99.9% reliability in a weekend”
Turns out disability laws require a real person be available to assist with the site. Imagine bidding on a website project and accidentally promising 24/7 live support.
That’s government tech in a nutshell.
"The software platform itself, a heavily modified version of Oracle’s PeopleSoft program, had yet to be encoded with many of the payroll complexities faced by a massive and diverse public service, such as unique shift rules for Coast Guard officers or prison guards."
It almost gives the impression that an air traffic control system is actually easier (although also failed.)
In the end, it seems a bit like "why is $social_network hard" I mean, why is it hard to show my my social graph?
And the answer is: it isn't. The issue is building a system that deals with everybody's social graph. Because you have in your social feed one person that has few friends (easy), that other person that is followed by several people and they leave lots of comments (and you are loading their information when the post is shown - and of course there's no way of sharding that so your info can be read from one shard exclusively) and everybody has their list of blocked persons, other permissions than on top of that you have things like "algorithmic reordering", prioritization, paid promotions, etc
Gay marriage: the database engineering perspective
https://qntm.org/gay
Complex pre-existing business rules of a large, sui generis employer (governments, especially sovereign ones rather than localities, often are because the laws that apply to other employees don't apply to them, and the rules that do apply often aren't coherent across-the-board policies determined by a central org focussed on HR, but often are set in special case laws applied to individual subunits of the government by a legislature then operationalized by the HR staff in that subunit) trying to be put in a COTS system make it hard, but probably what makes it most out of control is that the contractor is optimized in maximizing value extracted not efficient delivery and that governments generally suck at overseeing IT contracts (sometimes this is corruption, but it's often just lacking the knowledge and skills to do it effectively at the direct contract management staff level, often complicated by mandated IT contracting frameworks that are based on the underlying assumption that software engineering is basically structurally the same as civil engineering.)
So things like Bob negotiated with his boss to leave early on Thursday to cover childcare and, in exchange, is coming in early on Tuesdays... but got asked to come in Saturday and now has overtime and what does he get paid based on province or local rules? Does OT end at midnight or not? Do employees get paid non-required OT as a courtesy? And frankly, lots of payroll probably sometimes violates the rules in minor ways, but as long as the employee is happy and the org is happy no-one is upset. Computers are not good at letting small things like that go. And that's a pretty trivial case.
What about eg janitorial that is paid by multiple government orgs on some cost sharing arrangement because the orgs share office space, but is covering an event for a 3rd org in the 2nd org's space? And that janitor is on a particular contract, one of perhaps 5 or 6? If that has to get encoded into a central system, have fun with that. God only knows how many exceptions there are across a quarter of a million employees.
Which is a remarkably concise summary of my thoughts regarding self driving cars.
I believe the answer is "neither". Federal employees are exempted from both. That case would be decided based on federal law and the union contract. It's made more complex by the fact that the union contract is often applied retroactively to events several years in the past due to how long negotiations take.
It's hard to implement even a minor thing in a large company.
That vendor that has lots of technical issues and the manager that is refusing to change. Well, they know each other and the vendor is giving kickbacks for dragging it out as long as possible.
OTOH, yes, the kind with actual kickbacks absolutely exists, too.
Imagine people hired over the last 50 years. Each at a different rate, pay scale, union, non-union, some people in a position have certain perks, some don't, and it all varies from region to region, province to province, department to department of the government.
I recently oversaw a payroll transformation project in Canada for an energy co at a smaller, but most complex scale according to the payroll vendor.
A few key problem makers add to the payroll mess;
- Accountants and their spreadsheets who think adding one little tweak here or there isn't a big deal. They create so much technical debt. It's not realized, until 10 or 20 simple tweaks on top of each other, and then have to interpret them relative to labor or union rules of each province. Accountants are not a vision of innovation. You will hear many just talk about having to do "manipulations" of data to get it how it should be and quickly losing vision that computers are good calculators and followers of instruction.
- HR / Payroll practitioners do not have enough digital skills on average. Either they can't do the excel work and feel threatened, or do have the excel skills and feel threatened. Either way, change management is a constant threat.
- No one knows how to successfully procure, implement and roll out software as a non-technical manager. Whether it's accounting, hr, payroll, sales, every position is becoming digital first, and the powers that be are not. It's a good case as to why CTO's are needed reporting to CEO's who get the big picture.
Payroll othherwise, is a neat problem on it's own. In my case, the project ended up implementing a larger vendor of payroll that did a decent job that was saved by preparations we made at the CTO level.
Before selecting a vendor, a payroll script capturing the rules for all employees in all regions to calculate everything was developed in-house. It took months of hardening at a few payrolls per month. Using this in-house payroll tool, and not wishing to update it, it was used to hold the screws to the payroll vendor if their output didn't match. Hilarity inevitably ensued.
If someone built the stripe of payroll, it would be something.
Sadly the "global" payroll vendor did not have a global, multi-entity solution, and required individual accounts to be setup in each country and reconciled.
Creating a new multi-tenant, multi-entity, multi-country compatible mainframe is maybe something that needs further exploration.
For example, in the private sector, if my boss goes away on a 2 week vacation, usually the boss might delegate someone to answer questions and possibly run meetings... but that delegated person doesn't get paid more.
However, the public sector unions have negotiated that the delegated person get a pay bump for those 2 weeks only while the boss is away.
For payroll tax, only federal and provincial. Municipalities can only collect property tax, not sales or payroll tax. So, all in all, 14 tax jurisdictions (1 federal, 10 provincial, 3 territorial).
If they were also handling payroll for diplomats and other Canadian employees in other countries, this number could be significantly higher.
$460 million over 10 years is $46 million a year, which is still a lot of money, but if you break it down it's not nearly so mystifying. If all the money is being spent by contractors, we could assume that the loaded labour rate of the contractors is about $200K per year. It's been more than a decade since I was in Canada, but I'm guessing this is not an outrageous rate. That breaks down to 230 people working on the project. In truth, the payroll system seems to have been operational for the last 2 years and the system employs 500 employees (it was supposed to replace 1200 employees, thus saving money, but there is no indication if those 1200 were, indeed, replaced). I'm guessing that the $460 million includes those 500 employees for at least 2 years and may also include capital costs to house them (the reporter may not be interested in depreciating capital assets in their report).
So it's hard to say exactly how many contractors worked on this, but it's probably safe to say that there were more than 100. A government project with more than 100 contractors == not possible to succeed in my experience. Just the communication issues alone would be insurmountable. As time wore on and pressure to put something into use grew, the mistakes would have piled up causing absolute mayhem.
Unfortunately this is an all too common occurrence. If you go to any meetup in Ottawa and talk to government contractors I'm sure there will be a couple wiping their brow and saying, "This is a stroke of luck because it will take some of the heat off our our project that is burning even more money".
You could serve 80% of the payroll burden with software that cost $10m. 90% coverage costs $100m. 98% costs $1,000m. And 100% costs $∞
Yes, people do make huge blunders in powering forward with overpowered, unnecessary solutions to problems they largely do not have; but blunders this big usually don't take this long to suss out.
This is actually pretty minor league compared to U.S. government failures.
e.g. https://www.wired.com/story/us-border-patrol-hasnt-validated...
If you hire IBM, you're avoiding that reasoning, even if on any given project IBM or other large enterprise vendors may not be the best.
this is where crucial distinction and choice comes - enterprise software which supposedly have enough flexibility and configurability to cover all the cases or the software which allows and requires a bunch of custom code. The former is a monster and the latter requires good tech chops. And choosing the software with the right balance between what can be configured and what would have to be developed requires deep tech wisdom and experience on the part of the leadership which is usually a very tall order...
As techies, we think the technical solution is the most superior solution but that is not always the case. Here is an example of something that just happened to me 2 hours ago today. User called and said osTicket system I manage treats '0' as blank. So when they fill out a form with question 'How many X happened today?' and they enter '0', the system just treats the field as unfilled and does not print '0'. That's a big problem for auditing because they absolutely need to be able to show that it was '0'. I started digging into the source code to see where '0' int was treated as a blank. osTicket being PHP where 0, '', null, and false can all be treated as FALSE, it could've been 8+ hours worth of work digging into the right piece of code to figure out the fix.
So I called the user up and we discussed the best way to handle this. In the end, I added a new question above this one: 'Did one or more X happen today? YES/NO'. If it's 'NO', it doesn't matter the 'How many X happened today?' with '0' doesn't get printed.
To make a major implementation work, this kind of discussion needs to happen 10000x a day among 1000+ people, many of whom may not have the authority to just decide things on the fly.
So you end up assigning someone to 8+ hours of work to do it, you also have to end up having 2-4 meetings on the topic, and on and on.
Everything is bad design everywhere, at least when you're a consultant. You're generally not brought into projects that have good design, and everything's roses.
I've been on a few projects with amazing design, and it was a very different experience. I didn't even argue for more work for them, because they had it all under control.
They essentially, wanted to be confirmed how great they were. I don't know if the CIO was curious if it was true, or if they all just wanted pats on the back, but we taught them a few really detailed things they didn't know, helped on a few really tricky edge cases, and said you're golden, keep killing it.
If it works, then we spent less time on the business problem than I've spent writing my two comments here. If it doesn't work, we go back and decide if 8 hours to fix osTicket is worth it or not. Or maybe we find an osTicket consultant to do it for us. Or we find a hosted, mobile-friendly alternative to replace osTicket. Or we integrate this feature into our existing ERP system.
There are a million ways to do anything. Nobody has the time to do any of them in the real-world when we all have hundreds of things to do on a daily basis. The right/optimal/correct solution isn't often the most appropriate solution and vice-versa. People willing to take risks and defend their stances on difficult decisions can get work done. Otherwise you get 'Nobody Ever Got Fired for Buying IBM'.
Zeno's Featureset
I read something in a trade journal probably probably about 15 years ago, that really stuck with me: When businesses buy big software solutions (I think the article was about ERP systems), they should prepare to adjust and change their processes (within reason, of course) and not just insist that the software be modified endlessly to suit every little ossified process and preference. Not just because it's very expensive, but because the resulting system will be brittle and incompatible and taking advantage of further developments.
Too many buyers of software would benefit immensely from taking a good, hard look at those remaining 20% and making some hard decisions about what is actually important. Sure, for a government payroll system, there is limited room for movement, but I'd suspect that the majority of the long tail are people on weird, custom contracts and the correct solution is actually to hire a room full of clerks to handle these in Excel - and/or applying political pressure on the hiring parties to get their contracts adjusted to fit standard payroll schemes.
But that's where the lure of the IBMs of this world comes in: they claim no change will be required, it's all free. Of course, we know that eventually it'll be a big fuckup.
There is a lot of resistance from employees not willing to change the way they work. Even seemingly simple things like standardizing language / glossary between departments so we can have a single representation in the system can be an uphill battle.
“So Oracle has this great product that will do eighty percent of what you want just by throwing the right switches in the software—no special code or anything—and it all falls down because someone asks that innocent little question: ‘What else would you like it to do?’” (p. 231).
https://www.amazon.com/dp/B00AK78QVI/ref=dp-kindle-redirect?...
I hope they get the crap sued out of them. That stuff is toxic, and I pity any organization that Oracle got its claws into using that junk.
In my experience, no one is really duped by these big companies. The people writing the checks generally start their search with magic quadrant leaders and then further narrow it down to the biggest players because "we can't trust our business to some no-name." It's really a situation that companies willingly put themselves in.
The people who have to use it, who are probably best informed to point out it’s strengths and weaknesses are usually the last to see it.
The duping starts long before the Oracles/IBMs/Whomever of the world ever reach a customer too - the "analyst" firm role is a huge problem with companies like Gartner/Forester who dream up the "Magic Quadrant" that people purchasing this stuff end up stuck relying on. I've seen in several roles the enormous gulf between what Gartner/Forester will claim about a leading player's capabilities vs the reality, which of course is a state of affairs large Enterprise software companies are all too keen to encourage.
I've never seen a sales consultant answer "no" to a question about capability in an enterprise software sale scenario, ever, across many RFPs. If your job explicitly targets you to bring in x millions of dollars of business a year, you answer "yes, of course" safe in the knowledge that when it does blow up it's someone else's problem and far away from you by that point.
Falling behind? Ship half a feature, lie about it or explain how it's going to be working in the next release (and when that doesn't happen, blame the customer's requests for churn, or the height of the tide, it doesn't matter). In danger of actually completing a contract? Trod hard on the bugs, you don't want that money pump to stop. Larry Ellison needs a new dock for his yacht? Time to increment the product's version field and let a thousand consultants bloom. Ka-ching, baby.
My experience with the space, both as a developer of Enterprisey software at several start-ups and as a customer, was that this corner of the industry is ethically sick, and I won't have anything to do with it.
A lot of this purchasing is purely "cover your ass" thinking. Going with IBM, Microsoft or Oracle won't get you fired.
"The project was meant to save costs by firing 1,200 employees handling payroll at various departments around the country and replacing them with about 500 people in a centralized location using Phoenix to handle most of the government’s payroll needs."
It's not software problems, or not primarily software problems anyway. They fired EVERYONE who knew anything about payroll in the entire Canadian government, all at once, and hired 500 brand new people in one call center to work with the brand new software as it rolled out. The new software doesn't include code for a lot of "special cases" which together represent a lot of cases. The new people have no idea how to solve most pay problems (people problems OR software problems). They get four days of training.
http://www.cbc.ca/news/canada/ottawa/phoenix-falling-pay-cen...
"Phoenix was supposed to cut our work in half. It actually doubled our work. So, you can't get rid of 2,000 people ... and double the work and expect the work to get done," one said."
Overall, even if the software worked perfectly the execution would have been a disaster.
[0] https://www.scribd.com/document/100544020/Indiana-IBM-Decisi...
[1] http://www.in.gov/judiciary/opinions/pdf/03221601shd.pdf
[2] https://www.indystar.com/story/news/2017/08/07/court-ibm-owe...
http://www.oregonlive.com/politics/index.ssf/2016/09/post_18...
https://www.itnews.com.au/news/queenslands-ibm-ban-lives-on-...
My brother-in-law's uncle is a doctor (specialist) in Queensland. A couple of years ago he told us a story about this. Because of the massive screwups a bunch of private practitioners like himself weren't getting paid by the state government for work they did for the public health system.
It reached the point where a lot of doctors were simply not going to provide nonessential services if they continued to not get paid (this went on for a year or more). So some workers in the state health department intervened to help resolve the situation. I don't remember the specifics but they essentially approved an advancement that sort of looked like a loan as far as the government system was concerned.
So this guy did get paid. Almost a year later he started getting threatening letters from the government for defaulting on his loan and I think they were threatening court action.
The health department who organized this were sympathetic but said because of the system their hands were tied.
This actually reached the point of the government initiating an investigation and potentially having ramifications for his medical license with the health department seemingly powerless to stop these wheels that were put in motion and looked to be headed to court.
So this guy ended up sending a letter to the health minister threatening to not provide any medical services to the state government at all if this wasn't resolved. Now this would actually have left the state in a pickle as for some things he was the only provider in the state, basically.
Sure enough it was resolved within 2 weeks and the health minister apologized.
Anyway, from this one anecdote it certainly seemed like Queensland health was a disaster.
Teachers were getting overpaid, and then collection services would be sicked on them to recover the excess pay, often without contact first to simply ask for the money.
Meanwhile, there was no compensation for teachers who were underpaid.
> Since then, Queensland has initiated - and > comprehensively lost - attempts to recoup damages > from the failed project. A judge has ordered the > government to pay all IBM’s legal costs in the > failed suit.
Ouch. I know nothing about the rights and wrongs of that case, but many of these agencies wounds are self-inflicted.
I heard informally of one case where a federal department demanded a "big bang" change introducing multiple new systems all in one go, even though it was obvious (and IBM advised) that the thing be done incrementally. Of course the budget and timeline ended up blowing out.
I expect that sort of thing is quite common, and companies like IBM will be perfectly happy to take advantage of the blowouts.
http://www.abc.net.au/news/2016-11-25/ibm-to-pay-over-$30m-i...
I now work with someone who was at Main Roads Queensland who claims the whole of the Queensland government has an implicit ban on IBM for any new work, and are trying to replace all current IBM products and services.
Edit: My previous job used DB2 on an IBM server. During end of financial year time we had to ask them to turn on more server processors on the server. Jeeesus they charge like a wounded bull!
I'll bet there's plenty of IBM salespeople who hate that one guy who made bank on the QHealth deal...
15 years, $2.2billion and not finished.
http://www.dtic.mil/docs/citations/AD1002341
I think IBM bought an Irish company that did similar work in the UK, so it probably looked achievable, and a potential Cash cow for other states. (Accenture was able to sell child welfare solutions at great profit in the 90s) Its a great business as it generates alotmof complimentary sales and services.
In the 90s that would have been Andersen Consulting rather than Accenture. Although having worked with them at the time and found them worse in every way than IBM is described in this thread, I wouldn't be surprised to learn they profited at the expense of poor children.
It's a frustrating place to be put in charge of for a political appointee charged with reform, usually without alot of knowledge, or with a idealogical bent fueled by conservative dogma in this case.
IGS is a shitshow, but outsourcing stuff like this is ridiculously dumb. IBM was in a position where failure was the only option, and anyone attempting to do this will fail. IBM underestimated the difficulty of the task, probably because the customer had zero clue.
But after selling the deal, their workers have to solve a difficult engineering and organizational management problem simultaneously (the project).
They use contractors (devs) which proves they are not an engineering but sales culture. So, you need to bridge the two worlds. That's the PMs. These are the most incompetent members of the team. The devs very often are very smart.
Draw horizontal lines on the org chart. The sales guys are at the top. They are generally competent. Proof = they sold a massive deal. The devs are at the bottom. They are often competent. They have to be or they wouldn't get hired or find work. You can PROVE someone doesn't know dev work. But, they don't have enough POWER to change things or fix things.
Reasons for failure:
1. The middle. This is the breakdown. Many PMs often have no real skills. ORGANIZING for success, given a complex technical and organizational problem. 2. The projects are too big. Any huge project is from the get-go at an unacceptable level of risk. Decentralization, Deconstruction is powerful. The projects must be broken into smaller pieces to be managed.
An EDS executive's alleged overpromising, in landing a big contract to build a CRM system for British Sky Broadcasting, ended up costing EDS big bucks: USD $460 million, or more than four times the value of the original contract [0].
[0] http://www.oncontracts.com/eds-british-sky-overpromising/ (self-cite)
Seems like a math error.
You get team A who needs team B to make a new api, but they won't do it till they get a new device because their kpis don't improve for any work they do for team A, so HR needs to be involved the do hiring but they need to talk to accounting to approve the budget and on and on and on.
If you've ever seen Rick and Morty, the episode where aliens have accidentally pulled another character in and the leader is trying to find out why, when every department blames another and he says "oh, so it's nobodies fault" is a perfect example of most large projects
Breaking things up, not necessarily with the teams, but with time. Milestones, separate the projects over time. Also, the shorter deadlines I've found help keep focused. Longer than a couple months - things can languish.
The functional consultants had varying degrees of technical experience (some were highly technical, including having CS degrees), but in general they were people who in a previous life became really good at managing and hacking their company's ERP system, became the goto person to deal with crap, and figured out they their domain knowledge was highly valuable.
The competence of our technical consultants was questionable. On my first project the senior tech consultant told me that I was the first person he worked with (himself included) who structured his code into modules. He was like, "it makes it so easy to use your stuff!" :) But the functional expertise was excellent and the reason the company had an excellent project success rate.
The company basically imploded after some senior technical engineers and managers bought into the Java and XML fad. Their plans failed horribly because the technology was too complex for the technical consultants (not to mention too immature), and it left little room for leveraging the expertise of the functional consultants as pivoting to Java and XML effectively required reinventing everything. Chaos ensued and, after being bought by a major ERP systems vendor (ironically for their strong project success rate), effectively disbanded.
A functional consultant on our team who is great. Connected to their functional consultant who was wildly differing in skill level, from expert to can barely use a computer.
Technical Consultants, which involved 1 Senior who knew everything about one segment of topics. Another Senior who knew everything about a different segment of topics. A junior to learn things. And the PM which was also the sales lead looking for more work.
1 of those Seniors needed to be able to have social skills and diplomacy, the other didn't. You could hide the 2nd through preplanning.
The junior just needed to know to keep his mouth shut.
Then you'd have a variety of other seniors who you could call in on a particular topic, but you'd try not to use, as they are on projects.
And in our case, we had a GM who was more functional than all of us, almost as technical as all of us, and the best in front of customers. He could be brought in to deal with any special scenarios, and to gut check our plans.
So we were really, ridiculously successful with that model.
Basically, really well paid, SMEs, no cruft except a younger guy to learn on the job. Such a good structure.
What was your role in the project? Did you swoop in and do a copy and paste refactor into a bunch of tiny files and declare your victory?
I’m going out on a limb here but I’m going to say you’re a little bitter at the value of the consultant vs. your own textbook, indignant idea of value.
IIRC, a "tag" was the closest equivalent to a function, but it was basically a parameterized textual include. A "module" was a "tag" that didn't need to be installed into a special directory. During development the only easy way to not have the source code for your entire application in a single logical source file (split across unparameterized includes) was to use modules. Using tags was a PITA because of the need to install them in a special location, but for [reasons] people never bothered using modules at the time.
So to better understand the context, s/modules/functions/.
The fact that I uniquely used modules says less about my competency (or the competency of any particular consultant) and more about the general technical competency of the organization. As I said, some of the technical consultants had CS degrees so they fully understood the concept of functions, as well as more difficult concepts. The senior consultant I mentioned had a CS degree. I didn't mean to imply that I thought I was more competent than he was; quite the opposite. I remember what he said to me precisely because he did have a CS degree (which I lacked), was one of the most experienced consultants at the company who had worked with most every other technical consultant, and was someone I generally looked up to.
And you conveniently skipped my larger point which was that the organization was successful without strong technical competency. All things being equal you want strong technical competency, but especially in the world of ERP systems where everything is highly customized and a tremendous amount of code is ad hoc business logic, functional competency is incomparably more important for achieving project success.
I do understand what you meant here, and it's mostly right. But there are also obvious exceptions.
You can often make a sale by promising more than your competitors. If your competitors' bids are calibrated by what's actually possible, and yours isn't, then you'll win the bid... and then, years later, be unable to deliver what was specified, and have to renegotiate. But hey, you won the bid!
I guess that, if the market doesn't keep track of these renegotiations and failures-to-deliver, this is the optimum strategy over the short- and even medium-term. But it gets your company known to devs as a company that chews up and spits out talent. Devs that get stuck on projects trying to do the (literally) impossible, slogging forward each day with the knowledge that all of this is going to be ripped out when the deadlines pass and the renegotiation hits, don't tend to recommend to their friends the companies where they had to do that. So, over the long-term, this is a recipe for a talent shortage.
But, like you said, there's always contractors: fresh pools of devs who never signed up to work for something like IBM, headed by either unscrupulous or just plain naive management willing to take on such jobs with literally-impossible specs.
These kinds of projects fall under what used to be called IBM Global Services https://en.wikipedia.org/wiki/IBM_Global_Services. It could be compared somewhat to EDS (HP), Accenture, Perot Systems (Dell) etc. I've never heard of over-delivery from any of these kinds of outsourcing arrangements, they always seem so obviously destined for boondoggle.
IBM proper, the one that makes mainframes and POWER and DB2 and a ton of operating systems and storage etc is very much a technology company. Some of their best products have the worst sales and marketing efforts. I'm working directly with the senior leadership of the POWER group right now and there are no salesmen in sight.. the technology will either sell itself or not. When we met in person the first time the GM told me "we can build any kind of computer you want" - meaning microarchitecture changes, SERDES configuration, new board layout, sheet metal, OS, application tweaks. Not a lot of companies can do that. There is hubris, less technology, and lack of technical value at FANG or most startups or whatever your benchmark is in comparison.
IBM Research is one of the only remaining great industrial research organizations https://en.wikipedia.org/wiki/IBM_Research. Their results speak for themselves.
[0]: https://www.altera.com/content/dam/altera-www/global/en_US/p...
HP has GenZ, IBM is going to move from DDR or DDR buffers to CAPI attached RAM -- expect to see HBM2 attached to CPUs in 2019. I'd be happy to discuss that kind of thing in email.
Please disregard.
There isn't more buzz around the Stratix 10 MX because it's a phantom chip. Intel's December press release clearly states that the chips are available but you can not buy one of these chips today. A blogger did some research and came to the conclusion that the press release claims were simply not true:
https://semiaccurate.com/2017/12/19/intels-claims-fpgas-hbm-...
In contrast, Xilinx released a video of their competing chip working with all channels running at the full 460GB/s on day 1 of silicon bring up:
https://www.xilinx.com/video/fpga/first-virtex-ultrascale-pl...
And my intention not to bash IBM or even IGS, but describe WHY some, not just IGS but many others - mega-projects blow up.
My fault for not making that clear. I’d edit the post but I can’t now. Their technology LOBs I’m sure are a different story.
Companies as large as IBM don't have a single unified culture. The sales groups are sales oriented, and the engineering groups are engineering/ tech oriented. Most of the time the two aren't in the same building, and often not even the same city.
> Draw horizontal lines on the org chart. The sales guys are at the top. They are generally competent. Proof = they sold a massive deal. The devs are at the bottom. They are often competent. They have to be or they wouldn't get hired or find work. You can PROVE someone doesn't know dev work. But, they don't have enough POWER to change things or fix things.
> Reasons for failure:
> 1. The middle. This is the breakdown. Many PMs often have no real skills. ORGANIZING for success, given a complex technical and organizational problem. 2. The projects are too big. Any huge project is from the get-go at an unacceptable level of risk. Decentralization, Deconstruction is powerful. The projects must be broken into smaller pieces to be managed.
It seems like you're stretching to blame PMs, and I don't think it's deserved.
The sales guy's job isn't just to sell the biggest deal - it's to sell the biggest deal that the company can actually execute and deliver.
And it's no secret that there at least as many incompetent devs as there are smart, amazing devs. There's a reason "fizzbuzz" is a weed out question.
Likewise, there are good and bad PMs.
Without all of the information it's impossible to place blame on a single group of employees.
They will build the most complicated thing that could possibly solve the problem. And then you can’t get rid of them because holy shit nobody else wants to deal with their code.
If Rube Goldberg and H P Lovecraft had a child it would weep in despair knowing that in all its life it would never create something as sinister and complex as the stuff IGS makes before breakfast.
They were trying to charge $80k a year for a proxy server to handle XML RPC. For a system that was basically ftp with code signing. Our team took care of the code signing, soup to nuts. To this day I don’t know what they were doing with all that money for a tiny part of the system. Except try to take over. They didn’t expect the wall of competence they encountered.
This is no different than a good salesman selling some shipping contracts and then the company failing to deliver because some penny pinching nit-wit "saved" some money by skipping oil changes and tire maintenance causing the trucks to break down before the job was done.
Its just on a bigger scale.
Seems very flawed logic. Anyone can make a sale if you promise everything, charge a low price (that can't sustain your organisation/solution), and have no responsibility for actually being able to deliver within the timeframe they promised.
I fail to see how that shows the sales person is competent.
The two I've spoken to were also both knowledgeable and intelligent, so your rhetoric is glib but a little lacking in factual accuracy.
Do you give the CEO the right, expensive and difficult answer? Or assign accountability to a third party, bug out and leave the mess to the next guy?
In most companies with mediocre the latter is the answer.
There's too much empire building and career-minded politicking going on in companies of this size, which gets in the way of actually working on the product. Managers increase scope to increase their budget, then do busy make-work to justify the budget so they can get more next round. The engineers need to look busy even though things aren't defined, and optimize to internal-facing metrics as opposed to product-oriented productivity. Anybody who steps back and mentions that progress isn't being made towards actually getting the customer a working product is reprimanded as undermining the unquestionably accepted established processes (which usually aren't working anyway).
The actual deliverable gets lost in the shuffle, as the lumbering hulk of the company and career paths within it overshadow the customer.
The recruiter got him a job with a large bank going through a transition from a major platform from a third party that had been long neglected. The platform was on what we will call version 5 of the software while everyone else was running what we will call version 9 of the software.
My friend was hired as a technical project manager. He worked there for less than a month before he struck out on his own and got to realize his full potential.
In that brief period of time he learned the following:
* The bank had not started to put together the requirements for the new software to be integrated. * The software vendor had no upgrade path from the very old version to the new version. * The bank needed the migration done in under ninety days or they would start getting fined millions of dollars by regulators. * The project had already committed to spending $3M / month on a five year commitment with a hosting partner but they didn't have any developers working with them yet.
What I took away is that when big companies do stupid things they are big stupid things. I'm not surprised that government being as big as it is would also do stupid things on a larger scale.
The reality is, in all of them there are teams of amazing, insanely talented people, and teams of complete train wrecks.
It's just life. In any large company, there's a massive variance in ability.
You will see countless examples of companies with no assets, no employees, virtual offices, etc, pulling down multi-million contracts.
These big IT firms sell a premium on top of what the very best US-based development shops would charge, and then outsource it to the cheapest bidder overseas. It's a scam. I don't blame the overseas teams. They are getting paid a fraction of what they should be. So there's probably a lot of junior developers working on this stuff.
http://wiki.c2.com/?WasChryslerComprehensiveCompensationSucc...
https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...
These things are hard to get right in any complex enterprise.
It is also a political pork barrel project.
> The project was meant to save costs by firing 1,200 employees handling payroll at various departments around the country and replacing them with about 500 people in a centralized location using Phoenix to handle most of the government’s payroll needs.
What's not written there is that the decommissioned long gun registry was located in a rural part of New Brunswick. When it was killed by the conservatives, they had a big political issue on their hands because suddenly hundreds of federal government clerical workers were out of a job in this area. So what was the solution? Re-hire the same people and put them on the Phoenix payroll project. Never mind that not one of them had any experience with IBM, Oracle, PeopleSoft, payroll, finances, or anything else related. They wanted butts in chairs in front of keyboards and to keep those people on the federal workforce payroll.
https://www.google.com/search?num=100&client=ubuntu&hs=ssW&c...
quick edit, history lesson:
early 1990s, Chretien Liberals: we need a gun registry
liberal party: okay we passed a gun registry law
liberal party: we need a pork barrel project to make these people in small town new brunswick happy, let's hire 500 people in miramichi
liberal party: this gun registry has now cost 400% of what was originally budgeted
liberal party, paul martin government loses an election to stephen harper and the conservative party
harper conservatives: we promise to scrap the long gun registry
harper conservatives: we've removed the long gun registry but now all these people are out of work and pissed off
harper conservatives: we need to overhaul our old payroll system that runs on a bunch of old minicomputers, let's hire ibm
harper conservatives: let's re-hire those people we fired and put the payroll center in small town new brunswick!
harper conservatives lose the 2015 elections
trudeau liberals: wtf is going on with this contract the previous government signed. wtf is going on with this clusterfuck of oracle, ibm, peoplesoft and porkbarrel politics of employing people who are not payroll specialists.
trudeau liberals, later on: seriously this is broken beyond repair
> The Phoenix project was originally chosen by Prime Minister Justin Trudeau’s Conservative predecessors
This is assigning blame to Trudeau in the weirdest way.
1: the long gun registry was also a huge failed massively over budget project, so using the same people for Phoenix...
2: the long gun registry project was relocated from Ottawa to rural New Brunswick. Any competent civil servants switched to other departments to avoid the move. Only the civil servants without other options made the move...
I was working with one of the companies that came along to replace one part of the system with a much smaller component that just handled that individual piece and vividly remember the meeting with the IBM team. They had no code to show me, but they handed my a massive folder - probably verging on 10cm thick of use case diagrams. So many little stick figures staring up at me I was awestruck - but not in a positive way!
[0] https://en.wikipedia.org/wiki/INCIS
I remember interviewing a BA (business analyst) who had just come off of the INCIS project. She said their entity model was enormous, with tens of thousands of entities.
I couldn't understand how any project could have so many entities, let alone how anyone would ever hope to code up such a thing.
She said the commandment to the analysts was to model everything so they did, just like they learnt in data modelling 101.
So (INCIS was a project for the NZ Police), e.g.:
- A police station has zero or more cells
- A cell has one or more doors
- A door has one and only one key
- A door has many bars
Another data point in INCIS - my sister at the time worked in the police. Her workplace ran out of pencils (they did a lot of note taking with pencils and paper). INCIS had so thoroughly depleted the police's budget that there was nothing left for...say.. computers with email and the like.
The software itself, before fixing it, cost $182mm and covered payroll for 90,000 teachers. That's over $2000 per teacher. It cost another $45mm to fix it after this.
With that being said, IBM isn't a successful technology company with a proven record building good software products. They were an unwise choice from the get-go. IBM is a successful financial engineering and sales firm that sells things, then scrambles to hire/acquires to get things done to an ever-evolving, always justifiably rough spec. They quack and then start the hack-show.
What do you expect from the organization that invented "FUD" (https://en.wikipedia.org/wiki/Fear,_uncertainty_and_doubt) as a way to emotionally manipulate perspective clients into making a purchasing decision?
Except at $250-350/hour, below average performance is more profitable than successful and a lot of good people suffer on both sides.
It seems kind of simple in a way, if IBM was hired to build X and failed to do so, then there will be a law suit or some form of arbitration and some money will be recovered or something. It's pretty standard breach of contract type stuff. You can audit everything too, maybe IBM was billing for stuff they shouldn't have; I kind of doubt it but maybe there was something really amiss.
On the other hand if IBM was hired to build X, then it changed in to X' and then X'' and 10,000 changes later it turns our, Canada really wanted Y from the start, isn't IBM guiding them along their process to figure that out? It's an insanely expensive way to build and buy stuff.
Is there supposed to be some sort of internal customer's advocate that looks at the project and says "hmm, this is a payroll system, it shouldn't cost a $billion typically, or this was a $100m contract that is already up to $300m, something must be wrong..." Don't get me wrong, in most sales driven organizations, they're high-fiving whoever inked that contract and trying to get invited to the parties he throws on his yacht. The catastrophic failure sucks for everybody, it's hard to see someone else finishing the job more inexpensively that getting IBM to do it though.
What if X wasn't defined to begin, besides a high level of saying "one way or another we'll get you a working result".
Any reasonable firm is hired to map out X to Y as best as possible, or at least be sure they did everything to uncover the possibilities, as much as they may, or may not occur. I have a hard time believing IBM has only experienced this once. I'm not saying IBM would do this, but maybe it happened that certain risks or unknowns didn't get the full attention or support from either side.
Contingencies in contracts are often for when things go to hell, taking up any action or arbitration is still not getting the client the result on top of taking resources away.
Saying we tried when we took a lot of your money isn't exactly a satisfying experience and use of taxpayer dollar. If this had been in the private world, would it have happened as easily?
Your point about competency in procuring digital solutions is a neat one - I don't know of many organizations, be it commercial, government, or academic that are good at this.
I also want to clarify, I'm not saying IBM plotted X to Y in the most efficient way or that they did it when others could not in my last sentence. I was saying that now that Canada is a $billion deep. They have 3 choices: 1) kill it all and be $billion poorer and have no solution. 2) Somehow beat IBM in to a better deal and get a solution for like $1.2billion, I think news like this is part of the beating. or 3) Probably spend more than like $1.2b to get someone else to get it across the finish line or start over from scratch. Presumably they need something and at this point IBM is probably the closest to the finish line; change horses midstream and you usually get wet. Hey and let's throw out the other condition, IBM might own the product so switching vendors might be a defacto start from scratch.
Whoa, what? Sure, some of their achievements are in their past, but that claim really overlooks a ton of history.
If they want the benefits of sharing that brand between operations then they deserve the negatives that come along with it.
I strongly suspect they are subject to a variant of spectre meltdown. Yet, no news were made public on the topic. A tinge of "same cause produces the same effects" makes me think that they are immune to the scrutiny of security research on the topic because there is a tad of security by obfuscation, and secondly university and google don't care about mainframe. It does not mean attackers don't care about the backbone of banking system.
When their other ships were going down, they shifted to a growing area of demand, not an area of their expertise. A brand shell game.
How in any way is this "In IBMs defense"? We should think it OK that they pursued and committed to a large government contract that in reality they were not qualified to complete? The sentence reads more as a condemnation.
They messed up my own payroll, by sending too little, then too much, then we had a 6 month back and forth trying to resolve it.
prospective
> Some people were paid too much, others not at all. Those issues snowballed with the deluge of requests to fix incorrect paychecks.
> IBM said it’s “fulfilling its obligations on the Phoenix contract, and the software is functioning as intended.”
Okay, so the cost is coming from incorrect paycheck values, but the software is working correctly?
Does that mean it's human error? Can these issues be traced back to one or a few major instances of human error? Is this a widespread issue caused by a confusing or misleading UX design?
The article is quick to talk about the projects costs and failures as well as the politicians involved, but I can't learn anything or blame anyone until I understand what went wrong.
Of course, this requires testing and time to hash out. It would be a major mistake to scrap a project in it's final phase of refinement. They should have switched over slowly, department by department, handling and resolving issues along the way. But scrapping the entire project now only means another $1B investment and repeat down the road. Trudeau would join the ranks of the FAA in flushing multi-billion dollar software projects. It's a waste. I don't know about this project, but software engineers died on-the-job working on the FAA's AAS that never launched.
Now, I personally know people who've been getting incorrect pay for over a year, who submit a complaint to their HR, who pass it up the chain, where... nothing seems to happen. Presumably because the 500 people or whatever it is tasked to deal with this are literally dealing with tens of thousands of issues, many of which aren't trivial changes (and so block the many that actually are trivial changes, but aren't even being seen).
Basically it was released before it was ready, initial signs of issues were ignored, and now the human side of the system is overwhelmed. It's kind of like when a web server gets overwhelmed with traffic, and it isn't set to reject at a reasonable threshold, so things start getting queued and timing out, people start refreshing pages, and it just snowballs until the whole thing goes down. Which apparently is what has now happened.
Oh boy.
I've done a ton of projects in consulting, not IBM, but you've never heard of any of them, because they worked, and the customer was happy (although maybe some delays, or other randomness.)
It's self selecting that news is about massive failures.
What implementation of Peoplesoft isn't heavily modified?
Adapting business processes to COTS software packages is extremely challenging. Anyone remember the absolute bank that Peoplesoft consultants made back in the day when ERP and Business Process Re-engineering was all the rage?
1) Pay PeopleSoft/Oracle a metric f--kton of money.
2) A CD arrives in the mail. (This little envelope cost $800,000?)
3) Your PeopleSoft consultants show up, laugh at that CD (tossing it into the microwave) and pull out their own distros with a zillion customizations that they proceed to spray over your servers. You have no idea what they are actually installing.
4) You get out a firehose, fill it with money, and keep showering the consultants with cash until you are sure they won't succeed in delivering anything worthwhile, and you fire them.
I was only installing a local test instance of PeopleSoft (to do some integration work with another product). Somehow the PS consultants in the area got wind that someone in my company was doing an install, and I started getting all kinds of cold calls offering customization and optimization, with some of the most condescending and customer-hostile attitudes I've encountered. PS had pretty a pretty shitty ecosystem 15+ years ago, and it doesn't sound improved. I'd hate for anyone nontechnical to be at their mercy.
My company recently switched away from Peoplesoft fortunately.