Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

Saw someone buy a lot of bumper stickers to promote a political issue, without paying sufficient attention to the printing details. The ink was not UV tolerant. Used outside, the stickers became blank white within a week.

as i recall part of the slogan was "long term thinking"

I think you're probably asking about mistakes like deleting a database? But more often companies, orgs, or teams dying has to do with engineering management in my experience. I'll name a few I have experienced and you all can decide whether or not these count as "engineering mistakes?"

- Saying nothing about it, but "secretly" moving the team to another country by changing the manager and director, and not hiring anymore in this country, even when team members quit. It took awhile but I figured out that the plan was to let natural attrition take over and for every person who quits from here, hire their replacement there.

- Different company: after being acquired, the acquiring company communicated for eight months that no changes in staffing would take place, and that once the acquisition closes, everyone will be put on exciting new projects and a new bright future will emerge. Then, after the acquisition closed, laying off a third of the company immediately. Another third chose the exit soon after. Eventually, the company died.

- Third company: after being acquired by a large competitor, the director of technology for this third company promised them a new product for four years. I'm led to understand by those closer to the situation than myself that this director then proceeded to coast on vaporware demos for those four years, claiming the need to "pivot" or "reboot" the product as necessary, and promising more and more pie-in-the-sky fantasies until finally the gig was up and he was fired. That subsidiary also officially closed its doors eventually.

Just a few from my own experience!

To protect their identity I won't go into specifics, but not implementing anti-tampering on local and remote backups i.e. protection from root. Backups only residing on live systems. Not protecting systems from bad automation. Not deprecating old automation frameworks and continually adding new automation frameworks. I am intentionally excluding specific incidents.

Building/pivoting on the unverified assumption that a market demand exists.

Ignoring the cloud for too long.

Building the wrong thing.

Unintentionally making sensitive enterprise information publicly accessible & indexable/searchable thru a cloud service

Telling the entire engineering team that they need to move to a different city 1500 miles away.

Falling into the CMS trap[1] at a sensitive time in a startup can kill the business. In brief: when you try to build out too much complexity up front, everything easy becomes hard, everything hard becomes impossible, and generally any change takes too long. And since sometimes you don't have the luxury/runway to retry, you are stuck with it.

[1]: http://max.engineer/cms-trap

Attempting to rewrite a multi-million dollar application at the core of the business. Never designed to be an embrace-extend-extinguish pattern, so either it would all work perfectly, or the whole business would be sunk.

Running McAfee products on all production servers.

Lots of issues historically, and recently for that.

Palantir is operating under the idea that "The AI will figure it out once we can access all the data"

This is a mistake because:

1. A lot of the data at big orgs is garbage or only understandable within a certain context by specific people. 2. Internal politics within lots of organizations prevents access to this data. 3. The AI cannot just figure it out. You need tons of humans in the mix which brings you back to #1 and #2.

Not engineering, but CEO banking the entire future of the company on a partnership, where we were dependent on the partner company for future financing, but the success metrics of the partnership were entirely in the hands of their sales department.

Never bet the future of your company on metrics that are entirely out of your control.

Friend used to work for Knight capital that basically went bankrupt by inadvertently hooking up their test trading system to the live market.

I have seen a couple of startups that wanted build and launch their product as fast as they can. The product was supposed to be a SAAS but in that rush they have chosen technologies that made adding features very difficult. While they were considering and procrastinating adding features, competitions started to pop up and started to take over the niche.

The lesson is that, if you are just planning to make an MVP make sure you have your scaling figured out. The blue ocean gets red faster than you can imagine.

Well, it didn't end the company but only because the company was bought and bought again - then killed.

The new head of engineering had no experience in software. He developed a dislike for the language the product had originally been built in for his own reasons. He ordered a complete rewrite of the software in a different language. He hired contractors since we had no in house skills in the new language.

This lack of in house skills also meant that oversight of the code the contractors produced was poor. Several of us raised alarms over this but we were ignored. Eventually, the day came when the first paying customer was signed for the new platform. It was a complete disaster. The code was very unstable and full of bugs. The deadline kept getting pushed farther and farther out.

The new owners concluded the entire division was a failure. They fired the head of engineering who had been in charge of the disaster and his boss too. Then they started work on a new version of the platform using entirely different people. The ones left behind on the old platform (in the old language) had nothing to do but provide support for the dwindling number of open contracts.

Millions of dollars went down the drain, years of time were wasted, customers were badly served and a lot of people were incredibly frustrated, all because one executive decided to ignore the in-house talent and experience and follow his own inflated ego instead.

Not limiting access to production.

It’s both an operational and technical decision because it results in inadequate tooling and specials.

A service I know well had an issue, and as a result they started doing what modern services do - nuking things going it will clear as clean versions come up.

But unknown to the ops team, they nuked a bunch of custom stuff that no one knew how to build or really what they contained. Developers with direct access to production had rolled them out.

7 days of partial outage later they covered this up by hiding parts of the app until they could straighten things out.

Not sure if it will sink the company, but moving from very good native apps to a single pwa/electron style app that was poorly written.

The business justification seemly makes sense to have a single development train. But the execution was horrible. Bugs, horrible performance. Inconsistent features copied over from the various native apps, etc.

It was released as a major upgrade, but it was almost alpha grade in reality.

That org probably saved a lot in engineering, but the customer backlash was immense and likely cost them more.

(It was a very popular and beloved note taking app. that tried to become an all-in-one day planner/calendar/todo/note platform).

I think this community and the tech world in general overvalues engineering and I say this as an engineer myself. In my experience engineering or technology rarely seem to be the reason for a company's success or failure. There are certainly outliers in which a tech is so incredible that it alone can build a company. There are also some engineering mistakes that can be too costly to fix later and people can always be negligent and not do something obvious like proper backups. However those situations are usually extremely rare. The most common engineering mistakes can be fixed with more money. Typically some other factors are what make or break a company. That is what leads to not having the money to fix those engineering mistakes.

[Meta] I'd love the read the book with all the stories mentioned in the comments.

Worked at a startup that built a 3D modelling tool in the browser. They did a major refactoring that was still ongoing. At this point in the core team of 5 they churned through 10 engineers in 1 year. So when I started my job I learned that you could neither create a working build nor would any of the tests run. (Not to speak about the CI pipeline) Eventually the build could be fixed as well as the few tests. But as it turned out every basic user functionality had to be rewritten and the structural refactoring was still in progress.

IMHO the refactoring sounded very reasonable since the old code base was unmanageable. But there was a culture of extreme pressure from the CTO, making everyone haste through tasks preventing everyone to do "proper engineering". At the time the CTO was in some sort of permanent absence already. Afterwards the team lead left because of burnout, I also left and later on I heard they closed down the company.

Actually I've seen similar destructive refactorings at other places. At one they also had a lot of subtle problems leading to many user complaints. It could be fixed but by then it was already too late.

IMHO refactorings are great but it's always necessary to keep regressions in check all the time and really understand the design decisions of existing code.

A colleague accidentally swapped every product photo on one premiere league football team shop website with a kit photo of another premiere league team. He pushed up the change (with his testing image...) and went home without realising until angry phone calls started coming in.

I know of a company that was pretty successful with a VB6 app they wrote in the late 90s, but twenty years later they needed to modernize. They spent two years on a from scratch rewrite that was never completed (second system syndrome).

The company abandoned the rewrite, acquired a competitor, and rebranded it as the company's new product. It turns out the competitor's code base had serious deficiencies (couldn't scale to the company's size, had important bugs), and was very unpleasant to develop against. In the 6 months following the acquisition, well over half of the engineering team left the company (heavily skewed towards the highest-caliber engineers) because they couldn't stand what their jobs had become. Many long-time customers threatened to leave because of user facing technical issues in the acquired product.

A telling incident: the company needed to deploy an emergency fix to the mobile app. The only machine authorized to publish to the app store was a laptop that the owner of the acquired company had left at home overseas. To complete the deploy, he needed to call his wife and walk her through the publication process over the phone.

The company managed to fix enough of the problems that they're still around today, but for a while there was a lot of uncertainty around the company's future.