Ask HN: What's the worst you've ever screwed up at work?

Many years ago, when I was but a fresh faced idiot, the partition that contained the mSQL database which had All The Data filled up. I moved it into /tmp because there was plenty of space.

On a Solaris box.

Hilarity ensued when we next rebooted it.

About 30 years ago I deleted the JOBCONTROL process on an old VAX 11/780 thinking it might be the reason why someone's process was stuck.

It wasn't a but an hour before I lost sysadmin privileges.

Never "experiment" with a production system - ever.

The worst thing I have done is terminate a running production instance with no database backups.

Client, not happy.

In late 2008 when I was in the Marines and deployed to Iraq I was following too closely behind the vehicle in front while crossing a wadi and we hit an IED (the first of 3 that day).

Nobody was killed, but we had a few injured. Thankfully the brunt of it hit the MRAP in front of us. If it hit my vehicle (HMMWV, flat bottom) instead I probably wouldn't be here.

That was the first major operation on my first deployment, too. Hello, world!

My takeaway? Shit just got real.

We ended up stranded that night after the 3rd IED strike (our "rescuers" said it was too dangerous to get us). It was the scariest day of my life, but in similar future situations it was different. I still felt fear and the reality of the existential threat, but I accepted it. It was almost liberating. Strange.

I deployed for another year after that (to Afghanistan that time). After Afghanistan I left the Corps and started my company. Because if it fails, what's the worst that can happen? Lulz.

First job, circa 2000, at an ISP that was run very clearly as a business and cutting corners. Not only was it critically understaffed, but management was more interested in laughing their way to the bank than management. They had me - with literally no routing protocol experience - manage a live route advertisement transition between two peering providers. Result: all customers offline, ~24 hours.

Reaction was standard: mostly to point out I did my best in unfamiliar territory and things should be sorted soon.

Take aways were: (1) less support calls than expected - users put up with things. (2) you learn when you fail (3) always have a backup

They kept me on at that job but I left pretty soon anyway as I got a 'real' (as in creative) job hacking perl-powered VPN modules for those Cobalt Raq/Qube devices, and building a Linux-related online retail venture for the same employer ... that worked great, but failed commercially.

In 2001 my first IT tech job as help desk analyst I heard beeping in the server room on one of the Solaris/Oracle machines and pressed the power off/power on button on the chassis. DBA came running in and I promptly left saying "oh I think it rebooted itself". The company went bankrupt shortly after so no huge lashing came my way but all my more experienced friends where like "wtf never do that again!"

Introducing a master/minion update system to work I ran a batch update to take a certain percentage out of the cluster.

Unfortunately I got my selection criteria wrong and pulled out all of one cluster and half of a second, halting a few thousand operations.

Luckily the monitoring system was very quick to alert me of this and using the same (wrong) selection criteria it was a fairly simple process to stop the update and put them all back in the cluster.

Takeaways? The age old cliche of "With great power comes great responsibility". Oh and have good monitoring!

git push -f

A local Subway franchise was the very first company that hired me. I was extremely young, shy, and intensely socially awkward, yet excited to join the workforce (as I had my eyes set on a Pentium processor).

When I worked at Subway, the bread dough came frozen, but you would put loaves in a proofer, proof it for a certain amount of time, and then bake it. My first shift, however, got busy and I left several trays in the proofer for a very, very long time. Consequently, they rose to roughly the size of loaves of bread, as opposed to the usual buns.

It was my very first shift alone at any job in my life, so I did the most logical thing I could think of and put the massive buns in the oven. They cooked up nicely enough and I thought I was saved. Until I tried to cut into one.

Back in that day, Subway used to cut those silly u-shaped gouges out of their buns. In retrospect, I think this was most likely a bizarre HR technique designed to weed out the real dummies, but at the time I was oblivious (likely because I was one of the dummies they should have weeded out). When I ran out of the normal bread, I grabbed one of my monstrosities, tried to cut into it, and discovered that it was not only rock hard, but the loaf broke apart as I tried to cut it.

That night, my severe shyness and social awkwardness had their first run-in with beasts known as angry customers. I was scared I would get fired, so I promptly made new buns, but spent the rest of my shift trying to get rid of my blunder. I discovered some really interesting things about people that night. First, you'd be surprised how incredibly nice customers are if you are straight up with them. Some customers I never met before met the big, crumbly buns as an adventure and, in doing so, helped me sell all the ruined buns.

In the end, I came clean (and didn't get fired). That horrible night was a huge event in the dismantling of my shell. It taught me an awful lot about ethics. And frankly, that brief experience in food service forever changed how I deal with staff in similar types of jobs.

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remove when you're done typing the query.

  SELECT * FROM table AS OF TIMESTAMP some_timestamp;

I sent an email to three thousand insurance agents informing them of the cancellation of policy number 123456789, made out to Someone Funky. I learned to appreciate Microsoft Outlook's message-recall function, which got most of them. I also learned that just because you're using the test database instance doesn't mean nothing can go wrong.

easy:

me: "unix definitely won't just let me cat /dev/urandom > /dev/sda"

other: "sure it will"

me: <presses enter>

what I learned? unix will absolutely let you hang yourself. 1998, production server for a fortune 5 company.

During a server migration for our web based file sharing system our lead engineer (at the time) forgot to ensure that all cron jobs (for cleaning up files and sending out automated emails) had been turned back on.

Queue me 7mos later reviewing the system. Realizing that critical jobs were no longer running and that our users were all essentially receiving 100% free hosting for however much storage they wanted. SOOOO i turned the jobs back on.

The lead engineer before me left no documentation of what the jobs did other than that they should be run. In my stupor i did not review the code. The jobs sent out a blast of emails warning that files would be deleted if not cleaned up or maintained. Then seconds later deleted said files...

We nuked around 70GB worth of files before we realized what happened. WELL GET THE TAPES! Turns out our lead engineer ALSO forgot to follow up w/ system engineers and the backups were pointed at the wrong storage.

No jobs lost, thankfully the manager at the time was a word smith of the highest degree and can play political baseball like a GOD.

Tried to prevent a massive product failure.

It failed anyway, but I wasn't around when it did and there would have been no "I told you so" credit even if I were.

One of those "big company" lessons, but probably applicable to startups (which have an even higher ego density).

One time I tried to change a column name in a production database. I learned that when you change a column name, mysql doesn't just change a string somewhere, it creates a new table and copies all the values from the old table into the new one and when that table has millions of rows in it, it really slows down your production server.

I was in a remote meeting and failed to realise my laptop's camera was broadcasting. A roomful of people saw me, clad in horrid workout clothes, jam my finger up my itchy nose and scratch my balls.

Key takeaway: always check the cam.

I asked them for a job in the first place.

Not the worst at all, but probably one I found most amusing. One of my jobs included some sys admin tasks (this wasn't the position, but we all did dev ops), among my other responsibilities. I spent half a day going through everything with the person responsible for most of the admin tasks at the time. She was an extremely dilligent and competent admin, did absolutely everything through configuration management and kept very thorough personal logs and documentation on the entire network. One of my first tasks was to change backup frequency (or other singular change) and going by how I usually did things at the time, just sudid a vi session, changed the frequency and restarted the service.

She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows and gave me a look. She quickly reverted what I did, updated our config management tool, tested it, then deployed it, while explaining why this was the right way to do things. I slowly came around to doing things the right way and haven't thought much about the initial incident until we found her personal logs that she archived and left on our public network share for future reference.

In the entries for the day that I started, we saw the following two lines:

    [*] 2007/09/09 09:58 - yan started. gave sudo privs and initial hire forms.
    [*] 2007/09/09 10:45 - revoked yan's sudo privs.

One summer in college, I got an internship at a company that made health information systems. After fixing bugs in PHP scripts for a couple weeks, I was granted access to their production DB. (Hey, they were short on talent.) This database stored all kinds of stuff, including the operating room schedules for various hospitals. It included who was being operated on, when, what operation they were scheduled for, and important information such as patient allergies, malignant hyperthermia, etc.

I was a little sleepy one morning and accidentally connected to prod instead of testing. I thought, "That's weird, this UPDATE shouldn't have taken so long-oh shit." I'd managed to clear all allergy and malignant hyperthermia fields. For all I knew, some anesthesiologist would kill a patient because of my mistake. I was shaking. I immediately found the technical lead, pulled him from a meeting, and told him what happened. He'd been smart enough to set up hourly DB snapshots and query logs. It only took five minutes to restore from a snapshot and replay all the logs, not including my UPDATE.

Afterwards, my access to prod was not revoked. We both agreed I'd learned a valuable lesson, and that I was unlikely to repeat that mistake. The tech lead explained the incident to the higher-ups, who decided to avoid mentioning anything to the affected hospitals.

If it's any consolation, the company is no longer in business.

Just remember when you screw things up: Your mistake probably won't get anyone killed, so don't panic too much.

  DELETE * FROM TABLE x
  131054 rows deleted
  WHERE a = "foo"
  >> Malformed query <<

I wrote an update script for a database table not realizing I had the key wrong (I'm kind of fuzzy on the details, but essentially I think it was a composite key but I was only using one of the columns in my WHERE clause...) and accidentally updated all customers addresses in our database to the addresses of one account.

Luckily we had backups from that morning so we only lost any address updates people would have done that day, but it made for some interesting customer service calls for awhile...

I may, and/or may not have caused a production site's PLC to go into STOP mode during daily operations while making network updates remotely.

Possible outcomes of unplanned system haults include plugged machinery that would need to be manually cleared, mixed products which would become immediate net losses for the company and damaged motors.

Thankfully no product was being run at the time. I have also implemented changes across the board to our client sites that prevent this type of shit from ever happening again. You know when you look at a system and go "this is going to bite us in the ass eventually?" This was one of those systems, they just needed a new hire to give them the push.

I wrote a piece of code controlling an assembly line machine. These machines require manual operation, and would come with a light curtain, which detects when someone places their hand near the moving parts, and should temporarily stop the machine.

A relatively minor bug in the software that I wrote caused the safety curtain to stop triggering when a certain condition was met. We discovered this bug after an operator was injured by one of these machines. Her hand needed something like 14 stitches.

Lessons learnt:

1. Event-driven code is hard.

2. There's no difference between a 'relatively minor' bug and a major one. The damage is still the same.

Non-tech-related:

I was doing HVAC work while I was in college and we were removing an old air handler from underneath a house. Just inside the crawl space, under the access door was a water pipe. My boss told me to make sure I held it down while we slid the air handler out through the hole. I lost my grip on the pipe and the air handle snapped it in two, at which point gallons of water began to gush into the crawl space.

I ran for all I was worth to the road, which in this case was about 600 feet away, to turn off the water at the water meter. I ran up and down the road in front of the house and never found the water meter. So I ran back to the house and inside and told the homeowner who promptly informed me that they used well water. She called her husband and he told us where to turn off the well pump.

It wasn't really that bad in the grand scheme of things but letting the homeowner's water gush under the house for about 15 minutes does not bode well when you are supposed to be there to fix problems not create them.

Early on in the implementation of one of the PKCS "standards" while at a browser company many years ago, due to an improper interpretation of a spec that was still in flux. There wasn't enough testing and "release bits" went live.

I had to quickly get a patch in for the improper code and had to maintain that buggy implementation. In addition, the "standard" itself got a rather scathing write up from Peter Gutmann, which is completely valid:

https://www.cs.auckland.ac.nz/~pgut001/pubs/pfx.html

This is a critique on the "standard" itself, the process was just as ugly.