HN Explorer

The “Oh Shoot” Moment

3 points by bpyne ↗ HN
Seeing Adobe's problem with a release erasing content for a significant group of users made me think about the developers/engineers in the background. Did they have the "oh shoot" moment? I've had mine for much smaller issues. My heart raced, my mind raced with theories on what I missed, and I started to sweat.

What are your own "Oh shoot" moments? Did you suffer any consequences from management or anyone else in your organization?

4 comments

[ 4.4 ms ] story [ 15.4 ms ] thread
BlameKaneda
We have a dev version of our intranet and a prod one. To make a change to a page on the dev site, you drag and drop the HTML and JS files into a /dev-SiteAssets folder.

One evening I wasn't paying attention and accidentally dropped the files to the prod folder. I went to the prod page and sure enough, it was broken. It wasn't a high traffic page, but I was still afraid that we'd get a complaint or two.

Since the original prod files had been written over by the new ones, I went to the page's GitHub and downloaded the files from the repo.

I then dragged the repo files to the prod folder, and the page was back to normal. Definitely gave me a fright.

I informed my boss about what happened the next day and how it was resolved, and although he looked somewhat alarmed I didn't get in trouble.

lexfiend
25 years ago, junior sysadmin me had to restore our server's drives from tape. To my horror, all the backup tapes were corrupted. It took a while to figure out what happened; my 2am cron job script basically did this:

  run_backup & eject_tape
Fortunately, we were able to reconstruct the server's contents from individual workstations, but I had to endure "how's the backup?" ribbing for many months.

I also changed the script to:

  run_backup && test_backup && eject_tape
so if the tape was still in the drive when I got in to work, I knew to check on things.
AnimalMuppet
I was bringing up a new CPU board with the OS on a hard drive. It contained the only instance we had of the OS for that CPU board. And somehow I managed to delete the file that contained the OS kernel.

The board still booted, because the boot sector still pointed to the sector that the OS kernel file had been in, and the contents of those sectors hadn't changed. But the sectors were marked as unused on the disk, and could have been overwritten by any file write.

I booted for two days from that deleted file before I got to the point where I could re-create it.