Ask HN: We've all been there, what was your big stuff up?
To show that this sort of thing happens to the best of us, let's share some of our horror stories :)
A few months ago, I joined a new team and was still finding my way around the environments. I was tasked with performing manual deployments to a Dev, QA and Staging environment that weren't wired up to our automation system yet. We'd scheduled maintenance windows a week apart for the QA and Staging envs as we allow customers to test against these.
So the day of my QA deployment, I start by applying the database changes which all complete successfully. Next, I upload the new .ear files and deploy the new build of our web app. Again, all looks good, so I tell the QA team they can start testing.
Then the alerts started...
I deployed the app to the Staging env by mistake (and unexpectedly restarted the app server). I didn't realise the naming scheme of the hostnames indicated the environment in this case :/
Our UI broke immediately due to the schema changes, so my mistake was _very_ visible. I was lucky I could roll back the change easily, but I don't think I'll forget that day any time soon.
81 comments
[ 2.9 ms ] story [ 175 ms ] thread1993 I tar.gz'd it as I was leaving college and ftp'd that file NOT in binary mode; didn't discover it until too late.
1995, I blew away the mount point for my NFS server with all my home dir and data but had left the server mounted (and was running as root, no root squash, etc)
At work, training a new operator, I had them run the script that shutdown all web servers rather than regenerating the CMS caches on them. As the alerts rolled in, I reassured him that we'd done the right thing. Many minutes later, we looked at the logs and saw "webservers-shutdown-all" instead of "webservers-regen-all"
* (2010) When you're asked to restore last Sunday's backup to the dev CMS, make sure you're actually on the dev instance, and not, say, on the live instance. That literally every editorial person in the company uses. The day before deadline. (I got to restore 36 two-hourly incremental backups in sequence by hand. We lost only a couple of hours' work. But we verified our backups work!!)
* (2005) Never trust a UPS manual. Ever. Particularly, when it says that the "bypass" switch works smoothly, rather than, e.g., glitching the power and taking down all 75+ Windows PCs in the computer room. (The Sun boxes were of course unaffected.) Recovering the Windows network took most of the morning; the NT admins were less than impressed. And I was a contractor too. Fortunately working under the direction of the in-house admin.
The important thing being, of course, to recover and learn from the experience :-)
Cut to 1:30am after a full day of eking out 1 or 2 endpoints here or there, and I've figured out a new method to try. But first I need to test it and make sure it's not going to break anything else, so I create a separate asset group in the AV software and add only my machine to it. I add a simple "hello, world!" type script just to show that the script is executing and wait.
And wait.
No "hello, world!". It's 2am, I'm back in the office at 7am, my new insights will hold until tomorrow. I'm going to bed.
About 6:45 I'm in the queue at the shop to get coffee and bacon and my boss walks in for the same. We small talk and then he gets an incident call.
There's a virus affecting all of <locality's> machines. Uh-oh. He's getting ready to abandon his coffee and bacon aspirations (he's the Head of Security), when I ask what's actually happened.
"As everyone's logging in in <locality> this morning they're getting a command prompt pop up that just says 'hello, world!'"
Oh. Fuck.
I abandoned my coffee and bacon aspirations and assured him that this wasn't a virus, it was a misconfiguration that I'd made only hours before.
It was sorted within minutes and was broadly taken with good humour. But I was referred to as "World" for a while afterwards when people greeted me.
That feels meaner than it sounds...
I surely contributed to that list, but was too tired to remember any details. :)
sed -i -n '/foobar/p' /etc/passwd
trimmed the file to a single line's length. D'oh! Shouldn't have added that `-i`, should not have done that.
Luckily I could restore it from a (working) backup while still logged in. Phew!
The network had been rebuilt four times by three different people, and only half documented each time.
One time, each VM had been named after planetary bodies. Sol was the AD, Jupiter the print server, etc.
We found one called Mars. Completely undocumented. Doesn't exist so far as the docs knew. The previous admin didn't remember it.
I ran Wireshark, and got nothing.
So... I didn't just shut it down, but I deleted it.
Took 10 minutes for mass panic to hit the office.
Mars was the gateway for our publicly exposed servers. No website, no email, no VPN.
Our daily backup only copied data, not actual images.
So, just hoping, I threw a reverse pass through proxy up on the same IP, with routes for our servers.
Quiet returned, as I went about recovering the Mars image I had deleted.
Lesson learned: if you are working in unknown territory, let it break before deleting. Also, add VM images to the backup routine.
And a page in the documentation.
someone would inevitably restart the wrong one from the VM host, thinking it was the one they'd been SSHing to :/
Though we did, in the same mess of network, have dc01-<domain> and dc02-<domain>.
dc01 was the Domain Controller.
dc02 was... The backup of ad01-<domain>...
So, what do you know, I broke something in the source folder and the whole Trac install was unusable. I decided to nuke it and start again.
I'm sure you all know where this is going by now.
Back then, I had a habit of tryping ./* for anything in the current directory, rather than just * .
I forgot the .
Me being a total n00b and naive, I thought the permissions warnings I got were genuine (I didn't initially run the command as root) and that because I was chown'ing stuff to www-data... yep. sudo !!
And of course, even though --no-preserve-root was a thing even back then, that only works if the argument given to rm is '/'. Otherwise, bash resolves the wildcard and passes each one in.
It took about 5 seconds to kill my SSH session, just long enough for me to notice the missing . and go OHSHI-
Worse, the machine wasn't backed up. It had a reasonably concise wiki on the company in-house software. On the flipside, that meant the boss shared the blame with me because there was no backup. We were able to rescue the SVN repos, but the MySQL data tables were gone.
So I can totally relate to the poor Gitlab sysadmin who's probably suffering PTSD right now. For want of a single . I managed to trash a production machine too.
As one of my friends would later tell me, 'root is a state of mind'.
This script would do some stuff, and put a new website in place, and then remove the old one. So, my bash script had the line:
You can see it already. I ran it with $olddir unset. I think I had it in my head that the directory would simply not be found so that was fine. For those of you unfamiliar with bash, since olddir="", what actually ran was: Gigabytes lost (back then, a GB was a lot!). We had backups but they took hours to restore. Horrible, horrible day.It makes you think - if we had these threads more often, perhaps we'd all get to learn more about these little process changes that could avert a disaster.
Fortunately I had backups. But yeah.. don't do this.
I'd written a script to clean out /tmp (since it wasn't a virtual fs back then) at boot. Problem is that it hadn't successfully changed to /tmp but was running instead in /etc
Goodbye /etc, it was nice knowing you... first I knew about it was when my box spectacularly failed to boot.
However, this was _the_ best learning experience of my life. No internet (since that was my only computer at the time) gradually rebuilding /etc by hand from a root prompt.
So, I'd turned off the svn backups (dumps and post-commit incrementals) when the targets moved about a week before the final people move. We got into the new building and in the rush of getting everything setup, I'd forgotten to re-enable backups (had not made a checklist). Sure enough, svn server crashes, BDB corrupted, last backups about 8 days old.
Fortunately, we had nightly build snapshots, code on dev workstations, etc, so it was mostly a rock-fetch project to put things back together starting from a fresh repo. We had other automation that used the repo path and revision, so I created a "devtemp" repo and restored the backup re-imported all the code there and then laid on incrementals from nightly builds and dev workstations. In the process, I checked in the vast majority of our code as the author of "revision 6".
10 years later, I was still getting svn blame based questions "about this code you wrote (in -r 6)" "Man, that sokoloff dude wrote a whole lot of crappy code..."
Now that we've been mostly on git for 2 years and only have those repos for historical archeology, the questions are finally dying off.
That's fantastic haha
Or at least they tried to.
The backups system was incredibly wobbly at the time and would corrupt its archives pretty frequently which is exactly what happened. They lost everything on that server.
Did I mention that was their sole server and they had no other backups?
It turned out that they were a company providing services to a government entity and had some pretty strict record-keeping requirements which they relied on our service to fulfill.
I was freaking out thinking I was going to be fired after being there for less than a year but everything was resolved fairly well (somehow).
I learned to never trust backups and the rule of thumb "two is one, one is none" as it applies to them.
To this day I have no idea why I typed `rm -fr /var/lib/mysql` instead of `cd`. I blame muscle memory. Good news is MySQL pools its open files. This means the database was operational (but slowly failing opening less used tables) for 15 minutes or so.
We quickly promoted a database slave to primary and took the old one down for backup restore. I went for a teary walk. Shaken.
At the postmortem we concluded we needed more metrics from MySQL to avoid SSHing into servers. We also concluded how error prone one-off commands are. Personally, I'm much more careful nowadays.
I waited until everyone else had left the building, grabbed the disk out of the old server, stuck it into the shiny new server and proceeded to dd the old disk to the new disk.
Except I got the devices around the wrong way (/dev/sda , /dev/sdb) and proceeded to copy the contents of a blank hard drive over the top of the old server's drive. Didn't notice until the process had finished...
I then discovered the benefit of DR plans the hard way (backups are useless unless you test a restore).
Long story short, I managed to recover most of the files using a variety of disk recovery tools, but I was still in the office the next morning when other people started arriving and began to ask me why, for example, the payroll application couldn't find it's database. I spent the next few days in panicked forensics mode until the company was operating to everyone's satisfaction.
When I left that company years later I had implemented many redundant layers of backups, proper DR plans that I religiously followed, and developed a meticulous habit of testing any commands that needed to be run on any production server.
dd always makes me nervous as hell when I need to use it. I usually end up checking four or more times. Still got it wrong a few times.
Nothing like having to recover data with forensics to make you build a fantastic backup system with great redundancy.
Time to use it!
Nothing happens. It needs some time to read all users' configuration files before displaying the user interface, but it's taking too long. What's happening? I decide to interrupt it. Shortly after, we find out all user accounts starting with A, B, and C are wiped out.Apparently, unbeknown to me, somebody had previously written a utility to delete user accounts. It was named uc (user clear), and was installed in /usr/bin. Fortunately we had fairly recent backups.
That day I learned about hash -r.
All products at Tesco have an 8-digit product number (SKU) in addition to the EAN/UPC. There's also a three digit case size number. Like this:
05123456-024
Each product has an estikmated weekly sale and a capacity (shelf plus warehouse) to aid efficient warehousing. Each product has a case size of less than 1,000. Well, all but one -- white sugar. That has a case size of 1,024. It's annotated on the shelf ticket as '024', dropping the leading '1'.
I didn't know about this until ~43 tonnes of sugar arrived on 6 trucks the following day. For a new store. In a small town.
It turns out that me misreading '024' as a case size and over-ordering sugar by a factor of 43x was enough to have the internal ordering software updated.
Were you at the new store when the trucks arrived or did you get told afterwards?
Were there negative consequences for you personally or was it considered a process/tooling error only?
No obvious repercussions. This was peak Tesco profit era, so it was considered a blameless error (AFAIK).
I read about this happening to a novice commodities trader in London: several barges full of coal supposedly showed up at their building at Canary Wharf.
(Oh drat, I looked it up and the only reference I could find was on DailyWTF so...maybe not so true: http://thedailywtf.com/articles/Special-Delivery )
Thankfully I had been involved in a migration of the svn server between regions a few months back and in paranoia had tested the hell out of the backup and restore process, and that most repositories had been migrated to git. Still it did stop my heart for about 20 seconds when I realised my mistake.
At the very first job I ever had, with literally hours on the job, my employer somehow put me (at 17, with zero routing experience) in charge of a major change to the primary yet supposedly redundant ISDN routes at a small ISP. Needless to say there were some unhappy customers the next day or two as incorrectly advertised routes drew traffic down the wrong pipe, we figured that out, updated correctly and propagated. Luckily most customers were schools, internet was not yet a critically expected utility, and we buggered things up on a Saturday, so the majority of the customer flak was only on Monday morning. I can't remember what the routing protocol was in those days, but it was probably something early like RIPv1. Meh. Certainly a learning experience I felt bad about, but in hindsight totally not my fault!
Another one which was less my fault but I did blame myself for was dropping a server with 200.000 web sites on it because we had to move datacenters and it was xmas eve and very very slippery with ice. We slid and the server fell which wrecked the (hardware) RAID disks. This had working tape backups so there was a few hours downtime which was going to happen anyway as we were moving.
Now that I am writing this anyway; the most traumatic was mid of the 80s with my second computer when I was 10. I had one disk(!); they were expensive and I did not get a lot pocket money. I was learning assembly after Basic became too slow for what I wanted. I was building a game (Chuckie Egg rip off) and after a long time not saving I ran the game and it worked. I was happy and saved the game on the one disk with all the software I wrote with the save command. When I pressed enter I remembered, and I remember this very vividly, that I used the the disk basic ram space because I was running out of memory. The disk started spinning, computer rebooted and ... files (dir) command after gave a disk I/O error... The misery.
Edit: ugh. Just remembered a 1984 one; my father brought home a modem for the MSX-1 and those things were, for my notion of money, kind of bare gold, price wise. It was 100s of guilders. But it could only do Viditel. Which sucked. I wanted BBS access and that required shoving the thing into the MSX after Basic was already booted. The MSX cartridge ports are connected intimately to vital computer parts. So shoving it in crooked had my best friend looking at a purple screen after that I decided better would be to solder in a switch. I did that before with stuff I found by the road. I had to cut an IC pin to do it which I had done quite often; this time I cut it and it flew off... Eventually I was forgiven.
This shouldn't have been too much of a problem because a full dump of the MySQL database was made every night, copied offsite with scp, and kept for a week. But, when the time came to restore the affected table from one of these dumps I found that every dump for the last week had timed out silently and was incomplete.
The result was the best part of a week trying to reconstruct the missing parts of the table from what could be found in the dumps and to restore it successfully. I got most of it back, but we couldn't do much work for that period.