I wonder if kids these days manage to get as much experience recovering a borked computer? I vaguely remember poking at autoexec.bat and config.sys files to get games working, or to attempt to revert some screw-up when I failed to get one working. Or becoming very good at using Linux live CDs to grab personal docs/tax information/business slips... off a hard-disk before wiping it and reinstalling Windows, after a bad driver update or dodgy install of some software.
The first few times, you get that icy "oh shit" feeling, but figuring out how to fix it, and that it can (almost) always be fixed, is invaluable.
> Since then I’ve adopted custom prompt and color scheme for production.
An AHA moment for me. I've always wondered why people cared about all the terminal customization options (I grew up typing on an 80x48 AAA glass TTY). This is a good reason.
I've used a custom prompt with the host name built in for decades, but this kind of color issue makes sense!
I always have so many terminals open that this is a must for me. I have different colors for production and I used ansible to make them consistent. It's nice to not have to waste brain power trying to guess which terminal I'm on.
> I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages
This is everyday Operations work. There is no school that teaches the eclectic mix of OS and container, networks and routes, app/db/code, CI/CD, BC/DR, writing KB articles, running standups and meetings, triage technical issues, configuration management, Change Control, and Incident Management.
These diverse skill sets are often called "Devops", but nobody makes a Devops degree. The only way to assemble this experience is while working to support developers.
I also wonder how effective a program can be at teaching this kind of thing. No amount of studying ever ingrained something in my mind as well as the adrenaline shot of somehow fucking up production. I'm not aware of any pedagogical methods to engage the endocrine system, but I find it very useful for retention.
Well, part of the learning is making the mistakes yourself, and part of it is your paranoid coworkers cautioning you against certain behavior and relaying the horror stories of what it might result in as experienced by themselves or someone they knew. A thedailywtf.com for operations mishaps would probably go a long way.
There's a theory that the same person should not be both a Developer and an Operator for the same project - that crucial mental separation between Making something and Keeping It Online gets too blurred when one person wears both hats. I've witnessed this in my own personal and professional projects. I guess it is a counter-argument to the DevOps/Agile philosophy.
This is why I do a "mv" to "tmp" instead of using "rm" when I can. Having special messages or colours to differentiate between production and other environments is a great idea as well. Generally if I'm about to do something on production, I'll close everything else to do with other environments as it's just too easy to slip up.
You can also alias "rm -i" or "rm -I" to "rm" to avert major catastrophes.
From man rm:
-i prompt before every removal
-I prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes
This alias seems to be set by default in many linux distributions these days (and has been that way for many years now).
The problem is that you get used to this crutch and rely on it, until one day you're logged in someplace without the alias and end up deleting something you shouldn't.
I'm not a fan and normally disable it. Instead, I prefer to be very careful and deliberate when using rm, especially with any wildcards, making sure I'm in the right directory and that the pattern matches what I expect it to.
> This alias seems to be set by default in many linux distributions these days (and has been that way for many years now)
Which Linux distributions do this? I have not seen this alias set by default on any distribution that I've used recently (Gentoo, Linux Mint, Debian, Ubuntu, Crunchbang, Arch).
I stand corrected, it doesn't seem to be the case anymore.
I can find old rpms and deb files with rm aliased in /etc/skel/.bashrc or /etc/skel/profile/aliases.sh, but the most recent I found is in a bash 3.2 package, so this practice seems to have stopped quite a few years ago.
I'll probably get shit on for admitting it, but I generally keep rm aliased to `rm -rf`. Recursive, force.
It forces me to be careful and deliberate. There's never a safe option or a backstop. I treat rm like a gun - always assume it's loaded and going to murder whatever I point it at without hesitation. If I'm not 100% sure about what I'm about to delete or how a wildcard will expand, then I echo it first.
I messed up once, about 15 years ago now. Lost a few personal files. Haven't screwed up since.
yeah, that'd be great. but our staging units (i don't deal with production units) come and go so often that keeping a list of hostnames became unrealistic.
Is there something in your dev process that would actually require you to implement it with a list of hostnames, rather than just putting the "staging" version of .bashrc onto the staging machines, and the "dev" version onto dev machines?
I used to setup my shell so that the text on the production servers was red and local it was white. Made it easy to know which machine I was tying into :)
> I used to setup my shell so that the text on the production servers was red
I do the same. People always walk up and make some snarky comment like "wow, nice background color" and then I explain why I have a red terminal background and they say "oh... that's a great idea"
I tried this for a while after a goof up. However because my dev environments more or less looked like default, if I forgot to setup a new production server to change the color, I'd mistake it for local at some point.
So I change my dev to be the non-default now. If I see the default/typical color scheme, I assume production.
To prevent this, my iTerm2 has a custom background enabled when logged in on any production servers.
I wish the same were easily possible on a Linux terminal without much "trickery" :|
I have a different 'theme' for tmux on any production servers I set up than development/test servers and my local machine. It ends up that I nest my remote sessions in my local, but it's effective.
Whats wrong with just setting PS1 to indicate that its a production server, and using a decent host naming scheme that properly indicates the function of the machine you're logged into? I mean this is what I do for my Linux machines, and I haven't deleted a Linux production machine in decades ..
Maybe this is an obvious question but...how do you do that if you ssh into your remote machines through a bastion that normalizes everyone to the same end user? I'm unable to set a custom .bashrc on the remote server, because its a shared user, so how would you change the ps1 on the remote box?
One the last day of a colleague of mine, he decided to run `rm -rf --no-preserve-root /` to see what actually would happen on his workstation (it was going to be formatted shortly after). We were all standing around talking waiting for whatever would happen to happen before we saw him off. A few minutes in I asked if he had unmounted our NFS volumes. Nope. He quickly pulled the power, and that was that. I have no idea if anything was deleted, but he probably had ~ 2PB of mounted data attached to his workstation. Everything from build archives, customer logs dumps, test artifacts, dogfooding clusters (the product was a parallel distributed networked file system), and possibly a few random other things.
No idea if anything would have happened, most of the data was read only (builds, logs, etc) but some was not.
This is why --one-file-system should be default, with a warning if another file system is encountered that's skipped so if it's what is desired you can re-run it (or umount and try again).
> Moral of the story - Don't let anyone give you access to production servers.
In my case, it was "Don't let anyone have you do testing/debugging on the production machine."
I was 18, first dev job, small mom-n-pop which developed insurance claims management software. Early 1990s, green screen VT100 terminals connected remotely via modems and dedicated phone lines.
Anyhow, something was wrong, I forget what, and to fix it I needed to debug using a real dataset. Between each run I had to clear out and rebuild a series of test "tables" (that's in quotes because the system was PICK, and the DB isn't exactly a SQL RDBMS), then copy the production tables to the test tables, and re-run.
Well - at one point I cleared the wrong table - cleared the production and not the test table.
Within seconds all the phones lit up in the office. I groaned (rather loudly), because I knew I just dun f'd up. The client was one of 3 or 4 that the company had, and the largest. Needless to say, for every minute they were down, they couldn't process claims, and were losing money.
Did I mention there weren't any backups? Not that backups weren't made, just that they were "end-of-day" backups, so the backup for that day's work (head's down manual entry of claims for processing) had not yet happened.
Fortunately for me, the table I deleted could be recreated (mostly) from two other still-intact tables. My manager and I worked thru the evening rebuilding the table as much as we could, and also pulled in data from the prior day's backup to make it as complete as possible. Even so, they still had to re-enter some data lost that wasn't recoverable by that method.
Did we set up a test server or do anything else that any sane company would do after that incident? Nope. Things just continued on in the same manner, and somehow I kept my job. Honestly, we should have been doing the testing in-house, but this was using an IBM RS6K box that at the client was much larger than what we had as a dev box (we were using the smallest version of that machine at the time, which was meant to be a single user desktop workstation as I later found out), so we couldn't fit the data onto the hard drive, even if it was possible to download it all via modem (it wasn't - I'd probably still be waiting for the dump to complete given the modem was only 9.6 kbps). I have no idea what it would have cost to do it right, but the drives needed back then, coupled with the infamous IBM service contract - meant we couldn't just hop over to the nearest Insight to grab a cheap hard drive and slap it in.
Today, I'd probably have the admin spin up another instance of the PICK system as another process on AIX, and (somehow) replicate the live data/code over and run in that instance. Still not "safe", but probably would have been much safer than what we were doing. But alas, I was but a kid at the time, and barely understood *nix (but that was where I first learned about it - and I also gained a healthy lesson/understanding/fear/respect for terminfo - for the purposes of supporting a variety of serial terminals and PC terminal emulators).
I've seen a DBA drop a production table as well, fortunately it was a small in memory table for non persisting data that could be quickly repopulated. Still lead to about 5 mins of downtime.
I always have two terminals open, both running tmux. One is white text on black for dev work, the other is white text on red. I only ever ssh into production servers on the white on red terminal so there's a strong visual signal that I absolutely shouldn't be running drop, delete, truncate, rm, dd, whatever commands on them and if I do then I am extremely careful.
I used to do the same thing with RDPs to windows production servers. I'd set up a different colored color scheme on each computer, with prod being bright red :)
For Windows work, I made sure to have one regular user and one admin user. And set the desktop background of the admin user to red. An additional benefit is that the garish background is uncomfortable to work with, and so discourages being logged in as admin all the time...
Changing the desktop background alone isn't enough. You will inevitably get into a situation where fullscreen windows are occluding it. To avoid any ambiguity, set the color scheme to high contrast and then set all the windows title bars to red, and tint the task bar red. On QA, I tint everything purple, and on DR I tint everything green, or yellow.
I once got a call from someone with DBA privileges: "I just ran 'SELECT table_name FROM user_tables' and it says 'no rows selected." I logged in and checked, and said, 'Yep, you have no tables.' Fortunately, Oracle by then had the RECYCLE$BIN, and the interruption in service was minimal. I'm not sure how he dropped all the tables.
And let's not talk about my own (mostly minor) disasters.
I was on my way out the door at my first job, and an SEO person stopped me and asked me if I could fix something. I just needed to delete a single row from a database. Easy enough. I get as far as “DELETE FROM table_name” and for some unknown reason run the damn thing without a WHERE. All the rows, gone!
We did have nightly backups but the guy with access to them was notoriously hard to get the attention of and lived in a different state. I was on my way out the door, and now I ended up being stuck there for another few hours trying to get ahold of devops.
At the company I am at now, we had a devops person whipe our multi TB CDN on their first day trying to make an improvement to our asset sync script. It took probably 2 days to get the whole thing reuploaded.
Yes, I learned this from a sysadmin in the early 90s and did this when learning to use Unix and having used the technique in various other contexts (e.g. database DELETEs), I am confident this one weird trick has saved me from catastrophic mistakes countless times.
I count to five before hitting enter. I have stopped on 3 or 4 a number of times. 5 seconds is frequently enough time for your brain to catch up with your fingers. And nobody is going to notice it took you 15 seconds to do something instead of 10.
That works halfway for UPDATE statements as well. At the very least you can see which rows you’ll be updating.
Another tip is to write the WHERE clause and SET clause prior to filling in which table is being updated. That way you can’t accidentally run something prematurely.
Unless, like me, you then highlight the statement without the WHERE clause and run it, realizing your mistake a split second before you hear the key click...
I have a couple of SQL habits along these lines that I haven't seen others do.
If I'm typing an ad-hoc DELETE or UPDATE statement, I'll slap the WHERE keyword on the end of the main command line. For example:
UPDATE items
SET
active = 0,
deleted_at = now() WHERE
category = 'Foo'
DELETE FROM items WHERE
category = 'Foo' AND
customer = 123
Yeah, it looks ugly, and I wouldn't use that formatting for "real code," with tests and whatnot. But for messing around in the query editor, I just find that I'm much more likely to accidentally leave off whole lines. I never accidentally select parts of lines. More generally, the idea is to make your partial query invalid syntax, if you leave off the last line.
I also tend to wrap destructive DML in comment blocks. So in my query editor, I'd actually have:
/*
DELETE FROM items WHERE
category = 'Foo'
*/
The idea here is just that I want to be required to select this text and run it. If I have some other queries in the window and somehow accidentally run all statements in the window (didn't select something I meant to or maybe a query is hiding beneath the scroll line), I don't want destructive queries to run.
EDIT: I've actually never made a catastrophic mistake of this nature, but there have been some very close calls. And I've seen it happen several times, from excellent programmers who just made a mistake.
I did this on the users table in production but I was fortunate enough to have started a transaction. I was just about shaking when I typed 'rollback'.
The fact that I was entering a delete -- instead of setting to inactive and doing it in production from the command line should give you an idea of how likely the backups would have been effective. This was in '97, though when everything was seat of the pants during the full internet craze.
I worked at a public company where write access to the production databases was required to go through an intranet site which logged the query, required a code review id, and limited the number of rows that could be deleted. It was more or less for sox compliance reasons, but it provided a nice bit of safety net around dangerous queries too.
It slowed things down, but the forced code reviewer process saved me once in a big way.
You're likely going to mess up sometime. Have a backup plan.
I too have "restored" a database to the wrong server (production was now development). This was a member list for an arts organization... After about 20 minutes of panic (what is going on....) I remembered I had set daily backups. whew.
I was logged in over VPN to a client machine. I suspected the problem I was fixing was something to do with the network misbehaving. It was a Windows machine.
I thought "I know: I'll disable and reenable the network device!".
A few moments later: "Oh. Bugger.".
Shortly followed by an embarrassing call to the client's IT department to ask them to reenable the network device on that machine...
Our 3-year-old production server hard drive, containing all assets and a database for multiple customers, died. No backups. For a few hours I was picturing not only the anger of our customers, but the demise of the whole company. Luckily the other drive in the RAID was OK and we didn't lose any data.
Probably one of the stupidest, but also the most personally effective way to learn anything.
Account created an hour ago, and the only comment from this account is this one you’re looking at (which seems fine by my standards), and it’s insta-dead? I vouched for the comment, but am I missing something or is HN’s juvenile shadow ban broken again? (IP ban, maybe?)
If you have never screwed up anything in production, no one at your org trusts you enough to give you production access. Which is ok and great for you, but at the end of the day SOMEONE has to have production access. You can write a devops template that wipes out data just as easy as you can sudo rm -rf /
I remember at least one prod incident where a script made it out into prod and wiped out 100s of servers. Something like it had rm -rf /$SOME_ENV_VAR which was set in QA but not prod. Opps....
I am much more terrified of Terraform than myself, especially `terraform destroy`. I mean, a human can only do so much so quickly but a script could kill everything semi-instantly.
I look at `terraform plan -destroy` and it still terrifies to destroy an internal service.
After some bloopers early on in my career I developed a systematic "back it up" reflex if there's even the remotest chance of something being lost/broken. This has saved my bacon on countless of occasions... normally from problems I least expected.
Tried to get around a firewall for a windows remote desktop, changed the port number through registry, logged out, could not log back in. Was not able to change it either, detached the AWS volume, attached it to another device, could not access the registry to change the port back... I could see the screenshot, machine is running but couldn't remote in.
Doing a recursive sudo chown of the /etc folder, locked myself out of that ubuntu server.
Wrote a parser that had a condition that would not end, would watch the server spike to 100% usage.
We had weekly releases to production. during my first month I checked in some code without yet fully understanding the spaghetti'ness of the 20 year old codebase.
it turned out that our beta testers found a lot of cases that were broken on Friday when the release was next Monday. The problem was that a ton of other classes pointed to the one in which I made changes and altered the state..
Worked through the weekend with my teamlead and got it fixed somewhere on sunday afternoon.
good bonding experience though. All had a laugh about it the next week. :-)
Speaking of backups :)
We had a very large mission-critical system that we developed and hosted for a fortune 50 client.
Long story short - prod DB with tons of records took a crap, proper RCA was never done I suspected HW failure of some sort.
The kicker was - the client grudgingly OKd the restore understanding that they will loose few hours worth of data. BUT. When a DBA attempted to restore - the backup was corrupt lol. They went back day by day - all of the recent backups were corrupt, the "freshest" working backup was about a month old. Some heads rolled as a result.
That time I had to update the users table so users belonging to 'department A' should now belong to 'department B'. I wrote that really simple update sentence and even diligently tested it on one of the pre-production servers. Then I copied & pasted it into the production console just to notice that I hadn't copied the WHERE clause of the sentence and now all of the users belonged to the same department.
The fix was not painful, just restore de user table from last night's backup. but I felt a little ashamed when i had to explain the database administrator what has happened.
My big whoops was remotely bricking a server on top of Mt Haleakala.
I needed to update MySQL to take advantage of some new feature that young, eager me just _needed_ to use. Well, the package repo didn't have that version, so no worries, I'll install from source. Well that required some newer version of a core utility, so no worries, I'll just `yum update` that. No dice. No worries, I'll just force remove and install the newer one.
Well, young, eager me didn't realize essentially every command relied on that core utility, so although I had a prompt, I couldn't even `ls`, much less `yum install`.
Our site technician had to take an OS disk up with him next time he made the 2+ hour drive to the top. Luckily we did have a KVM set up so it didn't require a site visit from me (or is that unlucky...it would have been a trip to Maui!).
In Git, a push-force used to (or at least how I remember it) only force-push your current branch. I force-pushed on a branch... and it force-pushed master. Thankfully no one had pushed to master since I last pulled from it.
Here is the fix:
```
[push]
default = current
```
Or you can just, not force-push... or be explicit, which I do now. I use force pushes for rebasing and --amend. I am one of those weirdos who enjoys clean, readable git repos.
I guess we're both weirdos, since that's exactly what I do. I made it a habit to be explicit when doing the force-push (e.g. git push --force origin <my_branch>).
Back in the nineties, at a previous employer, I wrote a code generator that read a kind of DSL from stdin and spat out C source to stdout.
Having written the initial version and got it to compile, I thought I should test it before checking it into RCS for the first time. So I ran it with some simple input and piped the output to something like program.c. At that point I remembered that the code generator's main source code file was also named program.c. To my horror it had overwritten a large part of its own source.
I remember staying in the office until about 2am to re-write the lost code, but my boss never found out.
A couple years ago, I was writing a script to run a codesigning system (watch an input directory, sign the file, move to output directory). I had been working on it for a couple days. I got confused with my filenames, and I deleted the script, thinking it was a temp file I'd just created.
Luckily, it was Python, which compiles to a bytecode representation when you run the script, and I had done a test-run of the script recently. After a little research into the available tools, I was able to extract the original source (and comments!) from the pyc file. It was a simpler solution than any of the un-delete utilities that I found.
I`ve done that on a national coding competition. Not fun at all, being timed and rewriting your program with itself at the last 30mins of the competition.
Late 90's, I'm lead dev for a large site. Company hires a DBA. We partner with a VERY large content provider. I'm awake for 48 hours buttoning things up, last minute stuff. I finally collapse from exhaustion only to be woken up at 7am on - go live day.
Nothing works.
A few moments debugging reveals that all the column names in a key set of our datasets have been renamed. No time to waste, I rename them all back as quickly as possible. We go live, no one is the wiser.
We contact the DBA and explain the folly of his decision.
He then proceeds to rename everything AGAIN on a live system. No we didn't shoot him lengthwise, but we did fire him with some prejudice.
We later found he was acutely OCD and a perfectionist, which I've since come to find is common in DBA's. He HAD to change the column names. He quite literally couldn't stop himself. Changing it in dev, QA and staging wasn't enough for him, it was an all or nothing proposition.
Fortune 500 company, Junior dev. Tried to log into our staging database, which has the same username, but different password than production.
Tried 3 times, got the locked out error. Got frustrated, said I'd fix it after lunch, went out to eat, came back about 3 hours later after a doctor's appointment.
My boss was waiting for me at the department door and I got sent back to his office for a stern talking to. I had locked out the production user from the database, causing every app in the company to go offline. I was then told to read aloud back all of the projected sales losses from that day as well as write a bunch of letters for our top sales people who couldn't make sales that day.
Wasn't fired (thankfully). Made it to a senior lead, so if I had to say anything, it's that the punishment really made it really drive home and I _never ever did it again_ . :)
I agree with nieksand... badly designed system. It's better for companies to recognize these design flaws and fix them, rather than blame employees ignorant actions.
I dont know. Ive seen systems before where anyone could make catastrophic 'modifications' like what you describe. I tend towards blaming the system construction over the junior developer.
On the first team I worked on at [corp] we had a corner of one of the bigger labs set up as a test environment, with a row of headless PCs that were used for simulating deployment scenarios. Sitting at the end of that row was a nondescript beige box that was barely distinguishable from the test boxes, except for being a bit older, that had our team's internal website, all of our documents, scratch file shares, databases, etc. on it. Of course there were no backups. At least two times while I was there, they hired temp workers and on day one gestured in the general direction of the shelf full of machines and told them something like, "OK, you can get up to speed by reimaging those boxes for this week's test pass..."
As for "Why So Many Dropped Production Databases?", you can always configure ssh with a combination of LocalCommand or use tmux to change the background of the terminal, when you are on a production server.
We were group debugging a crash that happened while using our API for a network device for NetWare under heavy load. We weren’t sure if the problem was our code or the IPX driver we had written for the particular NIC, so we were swapping in every NIC we had in the lab, and running some samples. Actually, I was doing the swapping. Turns out, the one time I forgot to power down was removing an NE-1000 (cheapest NIC available) and swapping in a 3Com 3C505, worth about $900 in 1988 dollars. The NE-1000 was fine, but the 3C505 was toasted. The PC was fine. The boss totally clowned me with some mean mugging, then smiled and said it was ok.
136 comments
[ 4.0 ms ] story [ 145 ms ] threadOne for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root.
I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine.
I recalled a piece of advice a university friend had given me. "Never admit to anything."
I walked into the server room, switched off the PC and restored it from a backup tape that was a day old. 2 Hours later the machine was back on.
No one ever noticed, and I never mentioned it either.
Moral of the story - Don't let anyone give you access to production servers.
what if he messed up the restore, and instead of restoring from the only known backup, he wiped our the only known backup?
Would be epic fired story.
I vote for admit when you mess up, and ask for help. Don't be too proud, we all mess up.
The first few times, you get that icy "oh shit" feeling, but figuring out how to fix it, and that it can (almost) always be fixed, is invaluable.
Since then I’ve adopted custom prompt and color scheme for production.
I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages
An AHA moment for me. I've always wondered why people cared about all the terminal customization options (I grew up typing on an 80x48 AAA glass TTY). This is a good reason.
I've used a custom prompt with the host name built in for decades, but this kind of color issue makes sense!
This is everyday Operations work. There is no school that teaches the eclectic mix of OS and container, networks and routes, app/db/code, CI/CD, BC/DR, writing KB articles, running standups and meetings, triage technical issues, configuration management, Change Control, and Incident Management.
These diverse skill sets are often called "Devops", but nobody makes a Devops degree. The only way to assemble this experience is while working to support developers.
From man rm:
-i prompt before every removal
-I prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes
The problem is that you get used to this crutch and rely on it, until one day you're logged in someplace without the alias and end up deleting something you shouldn't.
I'm not a fan and normally disable it. Instead, I prefer to be very careful and deliberate when using rm, especially with any wildcards, making sure I'm in the right directory and that the pattern matches what I expect it to.
Which Linux distributions do this? I have not seen this alias set by default on any distribution that I've used recently (Gentoo, Linux Mint, Debian, Ubuntu, Crunchbang, Arch).
I can find old rpms and deb files with rm aliased in /etc/skel/.bashrc or /etc/skel/profile/aliases.sh, but the most recent I found is in a bash 3.2 package, so this practice seems to have stopped quite a few years ago.
It forces me to be careful and deliberate. There's never a safe option or a backstop. I treat rm like a gun - always assume it's loaded and going to murder whatever I point it at without hesitation. If I'm not 100% sure about what I'm about to delete or how a wildcard will expand, then I echo it first.
I messed up once, about 15 years ago now. Lost a few personal files. Haven't screwed up since.
https://gist.github.com/woodrowbarlow/52b02ab27d0f8088891c3d...
look at `get_hash_color`.
Staging prompts are yellow.
Development prompts are green.
https://github.com/sindresorhus/trash-cli
I do the same. People always walk up and make some snarky comment like "wow, nice background color" and then I explain why I have a red terminal background and they say "oh... that's a great idea"
So I change my dev to be the non-default now. If I see the default/typical color scheme, I assume production.
No idea if anything would have happened, most of the data was read only (builds, logs, etc) but some was not.
In my case, it was "Don't let anyone have you do testing/debugging on the production machine."
I was 18, first dev job, small mom-n-pop which developed insurance claims management software. Early 1990s, green screen VT100 terminals connected remotely via modems and dedicated phone lines.
Anyhow, something was wrong, I forget what, and to fix it I needed to debug using a real dataset. Between each run I had to clear out and rebuild a series of test "tables" (that's in quotes because the system was PICK, and the DB isn't exactly a SQL RDBMS), then copy the production tables to the test tables, and re-run.
Well - at one point I cleared the wrong table - cleared the production and not the test table.
Within seconds all the phones lit up in the office. I groaned (rather loudly), because I knew I just dun f'd up. The client was one of 3 or 4 that the company had, and the largest. Needless to say, for every minute they were down, they couldn't process claims, and were losing money.
Did I mention there weren't any backups? Not that backups weren't made, just that they were "end-of-day" backups, so the backup for that day's work (head's down manual entry of claims for processing) had not yet happened.
Fortunately for me, the table I deleted could be recreated (mostly) from two other still-intact tables. My manager and I worked thru the evening rebuilding the table as much as we could, and also pulled in data from the prior day's backup to make it as complete as possible. Even so, they still had to re-enter some data lost that wasn't recoverable by that method.
Did we set up a test server or do anything else that any sane company would do after that incident? Nope. Things just continued on in the same manner, and somehow I kept my job. Honestly, we should have been doing the testing in-house, but this was using an IBM RS6K box that at the client was much larger than what we had as a dev box (we were using the smallest version of that machine at the time, which was meant to be a single user desktop workstation as I later found out), so we couldn't fit the data onto the hard drive, even if it was possible to download it all via modem (it wasn't - I'd probably still be waiting for the dump to complete given the modem was only 9.6 kbps). I have no idea what it would have cost to do it right, but the drives needed back then, coupled with the infamous IBM service contract - meant we couldn't just hop over to the nearest Insight to grab a cheap hard drive and slap it in.
Today, I'd probably have the admin spin up another instance of the PICK system as another process on AIX, and (somehow) replicate the live data/code over and run in that instance. Still not "safe", but probably would have been much safer than what we were doing. But alas, I was but a kid at the time, and barely understood *nix (but that was where I first learned about it - and I also gained a healthy lesson/understanding/fear/respect for terminfo - for the purposes of supporting a variety of serial terminals and PC terminal emulators).
If you are root, make another user. you can delete it later.
I always have two terminals open, both running tmux. One is white text on black for dev work, the other is white text on red. I only ever ssh into production servers on the white on red terminal so there's a strong visual signal that I absolutely shouldn't be running drop, delete, truncate, rm, dd, whatever commands on them and if I do then I am extremely careful.
This is a great idea.
Good way to save your bacon.
Cool. Sorry, but I'm going to steal this idea from now on.
For UAT / Test environments that are different than dev, I use a yellow background.
And let's not talk about my own (mostly minor) disasters.
We did have nightly backups but the guy with access to them was notoriously hard to get the attention of and lived in a different state. I was on my way out the door, and now I ended up being stuck there for another few hours trying to get ahold of devops.
At the company I am at now, we had a devops person whipe our multi TB CDN on their first day trying to make an improvement to our asset sync script. It took probably 2 days to get the whole thing reuploaded.
For example, I often write my 'rm' commands as 'ls' at first and then swap out the command once I know it gets the right thing.
Another tip is to write the WHERE clause and SET clause prior to filling in which table is being updated. That way you can’t accidentally run something prematurely.
If I'm typing an ad-hoc DELETE or UPDATE statement, I'll slap the WHERE keyword on the end of the main command line. For example:
Yeah, it looks ugly, and I wouldn't use that formatting for "real code," with tests and whatnot. But for messing around in the query editor, I just find that I'm much more likely to accidentally leave off whole lines. I never accidentally select parts of lines. More generally, the idea is to make your partial query invalid syntax, if you leave off the last line.I also tend to wrap destructive DML in comment blocks. So in my query editor, I'd actually have:
The idea here is just that I want to be required to select this text and run it. If I have some other queries in the window and somehow accidentally run all statements in the window (didn't select something I meant to or maybe a query is hiding beneath the scroll line), I don't want destructive queries to run.EDIT: I've actually never made a catastrophic mistake of this nature, but there have been some very close calls. And I've seen it happen several times, from excellent programmers who just made a mistake.
The fact that I was entering a delete -- instead of setting to inactive and doing it in production from the command line should give you an idea of how likely the backups would have been effective. This was in '97, though when everything was seat of the pants during the full internet craze.
If you want to delete all rows, you should be made to explicitly type out that intention.
As far as I can tell, there's nothing like that in Postgres.
It slowed things down, but the forced code reviewer process saved me once in a big way.
Read up on "bus factor": https://en.wikipedia.org/wiki/Bus_factor
I too have "restored" a database to the wrong server (production was now development). This was a member list for an arts organization... After about 20 minutes of panic (what is going on....) I remembered I had set daily backups. whew.
I was sitting at my desk watching service after service slowly disappear in utter confusion, then hours spent cleaning...
I was logged in over VPN to a client machine. I suspected the problem I was fixing was something to do with the network misbehaving. It was a Windows machine.
I thought "I know: I'll disable and reenable the network device!".
A few moments later: "Oh. Bugger.".
Shortly followed by an embarrassing call to the client's IT department to ask them to reenable the network device on that machine...
Probably one of the stupidest, but also the most personally effective way to learn anything.
I remember at least one prod incident where a script made it out into prod and wiped out 100s of servers. Something like it had rm -rf /$SOME_ENV_VAR which was set in QA but not prod. Opps....
I look at `terraform plan -destroy` and it still terrifies to destroy an internal service.
Doing a recursive sudo chown of the /etc folder, locked myself out of that ubuntu server.
Wrote a parser that had a condition that would not end, would watch the server spike to 100% usage.
it turned out that our beta testers found a lot of cases that were broken on Friday when the release was next Monday. The problem was that a ton of other classes pointed to the one in which I made changes and altered the state..
Worked through the weekend with my teamlead and got it fixed somewhere on sunday afternoon.
good bonding experience though. All had a laugh about it the next week. :-)
The kicker was - the client grudgingly OKd the restore understanding that they will loose few hours worth of data. BUT. When a DBA attempted to restore - the backup was corrupt lol. They went back day by day - all of the recent backups were corrupt, the "freshest" working backup was about a month old. Some heads rolled as a result.
The fix was not painful, just restore de user table from last night's backup. but I felt a little ashamed when i had to explain the database administrator what has happened.
From that day, I keep autocommit always off.
I needed to update MySQL to take advantage of some new feature that young, eager me just _needed_ to use. Well, the package repo didn't have that version, so no worries, I'll install from source. Well that required some newer version of a core utility, so no worries, I'll just `yum update` that. No dice. No worries, I'll just force remove and install the newer one.
Well, young, eager me didn't realize essentially every command relied on that core utility, so although I had a prompt, I couldn't even `ls`, much less `yum install`.
Our site technician had to take an OS disk up with him next time he made the 2+ hour drive to the top. Luckily we did have a KVM set up so it didn't require a site visit from me (or is that unlucky...it would have been a trip to Maui!).
In Git, a push-force used to (or at least how I remember it) only force-push your current branch. I force-pushed on a branch... and it force-pushed master. Thankfully no one had pushed to master since I last pulled from it.
Here is the fix:
``` [push] default = current ```
Or you can just, not force-push... or be explicit, which I do now. I use force pushes for rebasing and --amend. I am one of those weirdos who enjoys clean, readable git repos.
Having written the initial version and got it to compile, I thought I should test it before checking it into RCS for the first time. So I ran it with some simple input and piped the output to something like program.c. At that point I remembered that the code generator's main source code file was also named program.c. To my horror it had overwritten a large part of its own source.
I remember staying in the office until about 2am to re-write the lost code, but my boss never found out.
Luckily, it was Python, which compiles to a bytecode representation when you run the script, and I had done a test-run of the script recently. After a little research into the available tools, I was able to extract the original source (and comments!) from the pyc file. It was a simpler solution than any of the un-delete utilities that I found.
Nothing works.
A few moments debugging reveals that all the column names in a key set of our datasets have been renamed. No time to waste, I rename them all back as quickly as possible. We go live, no one is the wiser.
We contact the DBA and explain the folly of his decision.
He then proceeds to rename everything AGAIN on a live system. No we didn't shoot him lengthwise, but we did fire him with some prejudice.
>He then proceeds to rename everything AGAIN on a live system.
What could his reasoning possibly be? Did he not understand your explanation?
I was relying on expressions like
to fail to type-check to spot all the places that I needed to update.Sadly some parts of the codebase weren't very idiomatic and had some
This caused a bit of involuntary testing in production and thought me not to trust anything while refactoring.I'm going to steal that one.
Tried 3 times, got the locked out error. Got frustrated, said I'd fix it after lunch, went out to eat, came back about 3 hours later after a doctor's appointment.
My boss was waiting for me at the department door and I got sent back to his office for a stern talking to. I had locked out the production user from the database, causing every app in the company to go offline. I was then told to read aloud back all of the projected sales losses from that day as well as write a bunch of letters for our top sales people who couldn't make sales that day.
Wasn't fired (thankfully). Made it to a senior lead, so if I had to say anything, it's that the punishment really made it really drive home and I _never ever did it again_ . :)
I did a blog post about it [1].
1: http://www.drinchev.com/blog/ssh-and-terminal-background/
My boss wasn't happy.
Sometimes works.