A likely culprit. Having worked on a bunch of early-stage products where best practices are a distant future dream, I’ve developed a few “seatbelt” habits I use to avoid these kinds of things. One of them is to always use a random high-number local port if I’m tunneling to a production service.
Another is to change my terminal theme to a red background before connecting to anything in production...never want to click the ‘psql ...’ tab, run “truncate table app_user cascade” and realize afterwards it was a lingering connection to production...
I think the most common reason I’ve had in the past was to connect to the RabbitMQ web admin while dealing with some emergent issue with task throughput, a common problem area with hacked together web apps that start to get real traffic. It’s also handy to be able to use a more advanced SQL client that’s not installed on the server (pgcli, emacs, etc) when digging around for something that’s causing errors in production.
Take the "drop database" bit (on the production database) away from your developers, too.
As well as pretty much every other privilege they don't legitimately need to use on a daily basis -- which, for prod, should be most of them (quite possibly including "delete").
If or when they really need to delete a ton of rows all in one go, they can be given a (temporary) set of credentials that they can use to do that, once, and which are then revoked immediately afterwards -- after another set of eyes reviews the script they've written to actually perform the operation, of course.
Basic best practices aren't difficult or some secret thing that only the experts know about. It is necessary to actually implement and follow them, however!
Sure, it can be a pain in the ass sometimes. Will it be worth it when it eventually saves your ass one day, "after a couple of glasses of red wine"? Absolutely.
As a developer at all the jobs I've had, I've never even had access to the production database, full stop - only the server admins did. If I needed something from prod I'd go through them. I don't even consider it inconvenient.
The distance between the developer and the production database is directly proportional with the size of the company. The same is true for the amount of paperwork needed for even the simplest things.
For some company sizes a feedback cycle measured in days is not good enough and may kill the business.
There is a whole lot of things that could be applied which would have prevented this, before needing to hire a separate server admin as an interface to the production database. :D
could also be that they just have an if staging/else somewhere for the credentials (but of course this isn't a counterpoint to the root of your point - the script wiping the database shouldn't be using anything that does something like this, and you probably shouldn't do it at all)
Yeah this stuck out to me as well. To be honest if they ARE using the same creds in prod as they are in dev, this is the first thing they should fix because it would have easily prevented the whole mess.
I'll give them the benefit of the doubt and suggest that it wasn't the case and that it's unlikely that a dev or OPs person was tunnelling through a bastion of some kind to run the script.
If they pulled their credentials from a store and populated the config object that way, it's very possible someone actually loaded the production credentials by mistake into the development secrets file. The CI/pipeline system has permissions/network access to deploy to any environment, hence how it ran the script to drop the table.
I'm purely speculating another alternative to the much worse case of the tunnelling scenario outline.
Given the short time frames that startups aim to capture value in its not worth investing time in safety until you have plenty of capacity to spare later.
Like this never happens in big corps: sure, mostly on departmental level but still enterprises, not startups. They try to escape the annoying and slow as molasses dba’s/devops and install/create/buy/saas systems to avoid the red tape. But then the same things as with startups go wrong. And rather often too; we often had these calls asking if we could restore.
Obviously there are plenty of large enterprise wide data breaches, which, I would say is actually worse than losing a day of data in a lot of cases. So also not so many satefy measures, again, worse than startups; at least they have an excuse of being understaffed and underfunded.
This is what I came to say. Postgres usually allows a connection string that contains all the details as the database name postgres://username:password@host:port/dbname and I think it will take priority over separately specified host, depending on the client.
>Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. We’re too tired to figure it out right now. The gremlins won this time.
Obviously, somehow the script ran on the database host.
some practices I've followed in the past to keep this kind of thing from happening:
* A script that deletes all the data can never be deployed to production.
* scripts that alter the DB rename tables/columns rather than dropping them (you write a matching rollback script ), for at least one schema upgrade cycle. you can always restore from backups, but this can make rollbacks quick when you spot a problem at deployment time.
* the number of people with access to the database in prod is severely restricted. I suppose this is obvious, so I'm curious how the particular chain of events in TFA happened.
I have a little metadata table in production that has a field that says “this is a production database”. The delete-everything script reads that flag via a SQL query that will error out of it’s set in the same transaction as the deletion. To prevent the flag from getting cleared in production, the production software stack will refuse to run if the “production” flag is not set.
This is also one place where defense-in-depth is useful. "Has production flag" OR "name contains 'prod'" OR "hostname contains 'prod'" OR "one of my interfaces is in the production IP range" OR etc. etc. You really can't have too many clauses there.
Unfortunately, the "wipe & recreate database" script, while dangerous, is very useful; it's a core part of most of my automated testing because automated testing wipes & recreates a lot.
one silly last resort measure I did on a project a while back was having a IS_STAGING file somewhere, only existing on localhost, and every request it would check if the hostname is that of the live site, and if so, delete that file. the file itself wasn't enough to make the server think it's in staging mode, but it was the only thing in the chain that if it were to go wrong, would fix itself automatically almost immediately (and log an error)
> the number of people with access to the database in prod is severely restricted
And of those people, there should be an even fewer number with the "drop database" privilege on prod.
Also, from a first glance, it looks like using different database names and (especially!) credentials between the dev and prod environments would be a good idea too.
One aspect that can help with this is separate roles/accounts for dangerous privileges.
I.e. if Alice is your senior DBA who would have full access to everything including deleting the main production database, then it does not mean that the user 'alice' should have the permission to execute 'drop database production' - if that needs to be done, she can temporarily escalate the permissions to do that (e.g. a separate account, or separate role added to the account and removed afterwards, etc).
Arguably, if your DB structure changes generally are deployed with some automated tools, then the everyday permissions of senior DBA/developer accounts in the production environment(s) should be read-only for diagnostics. If you need a structural change, make a migration and deploy it properly; if you need an urgent ad-hoc fix to data for some reason (which you hopefully shouldn't need to do very often), then do that temporary privilege elevation thing; perhaps it's just "symbolic" but it can't be done accidentally.
Someone SSHed to production and forwarded the database port to the local machine to run a report, then forgot about the connection and ran the deletion script locally.
Yeah, this. The problem is not that the production database was deleted by accident. The problem is that it was possible to (unrecoverably) delete the production database by accident.
You'll find that most users of cloud databases are in this boat. For example, on GCP, deleting the database instance deletes the backups! You have to write your own software if you want to survive that button click.
localhost is an abstraction, it's a non-routable-outside-your-machine network...except it's not. It's nothing more than normal TCP traffic except with a message to the OS and other programs that whatever is on that local computer network, you don't want it routed outside the local computer.
There's absolutely nothing stopping anything with access to localhost from routing it anywhere that process wants. Does not even take a malicious actor, all kinds of legit programs expose localhost. It's really not something you should use for anything except as a signal to other well-behaving programs that you are using the network stack as a machine-local IPC bus.
I like to use the lifecycle feature for suuuper core things that will never be deleted (VPC, r53 zone, etc) and eventually when I start targeting multiple DCs w/ lots of infra I'll eventually move to many state roots (or use tools like Terragrunt, which make things mildly scary again).
> Computers are just too complex and there are days when the complexity gremlins win.
Wow. But then again it's not like programmers handle dangerous infrastructure like trucks, military rockets or nuclear power plants. Those are reserved for adults
I feel that computers make it easier for this danger to be more indirect, however. The examples you give are physical, and even the youngest of child would likely recognise they are not regular items. A production database, meanwhile, is visually identical to a test database, if measures are not made to make it distinct. Adults though we may be, we're human, and humans can make really daft mistakes without the right context to avoid them
There are also countless safety measures on physical items that have been iterated on over decades to prevent all kinds of accidents. Things like putting physical locks on switches to prevent machinery being turned on while people are working on it.
Can you imagine if instead of a physical lock it just said “are you sure you wish to turn on this machine”. “Of course I want to turn it on, that’s why I pressed the button”
Some software makes it a lot harder for the user to mess up now. When deleting a repo on GitLab you have to type the name of the repo before pressing delete and then it puts it in a pending deletion state for a month before it’s actually deleted. Unfortunately for developers we typically get minimal cli tools which will instantly cause a lot of damage without any way to undo.
So, silly idea. What if, to work on the production database, you had to go into the special production room, colored in the special production color, scented with the special production perfume, and sit on a just tiny bit uncomfortable production chair.
Basically make it clear even to the caveman brain that things are different.
I'm not sure I follow your point - I think you'll find the same attitude towards complexity by operators of military rockets and nuclear power plants. If you look at postmortems/root-cause analyses/accident reports from those fields, you'll generally find that a dozen things are going wrong at any given time, they're just not going wrong enough to cause problems.
One explanation for the author feeling that way is that the system is has too much automation. Being in a situation where you take on more responsibilities of the system at a shallower level leads to less industry expertise. This, as it turns out, places the security of the system in a precarious position.
These is pretty common, as devs tool belts have grown longer over time.
I think at some point we will stop automating or reverse some of the automation.
I do a lot of work with middle/high school students. Without fail someone would yell "Why is it doing x!"...to which my standard reply is "Because you told it to".
Definitely, the likelihood of these things happens goes way up alongside the amount of concurrent tasks, meetings, or other forms of distraction. I’ve seen my fair share of production administration as a side task.
Indeed - after incidents like this I usually say, "This is called experience that you can't pay to get for any price. Learn from it well, and value your lesson."
Yeah, imagine if a bridge engineer said the same thing: "You aren't an engineer until your bridge collapses. Congrats!" I am starting to hate tech culture. Nobody cares about correctness and discipline. Mention "math" and everybody spreads like cockroaches.
Yeah, but that can still create a situation where you can create WALs faster than they're uploaded. (And if there's a good solution to that, I'm interested, because we ran a server out of disk this way last month at work)
This happened to me (someone in my team) a while ago but with mongo. The production database was ssh-tunneled to the default port of the guys computer and he ran tests that cleaned the database first.
Now... our scenario was such that we could NOT lose those 7 hours because each customer record lost meant $5000 usd penalty.
What saved us is that I knew about the oplog (binlog in mysql) so after restoring the backup i isolated the last N hours lost from the log and replayed it on the database.
Same happened to me many years ago. QA dropped the prod db. It's been many years but if I recall, I believe in the dropdown menu of the MongoDB browser, exit & drop database were next to each other...Spent a whole night replaying the oplog.
No one owned up to it, but had a pretty good idea who it was.
> No one owned up to it, but had a pretty good idea who it was.
That sounds like you're putting (some of) the blame on whoever misclicked. As opposed to everyone who has allowed this insanely dangerous situation to exist.
This. The person that erased the database in my case came forward to me as soon as we realized what had happened. At that moment I was very happy it was an "inside job", it meant I could discard hacking.
As its said before: he made a mistake. The error was allowing the prod database to to be port forwarded from a non prod environment. As head of eng that was MY error. So I owned to it and we changed policies.
Things like this would have been much less likely to happen in the past -- you know, when the developers only had access to the development database.
But then someone had an idea... think of how great it would be if we got rid of the operations folks and gave responsiblity for prod to the developers, too!
> And it bodes well for your firm that that doesn't get you fired either.
Maybe there's more than one interpretation of "bodes well" but not knowing how to do the one thing customers were paying you to do isn't consistent with my definition.
> having dealt with it responsibly and honestly as a team is something you can be proud of IMO.
"We were drinking wine and deleted the database and now your data's gone LOL" is not something that should make you proud.
Happens to all of us. Once I required logs from the server. The log file was a few gigs and still in use. so I carefully duplicated it, grepped just the lines I needed into another file and downloaded the smaller file.
During this operation, the server ran out of memory—presumably because of all the files I'd created—and before I know it I'd managed to crash 3 services and corrupted the database—which was also on this host—on my first day. All while everyone else in the company was asleep :)
Over the next few hours, I brought the site back online by piecing commands together from the `.bash_history` file.
It does but that was an exceptional role. The company needed emergency patches to a running product while they hired a whole engineering team. As such, I was the only one around doing things, and there wasn't any documentation for me to work off of.
I actually waited until nightfall just incase I bumped the server offline because we had low traffic during those hours.
I wouldn't classify it as that but they had had trouble in the past which lead to a lot of their team leaving, and were now looking to recover from it.
I was only there for a short time though. Hopefully they figured things out.
No. That is exactly what a transactional DB is designed to prevent. The journal gets appended with both the old and the new data and physically written to disk, and only then the primary data representation (data and B-tree blocks) gets updated in memory, then eventually that changed data is written to DB files on disk. If the app or DB crashes during any stage, it will reconstruct primary data based on journalled, comitted changes. DBs shouldn't attempt to allocate memory during the critical phase, and should be able to recover even on failed allocations at any time by just crashing and let regular start-up recovery clean up. Though a problem on Linux might be memory overcomitting.
Edit: and another problem is disk drives/controller caches lying and reporting write completion when not all data has actually reached stable storage
I like to think that, by addressing the known bugs as they pop up, over time you can box the complexity gremlins into tighter and more predictable spaces. Though as long as humans are building these systems, that box will always be there, and the predictability of those gremlins' behavior will only go so far.
In context, statement 1 regards proactively eliminating all bugs and risks. Statement 2 regards understanding the root cause of this particular incident and reactively fixing it so it won’t happen again.
Acknowledging statement 1 doesn’t mean giving up—it simply means being clear and realistic about the nature and scale of the problem we’re facing when we try to build complex software systems. In the face of that we can give up, or we can just do the best we can, and it sounds more like these people are doing the latter.
We have something similar with AWS Cognito. If a user signs up but doesn't go through with the verification process, there's no setting to say "remove them after X days". So we have to run a batch job.
If I screw up one parameter, instead of deleting only unconfirmed users, I could delete all users. I have two redundant checks, first when the query is run to get the unconfirmed users, and then again checking the user's confirmed status before deleting them. And then I check one more time further down in the code for good measure. Not because I think the result will be different, but just in case one of the lines of code is altered somehow.
I put BIG LOUD comments everywhere of course. But it still terrifies me.
Barman (1) is really easy to setup and lets you avoid the many pitfalls of RDS (lower performance, high cost, no superadmin, no arbitrary extensions, waiting for releases, bad log interface).
If you keep configuration in the environment (/etc/default/app-name) rather than in the application package, it's nearly impossible to make this mistake (especially with proper firewall rules). You can even package your config as a deb and keep it encrypted version control.
> after a couple of glasses of red wine, we deleted the production database by accident
> It’s tempting to blame the disaster on the couple of glasses of red wine. However, the function that wiped the database was written whilst sober.
It was _written_ then, but you're still admitting to the world that your employees do work on production systems after they've been drinking. Since they were working so late, one might think this was emergency work, but it says "doing some late evening coding". I think this really highlights the need to separate work time from leisure time.
No. Your systems and processes should protect you from doing something stupid, because we’ve all done stupid things. Most stupid things are done whilst sober.
In this case there were like 10 relatively easy things that could have prevented this. Your ability to mentally compile and evaluate your code before you hit enter is not a reliable way to protect your production systems.
Coding after drinking is probably not a good idea of course, but “think better” is not the right takeaway from this.
A highly experienced developer in their 50s who I used to work with said that they used to regularly sit down and code with a pint. Until on one occasion they introduced some Undefined Behaviour into their aplication during one of these sessions and it took them 3 days to track down! Probably less of an issue with modern tooling. Still, it certainly makes me think twice before drinking and coding.
You know it's totally feasible to make a car that won't turn on for drunk people. Should those systems be installed on all cars, in pursuit of creating systems that don't permit stupid actions?
Maybe such a breathalyzer interlock could be installed on your workstation too. After all, your systems and processes should prevent you from stupid things.
Don’t be absurd. Of course there are costs and tradeoffs to guardrails, and you have to balance the tradeoffs based on your requirements.
This person had to publicly apologize to their customers. One or two low-cost guardrails could have prevented it and would probably have been worth the cost.
Is that absurd? Systems should prevent mistakes, unless the part needed to implement that is a hundred dollars or so? That seems like quite the walk-back. Courts order alcoholics to install these things, they're established available technology. What tradeoffs are you balancing here?
Or a blog on being unable to drive your kid to an emergency room because you just finished a glass of wine over dinner.
A problem with devices of that type is that they only test for a potentialsource of inability to drive safely. What we want is to test for an inability to drive safely.
And while one is easy and might give some quick wins, the drawbacks scare me too much.
Being unable to take your car when your child needs to go to the ER would be terrible.
Actually getting in the car while under the influence such of stress and alcohol sounds worse.
I know someone who had a glass of wine just before her daughter needed to be brought to the hospital. This was just two days ago. She simply concluded she could not drive. Luckily, she was able to get a taxi.
Maybe we should, as a society, invest in taxis equipped with medical facilities and trained personnel so that they can provide first response medical treatment while on the way to the ER.
I'm sure that would save a lot of lives. An ambulatory medical service, if you will.
Sometimes ambulances are occupied and taking a taxi goes faster. Especially if it's something which isn't immediately life threatening.
I once dislocated my shoulder while on a large trampoline and was unable to get up from my hands and knees due to the intense pain whenever the trampoline wobbled. The ambulance was redirected to more serious injuries three times. I was stuck in that position waiting for two hours before it arrived.
In that scenario it would also be appropriate to wait for a driver to sober up before driving you to the hospital if neither ambulance nor taxi were available (or delayed). One glass of wine would be out of most people's systems after two hours.
Thus poking hole in the "drunk drive someone to the hospital" argument, which is what this was all about in the first place.
I did argue, in my original comment, that drunk driving should not be an option. I certainly stand by that. My original comment also mentioned a taxi, to which you replied about ambulances.
In my previous comment I just meant that sometimes ambulances can take a good while and a taxi might not.
In the unfortunate case of the trampoline there were several sober people with driver's license and cars available and a taxi would have been there immediately.
Unfortunately,they failed to get me out of there, meaning I still had to wait until an ambulance was available. It was beyond painful and exhausting both physically and mentally. But it was still technically not an emergency.
Yes. I was answering a comment suggesting the use of an ambulance (instead of a regular taxi). Simply pointing out that, in practice, there are times when a taxi can get you there faster.
Note that ambulance ride (depending on insurance) may cost an order or 2 of magnitude more than the taxi. Well worth it in some circumstances - but not always the best option.
This didn't spring to my mind as I'm Swedish and here it's less than a taxi and any medical costs beyond the first USD $130 per year is covered by the free health insurance.
You are still simply going with one glass of wine will affect everyone equally, and that is proven to be untrue. (Or stress, for that matter)
While your friend made a call judging their own abilities and the level of emergency, that's exactly how it should be: cars should not stop us humans for making that decision.
(Fwiw, if you were just having an alcoholic drink, a breathalizer would show a much higher concentration even though alcohol might not have even kicked in or there wasn't enough for it to kivk in at all)
There was a funny story recently in the UK, where a football team was late for a match because their breathalyser-equipped team bus refused to start. Turns out it wasn’t that the driver had been drinking, rather that the alcohol-based disinfectant they’d used to clean the bus triggered it.
It will be a great idea when the reliability of the system will have a large number of nines, so chances that you are stranded in the middle of nowhere and the car does not want to start because of a fault will be less than being hit by an asteroid. Other than that, people would consider it an unsafe product and refuse to use it and people vote for what finally becomes a law.
I heard the same argument for electronic gun safety measures, except that no government agency even consider using it for their own guns. Why? They are not reliable enough, yet.
Replace a breathalyzer with something that's less intruisive (like a camera with AI that would observe the person, AI with thermal imaging or air quality sensors, or another possibly-fictional-yet-believable piece of technology) and suddenly, in my eyes, the technology in this thought experiment becomes a no brainer.
If there was more of a cultural pressure against drunk driving and actual mechanisms to prevent it that aren't too difficult to maintain and utilize, things like the Daikou services ( https://www.quora.com/How-effective-is-the-Japanese-daikou-s... ) would pop up and take care of the other logistical problems of getting your car home. And the world would be all the better for it, because of less drunk driving and accidents.
That is because most companies these days have processes around drinking in workplace, coming in drunk and working drunk.
Most mistakes are done sober only in environments where drinking couple of vine cups and then doing production change is considered unacceptable. In environment where drunk people work, mistakes are made when drunk.
I am not a drinker myself (drink 1-3 times a year), but in the past I have coded while slightly buzzed on a few occasions. I could not believe the level of focus I had. I never investigated it further, but I'm pretty sure the effects of alcohol on our coding abilities is not nearly as bad as it affects our motor skills. Imo, fatigue is far worse.
Me too, partly because of the reduced short-term memory (which coding relies on heavily). But more than that, my motivation drops through the floor because I can’t shake the thought that life is too short to fight with computers all day.
When I was not yet a teetotaler, each time I was hitting my maths textbooks after a few drinks, I could not believe my level of focus, and everything was clear and obvious. Textbook pages were flying at a speed never seen.
Of course the next day, when re-reading the same pages, I was always discovering that the previous day I had everything wrong, nothing was obvious, and all my reasoning when with alcohol was false because simplistic and oblivious of any mathematical rigor.
Similar effect with psilocybin or LSD - you think you had a really profound and insightful experience, but once you think back on it you realize that (most of the time) you just got the impression that it was profound and insightful.
In short, ability to recall memories is at least in part dependent on being in a similar state to the time when memories are formed, e.g. something learned while being intoxicated will be more easily recalled only when intoxicated again.
For me it's really hit and miss - it can either increase focus and motivation, invigorate the mind, while reducing distractions. Other times it has the opposite effect. Same with cannabis. The cannabis absolutely took a decade or so of recreational use until I discovered/developed coding well under the influence.
Though I'm talking one or two drinks here, not firing up vscode after a night out or going through a bottle of rum.
The whole piece has a slightly annoying flippant tone to it. We were drunk! Computers just.. do this stuff sometimes! Better to sound contrite and boring in such a situation IMO.
Also I agree with other comments: doing some work after a glass or two should be fine because you should have other defences in place. “Not being drunk” shouldn’t be the only protection you have against disaster.
But it's just a side-project and I will continue late night coding with a glass of wine.
Normally, this would be fine. But, it appears the site has paying members. Presumably, it's not "just a side-project" to them. You owe them better than tinkering with prod while tipsy.
I don't think that's fair. We've all had occasions from time to time when we've had a drink at lunch time or even had to do emergency work in the evening after having a drink.
The bigger issue is the lack of guardrails to protect against accidental damage. This is also a common trait for hobby projects (after all, it's more fun to hack stuff together) but hopefully the maintainer will use this experience as a sobering reminder (pun intended) to put some guardrails in place now.
455 comments
[ 9.4 ms ] story [ 354 ms ] threadAnother is to change my terminal theme to a red background before connecting to anything in production...never want to click the ‘psql ...’ tab, run “truncate table app_user cascade” and realize afterwards it was a lingering connection to production...
As well as pretty much every other privilege they don't legitimately need to use on a daily basis -- which, for prod, should be most of them (quite possibly including "delete").
If or when they really need to delete a ton of rows all in one go, they can be given a (temporary) set of credentials that they can use to do that, once, and which are then revoked immediately afterwards -- after another set of eyes reviews the script they've written to actually perform the operation, of course.
Basic best practices aren't difficult or some secret thing that only the experts know about. It is necessary to actually implement and follow them, however!
Sure, it can be a pain in the ass sometimes. Will it be worth it when it eventually saves your ass one day, "after a couple of glasses of red wine"? Absolutely.
There is a whole lot of things that could be applied which would have prevented this, before needing to hire a separate server admin as an interface to the production database. :D
I'll give them the benefit of the doubt and suggest that it wasn't the case and that it's unlikely that a dev or OPs person was tunnelling through a bastion of some kind to run the script.
If they pulled their credentials from a store and populated the config object that way, it's very possible someone actually loaded the production credentials by mistake into the development secrets file. The CI/pipeline system has permissions/network access to deploy to any environment, hence how it ran the script to drop the table.
I'm purely speculating another alternative to the much worse case of the tunnelling scenario outline.
Obviously there are plenty of large enterprise wide data breaches, which, I would say is actually worse than losing a day of data in a lot of cases. So also not so many satefy measures, again, worse than startups; at least they have an excuse of being understaffed and underfunded.
Tough way to learn that lesson though.
Obviously, somehow the script ran on the database host.
some practices I've followed in the past to keep this kind of thing from happening:
* A script that deletes all the data can never be deployed to production.
* scripts that alter the DB rename tables/columns rather than dropping them (you write a matching rollback script ), for at least one schema upgrade cycle. you can always restore from backups, but this can make rollbacks quick when you spot a problem at deployment time.
* the number of people with access to the database in prod is severely restricted. I suppose this is obvious, so I'm curious how the particular chain of events in TFA happened.
Unfortunately, the "wipe & recreate database" script, while dangerous, is very useful; it's a core part of most of my automated testing because automated testing wipes & recreates a lot.
More likely, I'd suspect, is something like an SSH tunnel with port forwarding was running, perhaps as part of another script.
And of those people, there should be an even fewer number with the "drop database" privilege on prod.
Also, from a first glance, it looks like using different database names and (especially!) credentials between the dev and prod environments would be a good idea too.
I.e. if Alice is your senior DBA who would have full access to everything including deleting the main production database, then it does not mean that the user 'alice' should have the permission to execute 'drop database production' - if that needs to be done, she can temporarily escalate the permissions to do that (e.g. a separate account, or separate role added to the account and removed afterwards, etc).
Arguably, if your DB structure changes generally are deployed with some automated tools, then the everyday permissions of senior DBA/developer accounts in the production environment(s) should be read-only for diagnostics. If you need a structural change, make a migration and deploy it properly; if you need an urgent ad-hoc fix to data for some reason (which you hopefully shouldn't need to do very often), then do that temporary privilege elevation thing; perhaps it's just "symbolic" but it can't be done accidentally.
Whoa.
There's absolutely nothing stopping anything with access to localhost from routing it anywhere that process wants. Does not even take a malicious actor, all kinds of legit programs expose localhost. It's really not something you should use for anything except as a signal to other well-behaving programs that you are using the network stack as a machine-local IPC bus.
> terraform destroy
(And either a confirmation or a flag) and everything is deleted.
I know you can add some locks but still :/
I like to use the lifecycle feature for suuuper core things that will never be deleted (VPC, r53 zone, etc) and eventually when I start targeting multiple DCs w/ lots of infra I'll eventually move to many state roots (or use tools like Terragrunt, which make things mildly scary again).
I think the main reason of this accident is lack of separation between development and operations.
Wow. But then again it's not like programmers handle dangerous infrastructure like trucks, military rockets or nuclear power plants. Those are reserved for adults
Can you imagine if instead of a physical lock it just said “are you sure you wish to turn on this machine”. “Of course I want to turn it on, that’s why I pressed the button”
Some software makes it a lot harder for the user to mess up now. When deleting a repo on GitLab you have to type the name of the repo before pressing delete and then it puts it in a pending deletion state for a month before it’s actually deleted. Unfortunately for developers we typically get minimal cli tools which will instantly cause a lot of damage without any way to undo.
Basically make it clear even to the caveman brain that things are different.
These is pretty common, as devs tool belts have grown longer over time.
I think at some point we will stop automating or reverse some of the automation.
Literally just the automation of the test suite. That's 1 automation.
> These is pretty common
? Waiting for FB to delete their db
Seems like they though a casual "everyman" type of explanation would suffice, but really who would trust them after this?
The worst thing about computers? They do exactly what they are told to do.
Well, tell them to UNDROP and see what happens.
Then the client wanted that functionality back. Oops.
That said, I will happily accept consulting fees in return for deleting someone's database in prod, should they so desire.
Edit: Heck, being a white-hat licensed-to-create-mayhem chaos monkey for a few hours a week sounds pretty fun. Email in profile.
They could have used point-in-time recovery to not lose any data from this at all.
You can the restore from a recent base backup and roll forward the WAL to just before the snafu.
Now... our scenario was such that we could NOT lose those 7 hours because each customer record lost meant $5000 usd penalty.
What saved us is that I knew about the oplog (binlog in mysql) so after restoring the backup i isolated the last N hours lost from the log and replayed it on the database.
Lesson learned and a lucky save.
No one owned up to it, but had a pretty good idea who it was.
That sounds like you're putting (some of) the blame on whoever misclicked. As opposed to everyone who has allowed this insanely dangerous situation to exist.
Not immediately calling up your boss to say "I fucked up big" is not a mistake, it is a conscious bad action.
> in the dropdown menu of the MongoDB browser, exit & drop database were next to each other
So maybe they signed off for the night without realizing anything was wrong.
As its said before: he made a mistake. The error was allowing the prod database to to be port forwarded from a non prod environment. As head of eng that was MY error. So I owned to it and we changed policies.
Nice that you were a person he felt ok with sharing the mistake with, I suppose that's an important part of being head of eng.
There are ways around it, of course, but it prevents the scenario described above.
I haven't seen this design in practice using MongoDB Atlas or Compass, but would hope for an "Are you really sure?" confirmation in an admin UI.
Things like this would have been much less likely to happen in the past -- you know, when the developers only had access to the development database.
But then someone had an idea... think of how great it would be if we got rid of the operations folks and gave responsiblity for prod to the developers, too!
And it bodes well for your firm that that doesn't get you fired either.
These things happen to the best of us but having dealt with it responsibly and honestly as a team is something you can be proud of IMO.
Maybe there's more than one interpretation of "bodes well" but not knowing how to do the one thing customers were paying you to do isn't consistent with my definition.
> having dealt with it responsibly and honestly as a team is something you can be proud of IMO.
"We were drinking wine and deleted the database and now your data's gone LOL" is not something that should make you proud.
During this operation, the server ran out of memory—presumably because of all the files I'd created—and before I know it I'd managed to crash 3 services and corrupted the database—which was also on this host—on my first day. All while everyone else in the company was asleep :)
Over the next few hours, I brought the site back online by piecing commands together from the `.bash_history` file.
I actually waited until nightfall just incase I bumped the server offline because we had low traffic during those hours.
I was only there for a short time though. Hopefully they figured things out.
It can even fail while in the middle of a transaction commit.
So transactions won't fix this.
Edit: and another problem is disk drives/controller caches lying and reporting write completion when not all data has actually reached stable storage
> However, we will figure out what went wrong and ensure that that particular error doesn’t happen again.
How can you say statement 2 just after statement 1 ? Isn't statement 1 just plain acceptance of defeat ?
And looking at all the replies here, is this a feel good thread for the mistakes you made ?
Acknowledging statement 1 doesn’t mean giving up—it simply means being clear and realistic about the nature and scale of the problem we’re facing when we try to build complex software systems. In the face of that we can give up, or we can just do the best we can, and it sounds more like these people are doing the latter.
If I screw up one parameter, instead of deleting only unconfirmed users, I could delete all users. I have two redundant checks, first when the query is run to get the unconfirmed users, and then again checking the user's confirmed status before deleting them. And then I check one more time further down in the code for good measure. Not because I think the result will be different, but just in case one of the lines of code is altered somehow.
I put BIG LOUD comments everywhere of course. But it still terrifies me.
Otherwise having a binlog based backup (or WAL, I guess, but i don’t know PG that well) is critical.
The key point there is they provide point in time recovery possibilities (and even the ability to rewrite history).
(1) https://www.pgbarman.org/
> It’s tempting to blame the disaster on the couple of glasses of red wine. However, the function that wiped the database was written whilst sober.
It was _written_ then, but you're still admitting to the world that your employees do work on production systems after they've been drinking. Since they were working so late, one might think this was emergency work, but it says "doing some late evening coding". I think this really highlights the need to separate work time from leisure time.
In this case there were like 10 relatively easy things that could have prevented this. Your ability to mentally compile and evaluate your code before you hit enter is not a reliable way to protect your production systems.
Coding after drinking is probably not a good idea of course, but “think better” is not the right takeaway from this.
I’ve done some of my most productive work this way. Not on production systems fortunately, and not in a long time.
For ref: https://xkcd.com/323/
Maybe such a breathalyzer interlock could be installed on your workstation too. After all, your systems and processes should prevent you from stupid things.
This person had to publicly apologize to their customers. One or two low-cost guardrails could have prevented it and would probably have been worth the cost.
(But if your mission critical system relies on that scoreboard website, that's on you...)
A problem with devices of that type is that they only test for a potential source of inability to drive safely. What we want is to test for an inability to drive safely.
And while one is easy and might give some quick wins, the drawbacks scare me too much.
Actually getting in the car while under the influence such of stress and alcohol sounds worse.
I know someone who had a glass of wine just before her daughter needed to be brought to the hospital. This was just two days ago. She simply concluded she could not drive. Luckily, she was able to get a taxi.
I'm sure that would save a lot of lives. An ambulatory medical service, if you will.
I once dislocated my shoulder while on a large trampoline and was unable to get up from my hands and knees due to the intense pain whenever the trampoline wobbled. The ambulance was redirected to more serious injuries three times. I was stuck in that position waiting for two hours before it arrived.
In that scenario it would also be appropriate to wait for a driver to sober up before driving you to the hospital if neither ambulance nor taxi were available (or delayed). One glass of wine would be out of most people's systems after two hours.
Thus poking hole in the "drunk drive someone to the hospital" argument, which is what this was all about in the first place.
In my previous comment I just meant that sometimes ambulances can take a good while and a taxi might not.
In the unfortunate case of the trampoline there were several sober people with driver's license and cars available and a taxi would have been there immediately.
Unfortunately,they failed to get me out of there, meaning I still had to wait until an ambulance was available. It was beyond painful and exhausting both physically and mentally. But it was still technically not an emergency.
While your friend made a call judging their own abilities and the level of emergency, that's exactly how it should be: cars should not stop us humans for making that decision.
(Fwiw, if you were just having an alcoholic drink, a breathalizer would show a much higher concentration even though alcohol might not have even kicked in or there wasn't enough for it to kivk in at all)
I heard the same argument for electronic gun safety measures, except that no government agency even consider using it for their own guns. Why? They are not reliable enough, yet.
If there was more of a cultural pressure against drunk driving and actual mechanisms to prevent it that aren't too difficult to maintain and utilize, things like the Daikou services ( https://www.quora.com/How-effective-is-the-Japanese-daikou-s... ) would pop up and take care of the other logistical problems of getting your car home. And the world would be all the better for it, because of less drunk driving and accidents.
[1] https://media.daimler.com/marsMediaSite/en/instance/ko/ATTEN...
That is because most companies these days have processes around drinking in workplace, coming in drunk and working drunk.
Most mistakes are done sober only in environments where drinking couple of vine cups and then doing production change is considered unacceptable. In environment where drunk people work, mistakes are made when drunk.
Of course the next day, when re-reading the same pages, I was always discovering that the previous day I had everything wrong, nothing was obvious, and all my reasoning when with alcohol was false because simplistic and oblivious of any mathematical rigor.
Also, nothing is comparable between alcohol and psychodelics
In short, ability to recall memories is at least in part dependent on being in a similar state to the time when memories are formed, e.g. something learned while being intoxicated will be more easily recalled only when intoxicated again.
Though I'm talking one or two drinks here, not firing up vscode after a night out or going through a bottle of rum.
Also I agree with other comments: doing some work after a glass or two should be fine because you should have other defences in place. “Not being drunk” shouldn’t be the only protection you have against disaster.
But it's just a side-project and I will continue late night coding with a glass of wine. I find it hugely enjoyable.
I would have a different mind-set if I was writing software for power stations as a professional.
Normally, this would be fine. But, it appears the site has paying members. Presumably, it's not "just a side-project" to them. You owe them better than tinkering with prod while tipsy.
The bigger issue is the lack of guardrails to protect against accidental damage. This is also a common trait for hobby projects (after all, it's more fun to hack stuff together) but hopefully the maintainer will use this experience as a sobering reminder (pun intended) to put some guardrails in place now.