I share 3 stories about my previous experience when I deleted not on purpose data in my Hadoop clusters or when we ran terraform destroy over our GCP projects.
This is a personal post about what I learnt and some takeaways.
IAC promise was ease of re-creation. [My anecdotal] Reality shows that's not really the case. IAC automation helps to create 'easy parts' and to delete _everything_. Creation of hard pars, the ones which involve state is still a problem. After many ears of using cfengine/chef/puppet/ansible/terraform I can say those tools don't help much with solving infra complexity. Early arguments were that 'nobody understand those shell scrips' or 'know manually edited configs'. Now the same shell script live inside container 'sidecars' and do 'pip install' from them and instead of manually altered configs we have k8s annotations with unpredictable 'far reaching' inter-references.
The hard stateful parts should be your data layer (replicated db, etc) and everything else should be cached stateless data. IAC for the stateful layer is tough but contained. IAC for the rest should be fairly recreatable quite quickly.
> Now the same shell script live inside container 'sidecars' and do 'pip install' from them and instead of manually altered configs we have k8s annotations with unpredictable 'far reaching' inter-references.
This sounds really wrong - installation of all of your dependencies should be run as a part of building the container for a particular version of your application, say, as a directive within the Dockerfile.
That should also be done before any unit tests and integration tests are run against that container, well before it is deployed to any environment. The end result of doing that properly is a container that will work correctly until the end of time (or at least any vulnerabilities become apparent and a new version needs to be built, or for other reasons, such as bug fixes or added functionality).
Furthermore, all of your IAC should be treated just like regular code, which means full descriptions and comments for any non-trivial parts, or even explaining why the trivial parts are there to begin with (the greater circumstances, links to change requests etc. that the actual code doesn't otherwise contain). If you can't look at any single piece of your code and understand both what it does as well as why it's attempting to do it, then clearly there is still more work to be done. Whether that's refactoring or documentation, however, depends on the circumstances.
IAC doesn't absolve you from the need of having documentation. It simply allows you to lower the degree to which PEBKAC is a problem in manual deployments and allows you to document the actual code that does things, rather than creating .docx descriptions that no one will read.
A friend of mine once ran the Ansible equivalent of terraform destroy in our aws prod env in the middle of the day (of a fintech company). I could see how several if our prod machines got terminated. Fortunately the ones that mattered the most, had termination protection. But it was a scary moment.
This is why he is Data Engineering Coach and not actually responsible for production systems now. (apparently)
Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless!
"But this kind of thing could happen to anyone" - sure, anyone who is irresponsible.
Am I ranting? Let me continue. I work on a team now with someone who seems to consistently forget to save router configurations and another person who yawns through meetings because he doesn't sleep. These guys make careless mistakes and we "learn" from them. Except we don't learn the most important lesson: Irresponsible people need to be off the team. They make more work for everyone else and make the team look bad.
But I guess the tech horror stories that amuse everyone on the forum or at the pub are better than saying, "I am responsible and do good work."
Read the article. We're not talking about formal verification.
>It's Sunday morning and I just discovered that I've lost 3To of data and that all data pipelines have stop working because on Friday I ran for no reason
In my experience some people keep making the same careless mistakes, the first time, you let it pass, treat it as a learning experience. The second time, you start seeing that it's always the same person doing the same mistake.
Copy pasting is not an excuse, before you run anything destructive, you double check what you're running.
Anyone who is responsible will double check before running this kind of command, and if I can't get that person off the team, I'd severely restrict his access (in general, I think most people in a team should not have access to production data).
And we do have disaster recovery plans so there's very little that could be done that would be catastrophic. But still, a lot of the disaster recovery plans call for downtime because it's not worth the cost benefit to engineer the system to be completely resilient to idiocy.
Modern terminals marks pasted data as pasted, and similarly modern shells detect these marks and do not run the pasted data immediately, but shows it, highlighted, so that you can review it before confirming it.
Back in 2014, my first tech job, I wrote a clean-up script to delete HDFS artifacts listed in some text file. One day I modified the list and left a blank line at the end. :)
I mean if people did not make mistakes we would all be writing directly in binary target or assembly. Should you ever delete folders named data -- probably not without knowing what you are deleting. Should you place your production data on a single machine -- definitely not if you know anything about production or complex (file) systems. I mean it could be literally your computer dying on you. You trust your company on that single hard drive not dying really? Or any system bug -- OS/firmware etc? Are you that crazy? Just short S&P long term -- it's less risky.
> Should you place your production data on a single machine -- definitely not if you know anything about production or complex (file) systems. I mean it could be literally your computer dying on you. You trust your company on that single hard drive not dying really? Or any system bug -- OS/firmware etc? Are you that crazy? Just short S&P long term -- it's less risky.
OP was using HDFS which is a distributed/clustered filesystem, their data was replicated across multiple machines. But "rm" still deletes it from everywhere.
Mistakes happen. People forget which environment they are on. people forget where statement in their DELETE query. People misremember their cwd prior to running rm. Sometimes it's lack of experience, sometimes lack of sleep, sometimes just shit luck. Very rarely it's lack of responsibility.
And regarding your team members: if you don't trust them, don't rant about it on public forum, man, move them away from important systems, implement personal improvement plan, and when that fails, fire them.
Experienced solutions architects prevent mistakes from being costly with responsible infrastructure design, disaster recovery plans, and solid management workflows.
Companies cut corners by not hiring architects because they are not cheap, and then they cut corners on salary for roles on mission critical projects because they are also cheap, and they often pay the price anyway just later-on because failure is expensive.
Hire experienced people for the right money if it's a mission-critical system. The simple facts are all there.
Nobody successful designs a good building without a good architect, and the project never turns out well when pay isn't right.
Also important to note: The architect should not also be the one to physically build the "house", in order to maintain proper objectivity and in order to avoid conflict, even if the company insists on being cheap.
I do not see where I said that. I am sorry, but you're projecting. Safety rules are written in blood; production safety rules are written in critical incidents and cold sweat. To err is human.
Good old "human error", be glad it happens for it means you still have a job that hasn't been automated (yet).
Easy to mess things up when you routinely execute a bunch of commands everyday, many of which look similar in form but differ greatly in the pains they can create
I really like the effort you made in following more than 2 links to see what I'm doing. The issue is that you also don't have the full context on everything that happened.
I don't know on which kind of teams you work in, but I personally worked a lot in teams where I was the only person responsible to do data related tech stuff for other people. I never said that I wasn't responsible but I was alone, so setuping backup was often the last of my priorities. I was and I did all my best to fix things as fast as possible and to remove any issues it could lead in the future.
When I was setuping Hadoop cluster in 2014, almost no-one knows what it was in my local market. So let us try, then fail and learn.
If you also think from 3 stories I'm writing on the internet that I'm careless so be it. But I also recommend you to fix your "router configuration" problem rather than ranting on posts on internet.
I’ve worked with similar people and had similar experiences. They are negligent, never question their own abilities nor actions, and to compound this they don’t learn from the experiences.
Whilst no-one is perfect, it’s always possible to test before proceeding to cause uncontrolled havoc. Just remind oneself, check, check, check.
Assume that there will be problems before proceeding; forewarned is forearmed.
As they said in 'The Expanse'; welcome to the churn.
I've been on both sides of this - the person that wants everyone to think twice before they type... but also the one that messed up.
I'm really here just to say that... sometimes things aren't so clean-cut.
For example: 'Business needs' often push myself and others to make calls that we'd normally never make. Coming from the top, down - executives rarely care about 'fact'.
Just what they promised, and everyone suffers for it.
In the end, the whole experience is a wash. We don't hit the target, and everyone is more stressed out. Repeat this enough, the diligence of most will falter.
> Except we don't learn the most important lesson: Irresponsible people need to be off the team.
This is false dichotomy.
If people can screw something up, sooner or later they will. It doesn't matter how responsible or irresponsible they are - the processes in place should prevent mistakes from being made regardless of that, or to mitigate their consequences if they're unavoidable. Anything less and you're not addressing the root cause of the issue. These same processes should ensure that no one can create changes to the state of the system before them first going through another set of eyes and being validated.
Most sane OSes at least prompt you before deleting a file - the very kind of safeguard that makes you double check whether what you're doing makes sense. Similarly, you'll notice that cars have seat belts and air bags, even if you're not going to crash daily. Get the irresponsible people off the team if you'd like, but if you can't put the appropriate processes in place to prevent mistakes, you probably need to rethink your priorities and provide the adequate pushback against "the business", when they expect you to SSH into prod and do anything.
What that looks like in my current environment:
- all server configuration is managed through Ansible and Git
- developers have read only access to the servers when needed, no one can change the state of the system there
- all changes are versioned and use merge requests and automated CI processes, need to be reviewed first and synced with the change management system, to know who is attempting to change something, why and also how (with the appropriate permissions)
- furthermore, the Ansible processes are scheduled to run every day as well, just to make sure that the server status is as expected
- of course, the changes to servers are also first done on development environments, then on accept testing environments, then on clients' testing environments (any number of them that's necessary) and then finally to prod
- there is Zabbix for monitoring the infrastructure with alerting in place, as well as Skywalking for application APM, as well as some lower level tools
- the apps are also shipped as containers, which have very similar processes in place, e.g. full CI run on every merge request, before it gets merged
- those containers run in clusters, so if any of them fail for whatever reason, the load will be balanced as necessary and/or restarts will take place, each container also having health checks
- lots of manual QA in there as well, to catch the things that aren't easily automated, as well as regression testing
- automated integration and load tests done every now and then, to check that there are no regressions in a new release
- testing the reproducibility of this can also be done by wiping any server and letting the CI processes restore a new one, complete with the appropriate access roles, firewall configuration, cluster membership, observation tools for developers to use etc.
- everything also has backups and rollback strategies as needed
- despite all of the procedures in place, delivering a new version can be as easy as clicking a button on a CI pipeline, the container images being delivered and becoming available on the clients' side in a few minutes for further processing
Of course, that's still not good enough in my eyes and there are further improvements that can be done, as well as problems in place that prevent truly safe and easy development (e.g. no adherence to 12 Factor App principles due to historical reasons). If the projects used TDD and there was 90% test coverage, as well as ALL of the functionality was to be covered by integration tests (Selenium), then we could actually start talking about software engineering. Until then, i don't believe that the term is really apt, perhaps outside of the aerospace industry.
Disclaimer: of course, the amount of effort that goes into designing a...
Oh. Well why don’t you use your frustration into writing a message to the careless team member and firing him from the team? If you are working in a non professional team it doesn’t mean that others don’t learn from theirs mistakes. Also, how did end up in the company with such low quality team members?
I agree with that in principle, but I think the things OP writes about are egregiously bad. Both in the sense of the employees in question being incredibly careless, and the company being negligent in developing process around having this sort of "god mode" access.
I've seen my share of honest mistakes (and committed some of them), but... damn.
> Never blame the responsible, I prefer to think that if the mistake happened it's because the team or the company let the issue happens — be also careful when you joke about it afterwards
Although certainly people still do bad things, dumb things, and careless things -- this (the middle part -- team/company controls and culture) is a wise management approach, although best applied as a principle ahead-of-time.
I lost a PM job once because a new-hire junior admin deleted a symlink and took a major site down... The CEO who fired me was upset that I couldn't restore the massive database and run a diff on it versus a local backup I made out of being overly cautious the day prior.
I was supposed to just be a PM, not even supposed to be a Dev... Served me right for acting like I might have been able to fix the issue. No more volunteering outside of my role... Possibly also why job descriptions include so much out of normal scope "responsibility" now.
The company had a bad habit of underpaying roles, and regularly hired inexperienced admins to do key PROD work for clients.
I was also blacklisted for rehire from the company and took the fall I guess when the issue was explained to the clients.
Oh well.. Spilled milk... The company kept failing after I left and lost all it's major contracts weeks after I got cut anyway. I got a decent severance payout and still won a 25% raise on my next gig... Moved forward without scratches.
Lessons -
Back everything up well, no matter how much time it takes, before touching PROD or even before others touch PROD.
Don't work for places that are regularly dedicated to underpaying talent to do mission critical work.
When a door closes, another opens, learn from the past, but don't carry it forward, and don't fight ignorant and tech-blind arrogance in leadership, it only hinders your career growth if you crusade against bad leadership. Karma does the real work for you eventually if you remain calm & composed.
Quickly identify if tech talent and problem solving stops at your level, and if so, protect yourself from being a scapegoat for leadership failures.
Don't be the one that screws things up, be the one that both anticipates and quickly and fixes all varieties of screw-ups from everyone else.
> I lost a PM job once because a new-hire junior admin deleted a symlink and took a major site down... The CEO who fired me was upset that I couldn't restore the massive database and run a diff on it versus a local backup I made out of being overly cautious the day prior.
But... the database wasn't gone? If you delete a symlink, you can just put it back.
The (devops) admin deleted the link to an assets folder for all the file uploads for a pretty large web site and didn't even know how to explain what he did. We all didn't know it was a symlink that was deleted until later after the failure, but the disconnection caused the entire site's db to go corrupt... I restored the link but that didn't solve the resulting corruption, so a DB recovery/diff was in process from my desktop backup when they turned to blame on me...The DB was really FUBARED... Actually quite glad I didn't have to finish the fix on my own after all that drama TBH. :/
If you are working later at night over the weekend something bad is going to happen (except if you work on a Wednesdays off Sundays working days schedule).
If you mix production with dev something bad is going to happen.
If you expect no mistakes to happen, well, something bad is going to happen. Authorization and prompts are there for a reason.
@comments discussing about dismissing OP -- doesn't matter: you don't have the data anymore. Production data should be indestructible practically or it doesn't matter.
Sure your client/user is going to be OK if you fired your junior dev Bob... Just comment how many times that has worked.
I've had someone do, in essence, a `terraform destroy`. I don't think we ever learned if it was that, or some other command … just that it was indeed a terraform removal. The plan was not read, of course. Instant incident.
More recently, had a coworker want to run terraform plan, couldn't, because the state file is access controlled, and so he just reset the configuration locally to a blank local state, plan (which wasn't read, again), and then subsequently created a lot of duplicate resources. (Since TF was essentially starting from a blank slate.)
We had an admin interface that used a generic UI; every database table was just a generic CRUD "it's a table, here's the columns & values" to it. The mobile app was an OAuth app, just like any third-party app was. One day, someone on the QA team (why did QA have access to prod, you ask? Good question…) deleted the mobile app's OAuth client from the database. "Why did you do this?" :shrug: "Why do we even have this lever, Kronk?" I suggested at the time we should have kept the delete button, but special cased it to remove the admin privs of anyone who clicked it, since clearly they failed the test…
I can't tell why the copy in the article didn't work? You should be able to move /usr to another partition, no? (I'm not sure I'd want to try that on a live OS … and … if this is the cloud, usually growing the underlying EBS or cloud disk or whatever is pretty easy to do? But I also usually try to keep any "it could grow" data on a separate partition to spare / the pain…) Though I think I'd bind-mount it back in place, not symlink it.
> I can't tell why the copy in the article didn't work? You should be able to move /usr to another partition, no?
The way it reads, they didn't know how POSIX permissions work, and maybe still don't. The article says that sudo needs to be owned by uid 0, but does not mention the critical suid bit. Maybe they just recursively copied /usr without preserving permissions?
Also, the mention of sudo makes it pretty clear that they did not do this in an interactive root shell (obtainable with sudo -s/-i or su if there is a root pw). If you do this kind of shenanigans, you better have multiple open root shells in front of you, and some statically linked binaries (busybox?) to recover from the inevitable chaos of missing binaries, shared libraries, and dynamic loaders that you'll transiently have.[1] It's not impossible to perform, but requires careful planning and a good understanding.
[1] Though if /usr was indeed copied without preserving permissions, recovering would have been very tedious.
Yeah I made some shortcuts about it in the post for the sake of trying to keep it short. But the biggest issue with sudo was the suid bit you're right.
You're totally right in your message. I did not had access to root password.
I was junior and while I was trying to fix stuff I was adding more problem to the initial one that was a disk space issue.
My favourite was when a "helpfull" sysadmin deleted all the ActiveMQ data(base) files for a prod system (containing unprocessed queue items) I assume they got a low disk space alert and started deleting big files with no notice, recreating the messages from logs was fun :{
I'm really torn on this stuff. On one hand, the big failure is a company process one: people should not have the permissions to just run "terraform destroy" and have it actually destroy a production environment (ditto for the HDFS /data deletion). Or at the very least, there should be a strong culture of never getting into the position of being able to do that without someone looking over your shoulder, double-checking every command before you run it.
But... some of this just feels like carelessness. I've certainly made my share of mistakes, but these just feel egregiously bad. This takeaway in particular is just all wrong:
> Measure the risk when you give all the permissions to one developer, one data engineer or one SRE — it means similar stories could happen
There is no need to measure that risk, because no one should have permissions to take down your production infrastructure so easily. This just seems like table stakes for running or working at a company that operates this kind of infrastructure.
I think it's three things: first, abide by the principle of least privilege, and make company-destroying permissions hard to come by; second, put safe (web, command-line, whatever) interfaces in front of common tasks that need to be done that could turn into accidental downtime; and third, when it's required that you do things outside of the safe interfaces, drill it into people that you never do them without a copilot who can check over your work, in real-time, before you do anything (and if anyone develops a reputation for being a cowboy... seriously, fire them).
There are three examples in the article: removing /usr (!!), running "hdfs dfs -rm /data" and "terraform destroy". Even with basic knowledge, these are clearly destructive operations - and to top it all, being run in Production. What strikes me as odd is that in all the cases, there is no change control. And by that I don't necessarily mean a full-on ServiceNow-type change process; I mean there was no "review". Even if you are a small firm, it is not difficult to ensure that you ask atleast one colleague to review any destructive step in Production prior to executing. Ignoring to do so is inviting disaster.
Talkinf of personal stories, I'd be very happy to have a tool that enforces some verification process over SQL operation ran on production database (you know, to fix that horrible data quality issue on 2 rows that screws the whole business logic in the upper layer that would take a month to fix).
Basically a tool that allow someone to run some SQL on production but, before doing it, request his credentials, then run the sql command in some "--dry-run" mode and if you modify more than 10 rows, then it stops everything. And if it goes on, record the SQL that was run somewhere so we can audit...
As a first step toward your wish, I suggest using mycli or pgcli on servers instead of the default mysql or pgsql client. By default, it asks for confirmation whenever you type a destructive command. It does have a log, just like the official CLI clients. It also provides completion and syntax highlighting.
https://github.com/dbcli/
Instead of playing an elaborate finger pointing game, we decided to develop internal tools which put our most dangerous operational activities on rails.
It's amazing how much impact a small console application can have on preventing mistakes. All you really need to do is wrap the danger with "are you sure?" and "enter the exact commit hash..." kinds of prompts. For the super dangerous stuff you can bake in some centralization and approval loops.
That's an interesting project, however you might want to consider slightly altering the CSS for the homepage: http://rupy.se/
Currently the main font:
font-size: 0.6em;
ends up being around 9.6px on Firefox with default settings in regards to font size, which makes it pretty much unreadable on a 1920x1080 monitor (21.5").
> Create a good wheel environment, ask for help and do hide stuff from colleagues
(emphasis mine) Was this a typo? If it is, it's kind of funny that an article about making mistakes in production also made this tiny but critical mistake.
I have sympathy for the author and find a lot of the criticism a bit harsh.
On my teams, I tend to be the person put in the role of doing DevOps-ish work because it's something I'm familiar with. I honestly find it incredibly stress inducing. It seems to me there is an asymmetry in the risk of work between team members that never gets captured by management.
There are degrees of difference with security and data concerns in regards to frontend versus backend versus devops tasks. There is some work that can carry the risk of permanent and irreversible data loss, yet everywhere I've worked, it is apportioned as a regular tasks. Failures like Ops are first and foremost an organizational issue.
Hey, OP here. I wasn't ready to get so much negativity from my post. OK the title was a bit clickbait but I just shared honest feedback because I was sure that a lot of people could recognize from my stories.
I never said I was an expert in the post, I never said I consider myself better than everyone, and moreover I'm 100% sure I can bet I've not impacted the life of one person that commented my post.
Is it my fault when I just graduated that I had been given the responsibility of building an Hadoop cluster for a +$100m revenue company? No, I just tried to do my best. I fucked up things and I fixed it. Period.
I'd love to work with all people telling me I was incompetent, but unfortunately you weren't here to bring me the light in 2014.
7 years after I can sleep at night and live with my mistakes. So, yeah, sorry for that.
It reminds me that back in July 2019. I was using the Windows file explorer to investigate an issue. When clicking on a folder to open it, my wrist slightly twitched while clicking on the folder, which led to the accidental move of a 650 GB operational data folder into a code repository folder.
For some reason the move into the repository folder was instantaneous but the opposite wasn't. All data import attempts errored during the 14 hours required for the data to be moved back to its original place. I emailed and voiced my apologies to the team present both in India and in France.
While I was not fired on the spot, it might have indirectly led to the disappearance of our devops (devoops?) team in Paris. Fortunately enough I guess I was the last team member to be sent away.
For my defense, our dev team had a intervention scope so big that they were essentially super-men once they have been granted access. One the rights granted to us was full write-access to all incoming data folders.
67 comments
[ 2.8 ms ] story [ 208 ms ] threadThis is a personal post about what I learnt and some takeaways.
Did you ever feel the same way?
Quoi? Do *not* hide stuff from colleagues.
That's probably what ended up causing `terraform destroy`.
Do not hide stuff from your coworkers. It's toxic in a workplace. Hopefully it's a typo.
This sounds really wrong - installation of all of your dependencies should be run as a part of building the container for a particular version of your application, say, as a directive within the Dockerfile.
That should also be done before any unit tests and integration tests are run against that container, well before it is deployed to any environment. The end result of doing that properly is a container that will work correctly until the end of time (or at least any vulnerabilities become apparent and a new version needs to be built, or for other reasons, such as bug fixes or added functionality).
Furthermore, all of your IAC should be treated just like regular code, which means full descriptions and comments for any non-trivial parts, or even explaining why the trivial parts are there to begin with (the greater circumstances, links to change requests etc. that the actual code doesn't otherwise contain). If you can't look at any single piece of your code and understand both what it does as well as why it's attempting to do it, then clearly there is still more work to be done. Whether that's refactoring or documentation, however, depends on the circumstances.
IAC doesn't absolve you from the need of having documentation. It simply allows you to lower the degree to which PEBKAC is a problem in manual deployments and allows you to document the actual code that does things, rather than creating .docx descriptions that no one will read.
Everyone loves reading tech horror stories and peeling off the take-away lessons. My lesson would be, don't hire this guy and don't use him as a coach. He's careless!
"But this kind of thing could happen to anyone" - sure, anyone who is irresponsible.
Am I ranting? Let me continue. I work on a team now with someone who seems to consistently forget to save router configurations and another person who yawns through meetings because he doesn't sleep. These guys make careless mistakes and we "learn" from them. Except we don't learn the most important lesson: Irresponsible people need to be off the team. They make more work for everyone else and make the team look bad.
But I guess the tech horror stories that amuse everyone on the forum or at the pub are better than saying, "I am responsible and do good work."
Be humble because the complexity of modern systems is insane, there is no way you have all cases covered, if you had you would had a formal proof.
We all are just doing our best to cover most of the edge cases. That is why we need to keep learning from other peoples mistakes.
>It's Sunday morning and I just discovered that I've lost 3To of data and that all data pipelines have stop working because on Friday I ran for no reason
This is profound incompetence.Copy pasting is not an excuse, before you run anything destructive, you double check what you're running.
Anyone who is responsible will double check before running this kind of command, and if I can't get that person off the team, I'd severely restrict his access (in general, I think most people in a team should not have access to production data).
And we do have disaster recovery plans so there's very little that could be done that would be catastrophic. But still, a lot of the disaster recovery plans call for downtime because it's not worth the cost benefit to engineer the system to be completely resilient to idiocy.
We had nightly back-ups though.
OP was using HDFS which is a distributed/clustered filesystem, their data was replicated across multiple machines. But "rm" still deletes it from everywhere.
And regarding your team members: if you don't trust them, don't rant about it on public forum, man, move them away from important systems, implement personal improvement plan, and when that fails, fire them.
Companies cut corners by not hiring architects because they are not cheap, and then they cut corners on salary for roles on mission critical projects because they are also cheap, and they often pay the price anyway just later-on because failure is expensive.
Hire experienced people for the right money if it's a mission-critical system. The simple facts are all there.
Nobody successful designs a good building without a good architect, and the project never turns out well when pay isn't right.
Also important to note: The architect should not also be the one to physically build the "house", in order to maintain proper objectivity and in order to avoid conflict, even if the company insists on being cheap.
It's kind of "welcome to the club" right of passage
Easy to mess things up when you routinely execute a bunch of commands everyday, many of which look similar in form but differ greatly in the pains they can create
Yes, everyone should experience making catastrophic mistakes but please spare some effort to only do it in non-prod...
Friendly advice: you don't sound like you rant, you sound like something else.
I don't know on which kind of teams you work in, but I personally worked a lot in teams where I was the only person responsible to do data related tech stuff for other people. I never said that I wasn't responsible but I was alone, so setuping backup was often the last of my priorities. I was and I did all my best to fix things as fast as possible and to remove any issues it could lead in the future.
When I was setuping Hadoop cluster in 2014, almost no-one knows what it was in my local market. So let us try, then fail and learn.
If you also think from 3 stories I'm writing on the internet that I'm careless so be it. But I also recommend you to fix your "router configuration" problem rather than ranting on posts on internet.
I’ve worked with similar people and had similar experiences. They are negligent, never question their own abilities nor actions, and to compound this they don’t learn from the experiences.
Whilst no-one is perfect, it’s always possible to test before proceeding to cause uncontrolled havoc. Just remind oneself, check, check, check.
Assume that there will be problems before proceeding; forewarned is forearmed.
I've been on both sides of this - the person that wants everyone to think twice before they type... but also the one that messed up.
I'm really here just to say that... sometimes things aren't so clean-cut.
For example: 'Business needs' often push myself and others to make calls that we'd normally never make. Coming from the top, down - executives rarely care about 'fact'.
Just what they promised, and everyone suffers for it.
In the end, the whole experience is a wash. We don't hit the target, and everyone is more stressed out. Repeat this enough, the diligence of most will falter.
This is false dichotomy.
If people can screw something up, sooner or later they will. It doesn't matter how responsible or irresponsible they are - the processes in place should prevent mistakes from being made regardless of that, or to mitigate their consequences if they're unavoidable. Anything less and you're not addressing the root cause of the issue. These same processes should ensure that no one can create changes to the state of the system before them first going through another set of eyes and being validated.
Most sane OSes at least prompt you before deleting a file - the very kind of safeguard that makes you double check whether what you're doing makes sense. Similarly, you'll notice that cars have seat belts and air bags, even if you're not going to crash daily. Get the irresponsible people off the team if you'd like, but if you can't put the appropriate processes in place to prevent mistakes, you probably need to rethink your priorities and provide the adequate pushback against "the business", when they expect you to SSH into prod and do anything.
What that looks like in my current environment:
Of course, that's still not good enough in my eyes and there are further improvements that can be done, as well as problems in place that prevent truly safe and easy development (e.g. no adherence to 12 Factor App principles due to historical reasons). If the projects used TDD and there was 90% test coverage, as well as ALL of the functionality was to be covered by integration tests (Selenium), then we could actually start talking about software engineering. Until then, i don't believe that the term is really apt, perhaps outside of the aerospace industry.Disclaimer: of course, the amount of effort that goes into designing a...
I've seen my share of honest mistakes (and committed some of them), but... damn.
Although certainly people still do bad things, dumb things, and careless things -- this (the middle part -- team/company controls and culture) is a wise management approach, although best applied as a principle ahead-of-time.
I was supposed to just be a PM, not even supposed to be a Dev... Served me right for acting like I might have been able to fix the issue. No more volunteering outside of my role... Possibly also why job descriptions include so much out of normal scope "responsibility" now.
The company had a bad habit of underpaying roles, and regularly hired inexperienced admins to do key PROD work for clients.
I was also blacklisted for rehire from the company and took the fall I guess when the issue was explained to the clients.
Oh well.. Spilled milk... The company kept failing after I left and lost all it's major contracts weeks after I got cut anyway. I got a decent severance payout and still won a 25% raise on my next gig... Moved forward without scratches.
Lessons -
Back everything up well, no matter how much time it takes, before touching PROD or even before others touch PROD.
Don't work for places that are regularly dedicated to underpaying talent to do mission critical work.
When a door closes, another opens, learn from the past, but don't carry it forward, and don't fight ignorant and tech-blind arrogance in leadership, it only hinders your career growth if you crusade against bad leadership. Karma does the real work for you eventually if you remain calm & composed.
Quickly identify if tech talent and problem solving stops at your level, and if so, protect yourself from being a scapegoat for leadership failures.
Don't be the one that screws things up, be the one that both anticipates and quickly and fixes all varieties of screw-ups from everyone else.
But... the database wasn't gone? If you delete a symlink, you can just put it back.
If you mix production with dev something bad is going to happen.
If you expect no mistakes to happen, well, something bad is going to happen. Authorization and prompts are there for a reason.
@comments discussing about dismissing OP -- doesn't matter: you don't have the data anymore. Production data should be indestructible practically or it doesn't matter.
Sure your client/user is going to be OK if you fired your junior dev Bob... Just comment how many times that has worked.
More recently, had a coworker want to run terraform plan, couldn't, because the state file is access controlled, and so he just reset the configuration locally to a blank local state, plan (which wasn't read, again), and then subsequently created a lot of duplicate resources. (Since TF was essentially starting from a blank slate.)
We had an admin interface that used a generic UI; every database table was just a generic CRUD "it's a table, here's the columns & values" to it. The mobile app was an OAuth app, just like any third-party app was. One day, someone on the QA team (why did QA have access to prod, you ask? Good question…) deleted the mobile app's OAuth client from the database. "Why did you do this?" :shrug: "Why do we even have this lever, Kronk?" I suggested at the time we should have kept the delete button, but special cased it to remove the admin privs of anyone who clicked it, since clearly they failed the test…
I can't tell why the copy in the article didn't work? You should be able to move /usr to another partition, no? (I'm not sure I'd want to try that on a live OS … and … if this is the cloud, usually growing the underlying EBS or cloud disk or whatever is pretty easy to do? But I also usually try to keep any "it could grow" data on a separate partition to spare / the pain…) Though I think I'd bind-mount it back in place, not symlink it.
The way it reads, they didn't know how POSIX permissions work, and maybe still don't. The article says that sudo needs to be owned by uid 0, but does not mention the critical suid bit. Maybe they just recursively copied /usr without preserving permissions?
Also, the mention of sudo makes it pretty clear that they did not do this in an interactive root shell (obtainable with sudo -s/-i or su if there is a root pw). If you do this kind of shenanigans, you better have multiple open root shells in front of you, and some statically linked binaries (busybox?) to recover from the inevitable chaos of missing binaries, shared libraries, and dynamic loaders that you'll transiently have.[1] It's not impossible to perform, but requires careful planning and a good understanding.
[1] Though if /usr was indeed copied without preserving permissions, recovering would have been very tedious.
You're totally right in your message. I did not had access to root password.
I was junior and while I was trying to fix stuff I was adding more problem to the initial one that was a disk space issue.
But... some of this just feels like carelessness. I've certainly made my share of mistakes, but these just feel egregiously bad. This takeaway in particular is just all wrong:
> Measure the risk when you give all the permissions to one developer, one data engineer or one SRE — it means similar stories could happen
There is no need to measure that risk, because no one should have permissions to take down your production infrastructure so easily. This just seems like table stakes for running or working at a company that operates this kind of infrastructure.
I think it's three things: first, abide by the principle of least privilege, and make company-destroying permissions hard to come by; second, put safe (web, command-line, whatever) interfaces in front of common tasks that need to be done that could turn into accidental downtime; and third, when it's required that you do things outside of the safe interfaces, drill it into people that you never do them without a copilot who can check over your work, in real-time, before you do anything (and if anyone develops a reputation for being a cowboy... seriously, fire them).
Not _these_ mistakes, though. The author describes incompetence, nothing else.
Basically a tool that allow someone to run some SQL on production but, before doing it, request his credentials, then run the sql command in some "--dry-run" mode and if you modify more than 10 rows, then it stops everything. And if it goes on, record the SQL that was run somewhere so we can audit...
It's amazing how much impact a small console application can have on preventing mistakes. All you really need to do is wrap the danger with "are you sure?" and "enter the exact commit hash..." kinds of prompts. For the super dangerous stuff you can bake in some centralization and approval loops.
That way it's almost impossible to have anything catastrophic happen unless the whole planet goes boom!
Currently the main font:
ends up being around 9.6px on Firefox with default settings in regards to font size, which makes it pretty much unreadable on a 1920x1080 monitor (21.5").Example screenshot: https://imgur.com/IsPvHqE
(emphasis mine) Was this a typo? If it is, it's kind of funny that an article about making mistakes in production also made this tiny but critical mistake.
On my teams, I tend to be the person put in the role of doing DevOps-ish work because it's something I'm familiar with. I honestly find it incredibly stress inducing. It seems to me there is an asymmetry in the risk of work between team members that never gets captured by management.
There are degrees of difference with security and data concerns in regards to frontend versus backend versus devops tasks. There is some work that can carry the risk of permanent and irreversible data loss, yet everywhere I've worked, it is apportioned as a regular tasks. Failures like Ops are first and foremost an organizational issue.
I never said I was an expert in the post, I never said I consider myself better than everyone, and moreover I'm 100% sure I can bet I've not impacted the life of one person that commented my post.
Is it my fault when I just graduated that I had been given the responsibility of building an Hadoop cluster for a +$100m revenue company? No, I just tried to do my best. I fucked up things and I fixed it. Period.
I'd love to work with all people telling me I was incompetent, but unfortunately you weren't here to bring me the light in 2014.
7 years after I can sleep at night and live with my mistakes. So, yeah, sorry for that.
I'm closing this, love.
For some reason the move into the repository folder was instantaneous but the opposite wasn't. All data import attempts errored during the 14 hours required for the data to be moved back to its original place. I emailed and voiced my apologies to the team present both in India and in France.
While I was not fired on the spot, it might have indirectly led to the disappearance of our devops (devoops?) team in Paris. Fortunately enough I guess I was the last team member to be sent away.
For my defense, our dev team had a intervention scope so big that they were essentially super-men once they have been granted access. One the rights granted to us was full write-access to all incoming data folders.