To be fair, the article states that the junior dev made one mistake, whereas the company made several. However, I agree with you in that it should emphasize that the mistake made by the junior dev is lesser in both size and quantity.
A old supervisor of mine told me that "You treat a new developer like a duckling near a pond with Northern Pike". There is no way you let them have access to production (or frankly, anything past a development environment) until they have been there a while and gotten the hang of it.
On a tangent, I was paranoid enough in one position to have all my production terminal windows have a red background with yellow text. That brought the proper, serious attitude towards my work.
That's not paranoid! That's a fantastic thing to do and one that I highly recommend for anyone. It's terribly easy to forget which environment you're working in when you get distracted for a moment and that's how mistakes happen.
I went the whole nine yards on that project. I had white (black font) for my own machine, green for development, blue for integration test, yellow for system test, and red for production.
I liked the green / white text theme the most with blue being my second favorite to write code in so those got the ones I would use the most. I was never a lover of the old amber terminals and red is just plain painful to write on. I guess the more pain your terminal causes you, the more dangerous the environment.
I did something similar with a shell script for MySQL databases, both to support shorthand invocation (i.e. "mysql foo-bar-baz" to look up the named configuration, expand it into whatever set of arguments is defined therein, and invoke mysql(1) accordingly), and to allow for detailed and environment-specific prompts which warn me when I'm targeting production.
It's hideous, of course. But it works very well for me, and I don't suppose I'd be averse, if interest becomes evident, to sharing it somewhere for others to benefit from and/or peruse in horrified fascination.
Outside GNU coreutils, the only dependencies are mysql(1) (of course) and jq(1), which is more or less "awk for JSON" and which I strongly recommend to anyone who regularly deals with that file format.
I cannot remember the terminal program, but I had a file that set the colors and logged me in to the environment then set a bunch of variables. I was on a NT then 2000 box. Might have been a version of teraterm.
Indeed. The difference between an experienced person a naive one is not that the experienced person doesn't make mistakes, but that they fully expect to make mistakes so they take precautions to mitigate their effects, take measures to prevent them, and minimize doing work that is mistake-prone.
Inexperienced folks really haven't had that "oh hell" moment of contemplating what could go wrong and the experience to know how bad it can go wrong.
I was at a site when a truck hit the power main and the UPS (diesel if I recall) for the server room failed to work. The new people were in the "It shouldn't take too long to reboot the servers" where the experienced people were "this is going to be a very long day". The later group was rewarded with a 36 hour day.
and get something a little more lively! (For many more things you might be able to do with "tput", refer to "man 5 terminfo" on your local Linux machine.)
I really like having a strong, visual indicator when you're doing something dangerous, as well.
I had a job where we would often sudo as other users and ssh into other boxes. I kept trying and trying to find a way to change the terminal when sudoed as someone else because occasionally I would execute something in the first terminal I could grab. At the time I couldn't find anything or think of a way.
Yep, fairly horrific. They will also attack a swimmer for no apparent reason other than they are an evil, foul-tempered fish. The bigger ones can make you bleed quite quickly. River barracuda is a name I heard once.
Indeed. To capitalize on a meme in current circulation, "One does not simply 'accidentally destroy' a production database."
When things like this happen, they happen because someone who almost certainly did, in fact, know better -- at some point made a conscious decision to allow them to happen, by consciously cutting corners in certain matters of due diligence that we all know about. Things like backup systems, database roles, and hiring the right people to set up and maintain these things.
It's really, really simple. Just restrict access to the production databases. Structure the access in such a way that this kind of honest mistake simply cannot happen. Add an extra hoop to jump through to access prod. Easy as that.
Better yet, prod should be in a different "cloud universe" (root account, etc) from dev.
> What would be the motivation behind fabricating a story like this?
Lulz, getting one over on the Internet, being able to fabricate a plausible story and see people all over the world reacting to it. Some people get off on weird stuff.
It doesn't make any sense that anyone would put a junior on his first day even close to a production environment, much less using a tool that can delete a database that btw you can't restore from a backup.
IMO this is either a false story, or this was done on purpose.
What if the company wanted to get rid of some data that would put the company in a difficult position in front of its shareholders or even the law? What if the lost data could prove that the CTO had made some other much bigger mistake that allowed hackers to get private information?
That still only applies to the second option. The first one (false story) is very much possible and it seems very plausible to me. It could be a bet, a sociological research or one of those "why not" ideas... we'll probably never know.
One of the first things the contractors at my current company used to do was to make new guy (even a junior) a sudoers enabled account on production (we had no in house development before I arrived, so they had access to everything).
Never ascribe to malice that which can be explained by incompetence.
I've worked at two profitable software companies where every process was manual, and everyone had access to everything. At one, the production servers were just sort of sitting in ... not quite a hallway, but a random open space near the front door. I don't know what's hard to believe about this. Some companies put all their focus/investment on selling a product.
I wish more comments would approach this from the angle of "if this is true..." because it seems pretty far out there honestly. It's like that `rm -rf` story from a few months back where someone claimed on a Stack Exchange site, that they erased all their company data and backups.
Fun story! A long time ago I interned for a very large catalog company. It was my first real internship in IT.
One of the first things (so a couple days in, not exactly day one) the CTO had me do was "map" their server room (e.g. figure out where data and power cables were going to and coming from, physically and virtually). While working my way through the back of a massive UPS rack, I stepped on one power cord that happened to traverse over another 10 or 15 cords. I did it in such a way that all of those cords were pulled ever so slightly from their UPS connections. I didn't even realize it until one of the staff IT guys came in the room a few minutes later and asked me what happened. Turns out those cords powered the 300+ Citrix thin clients in use by the national call center downstairs.
I thought for sure I was in huge trouble, but everything came back online fine, the staff IT guys poked some fun at me and the CTO helped turn it in to a valuable lesson. I learned a lot of great stuff during that interneship and to this day, 15 or so years later, I still think of that experience when doing risky work (or when I fuck it up).
The original story (junior dev was given access to production / cleared the database while trying to setup his dev env) was funny to me. Actually, I can't really believe that what was described is possible. For me, there are two explanations:
- It's a prank by the OP. He posted a story out of his mind for whatever reason.
- It was all manipulated by a different employee of the company. He (by accident) broke the production db and decided to blame the junior dev: Most junior devs will just believe you (out of fear and inexperience) when you say to them that they broke the production.
Unrelated to the story, but I've had a similar experience as to some of the stories described in the article - where a cascade of unseen problems landed on my lap and I took the blame even though little of it was my doing.
(Note, this was many years ago and mildly traumatic, so some details might be fuzzy)
In 2001 I was working for one of the larger internet server co-location companies in the US as a dedicated customer support engineer (remote hands) for several large clients.
One of my clients requested that I perform a memory upgrade on their Sun 420R servers at 4am (on my birthday none the less).
So I come in. Confirm with the client they are ready to be shut down, confirm with the NOC that the servers are coming down. The client remotely shut down the servers in question but they didn't power off (missed clue). One by one I power them down, roll them out of the rack, insert memory.
This was a pretty routine request, so it went smoothly - so I had thought.
Switch the console to the database server, boot. The kernel starts without issue, then a slew of error messages. Too many to read.
The Sun 420R's were known to have memory riser issues, so I power off reseat the riser, and all the memory. Same problem.
I take out the new memory. Same problem.
Before I could troubleshoot the client guesses it's a hardware issue, suggests we bring up the other systems and the cold-backup systems to compensate.
Nothing comes up.
Not
One
System
So at this point the client starts to panic over the phone. They hang up on me.
So I pull out my laptop and switch to a serial console on one of the servers in attempt to troubleshoot the error messages.
All the error messages are Bourne Shell (sh, not bash) errors.
Realizing this doesn't appear to be a hardware issue, I boot from a recovery cdrom, and start inspecting the init system. Everything looked great. So I tried to run a script, and then it became obvious - everything LOOKED too great.
All the init files had been reformatted. It said as much in a comment. A certain very well known perl developer (at the time) who developed the site for the client had found the formatting of the init scripts atrocious and unreadable, so took it upon himself to reformat every init script on every server. The problem is, bourne shell didn't like some of the formatting changes - mainly space between variable, equals sign, and value declarations. Had he tested his changes, it would have been obvious what the problem was.
Testing my theory I fixed the formatting of a single script, and it immediately works without error. Problem found, now I need to fix all the scripts on all servers.
I call the client. No answer.
I call the client again. Answer. "Did you fix our servers yet?" "No, I found..." phone disconnects.
I call again, no answer.
I call again, no answer.
I call the NOC, asking they try to inform the client of the status.
So I proceed to start fixing the first server. It took an hour, but the server came up. The second server came up a little faster, and I proceeded to work on the third and fourth.
Meanwhile, I find out the client had been calling around escalating to senior management at my employer petitioning that they fire me and put someone competent on the job.
I find this out as one of the analysts comes up from the NOC and tells me I'm relieved, and was asked to go home with no further information. So I explain what had happened and what I was doing to fix the issue.
The next day I am called into the office. Told I was fired for my obvious screwups, and that thankfully the NOC analyst had been there to save them from losing such a large client.
Happy birthday.
Without question, there were things I could have done better in that situation, but could not have predicted in a million years that someone would reformat init files without testing them.
Wow, incredible story. I used to be a Sun SSE, so I can feel for you. I know hindsight is 20/20, but this was my approach when replacing failed hardware like memory DIMMs remotely (when the client wasn't there in the datacenter):
- I always asked them to do a full shutdown,"init 5", and would wait until the server physically powered off before working. That would verify I was on the right server. If the server didn't power off because the customer did something wrong like "init 0" instead of "init 5", I would connect to the serial console and verify that it was sitting at an openboot prompt.
- When I ran into a software-related issue like "DB won't start," I would never touch anything or try to fix it. I know you had great intentions, but fixing the init scripts opens you up to a ton of potential hurt. As they say: "no good deed goes unpunished..."
I really don't want you to think I'm criticizing your actions - you were a true hero to the customer in a crazy situation. I think its truly unfortunate that they terminated you for this. Best of luck!
Good for you taking something positive from the experience, but man that must have hurt at the time! I know that at I at least was a lot less humble at the start of my career than I am now.
This is why, when you see gaping holes in production systems, security or otherwise, and those holes aren't your fault, your best course of action is to do nothing. If you explore it, you might trigger cascading effects and you'll be blamed for it. If you expose it, you'll embarrass someone potentially much more senior/powerful than you. If you do nothing, someone else falls in and you plead ignorance.
Once I was put in charge of running regularly some DB scripts for a social site (non profit, not a payed job) with about 10K active users. It was all manual as the nobody had the time and skill to automate it. There was no development site, there was only production so adding any automation was also risky. And of course one fateful Sunday I pasted a script that was not properly adjusted and the DB was fubar. It took us more than a day to recover - thankfully we had a recently current backup. Later we all spoke about it and also did a write-up. The debriefing helped a lot processing the shock but for a long time I loathed to log into the DB.
When I'm fully in control I engineer things in a way that make disasters unlikely and easily recoverable. But that is not always possible. I tend to avoid touching any production DB but again that is not always possible.
I always believed I have a developer mindset more than an operational mindset and worked more in roles compatible with the former than the larger. When looking at the devops culture I always wonder whether the lessons of strict control of production environments are getting enough attention. There is value is closing the gap between the two world but please leave a gap.
I am surprised they even have their prod data cloned into test servers: Get to any reasonable size and this becomes a huge no-no. There were major measures in MS/Amazon (where I spent a bunch of time) and now at snap to ensure prod data isn't even visible without a bunch of hoops to jump through.
61 comments
[ 4.4 ms ] story [ 142 ms ] threadAs a first-day developer, he shouldn't really have had the ability to make a mistake with any consequences beyond lost time.
On a tangent, I was paranoid enough in one position to have all my production terminal windows have a red background with yellow text. That brought the proper, serious attitude towards my work.
I liked the green / white text theme the most with blue being my second favorite to write code in so those got the ones I would use the most. I was never a lover of the old amber terminals and red is just plain painful to write on. I guess the more pain your terminal causes you, the more dangerous the environment.
It's hideous, of course. But it works very well for me, and I don't suppose I'd be averse, if interest becomes evident, to sharing it somewhere for others to benefit from and/or peruse in horrified fascination.
https://gist.github.com/anonymous/8ec128e88dac85d93ed58fb518...
Outside GNU coreutils, the only dependencies are mysql(1) (of course) and jq(1), which is more or less "awk for JSON" and which I strongly recommend to anyone who regularly deals with that file format.
Share and Enjoy!
https://iterm2.com/documentation-escape-codes.html
[0] https://biology.stackexchange.com/questions/3871/which-shade...
I was at a site when a truck hit the power main and the UPS (diesel if I recall) for the server room failed to work. The new people were in the "It shouldn't take too long to reboot the servers" where the experienced people were "this is going to be a very long day". The later group was rewarded with a 36 hour day.
I had a job where we would often sudo as other users and ssh into other boxes. I kept trying and trying to find a way to change the terminal when sudoed as someone else because occasionally I would execute something in the first terminal I could grab. At the time I couldn't find anything or think of a way.
https://www.youtube.com/watch?v=2_E4BUt1z8Y
Indeed. To capitalize on a meme in current circulation, "One does not simply 'accidentally destroy' a production database."
When things like this happen, they happen because someone who almost certainly did, in fact, know better -- at some point made a conscious decision to allow them to happen, by consciously cutting corners in certain matters of due diligence that we all know about. Things like backup systems, database roles, and hiring the right people to set up and maintain these things.
Not because of something a junior dev did.
Better yet, prod should be in a different "cloud universe" (root account, etc) from dev.
> The company/CTO made the huge mistakes.
Absolutely.
It's still amazing how badly that company is being managed from a technical perspective.
I see posts of this nature all the time on r/csquestions. This one just happened to go viral.
Lulz, getting one over on the Internet, being able to fabricate a plausible story and see people all over the world reacting to it. Some people get off on weird stuff.
See -- Programmer Automates His Job Away for 6 years and then forgets how to program.
I don't think any individual aspect of the story is unbelievable, but when taken as a whole it stretches credulity.
Silicon Valley and Veep are both disturbingly prescient.
A rule of thumb is that the really ridiculous stuff in Silicon Valley like the "three commas" or the "blood boy", actually happened.
https://news.ycombinator.com/item?id=14476421
It doesn't make any sense that anyone would put a junior on his first day even close to a production environment, much less using a tool that can delete a database that btw you can't restore from a backup.
IMO this is either a false story, or this was done on purpose.
What if the company wanted to get rid of some data that would put the company in a difficult position in front of its shareholders or even the law? What if the lost data could prove that the CTO had made some other much bigger mistake that allowed hackers to get private information?
We will never know of course.
Never ascribe to malice that which can be explained by incompetence.
One of the first things (so a couple days in, not exactly day one) the CTO had me do was "map" their server room (e.g. figure out where data and power cables were going to and coming from, physically and virtually). While working my way through the back of a massive UPS rack, I stepped on one power cord that happened to traverse over another 10 or 15 cords. I did it in such a way that all of those cords were pulled ever so slightly from their UPS connections. I didn't even realize it until one of the staff IT guys came in the room a few minutes later and asked me what happened. Turns out those cords powered the 300+ Citrix thin clients in use by the national call center downstairs.
I thought for sure I was in huge trouble, but everything came back online fine, the staff IT guys poked some fun at me and the CTO helped turn it in to a valuable lesson. I learned a lot of great stuff during that interneship and to this day, 15 or so years later, I still think of that experience when doing risky work (or when I fuck it up).
- It's a prank by the OP. He posted a story out of his mind for whatever reason.
- It was all manipulated by a different employee of the company. He (by accident) broke the production db and decided to blame the junior dev: Most junior devs will just believe you (out of fear and inexperience) when you say to them that they broke the production.
The 2nd one seems more possible to me.
(Note, this was many years ago and mildly traumatic, so some details might be fuzzy)
In 2001 I was working for one of the larger internet server co-location companies in the US as a dedicated customer support engineer (remote hands) for several large clients.
One of my clients requested that I perform a memory upgrade on their Sun 420R servers at 4am (on my birthday none the less).
So I come in. Confirm with the client they are ready to be shut down, confirm with the NOC that the servers are coming down. The client remotely shut down the servers in question but they didn't power off (missed clue). One by one I power them down, roll them out of the rack, insert memory.
This was a pretty routine request, so it went smoothly - so I had thought.
Switch the console to the database server, boot. The kernel starts without issue, then a slew of error messages. Too many to read.
The Sun 420R's were known to have memory riser issues, so I power off reseat the riser, and all the memory. Same problem.
I take out the new memory. Same problem.
Before I could troubleshoot the client guesses it's a hardware issue, suggests we bring up the other systems and the cold-backup systems to compensate.
Nothing comes up.
Not
One
System
So at this point the client starts to panic over the phone. They hang up on me.
So I pull out my laptop and switch to a serial console on one of the servers in attempt to troubleshoot the error messages.
All the error messages are Bourne Shell (sh, not bash) errors.
Realizing this doesn't appear to be a hardware issue, I boot from a recovery cdrom, and start inspecting the init system. Everything looked great. So I tried to run a script, and then it became obvious - everything LOOKED too great.
All the init files had been reformatted. It said as much in a comment. A certain very well known perl developer (at the time) who developed the site for the client had found the formatting of the init scripts atrocious and unreadable, so took it upon himself to reformat every init script on every server. The problem is, bourne shell didn't like some of the formatting changes - mainly space between variable, equals sign, and value declarations. Had he tested his changes, it would have been obvious what the problem was.
Testing my theory I fixed the formatting of a single script, and it immediately works without error. Problem found, now I need to fix all the scripts on all servers.
I call the client. No answer.
I call the client again. Answer. "Did you fix our servers yet?" "No, I found..." phone disconnects.
I call again, no answer.
I call again, no answer.
I call the NOC, asking they try to inform the client of the status.
So I proceed to start fixing the first server. It took an hour, but the server came up. The second server came up a little faster, and I proceeded to work on the third and fourth.
Meanwhile, I find out the client had been calling around escalating to senior management at my employer petitioning that they fire me and put someone competent on the job.
I find this out as one of the analysts comes up from the NOC and tells me I'm relieved, and was asked to go home with no further information. So I explain what had happened and what I was doing to fix the issue.
The next day I am called into the office. Told I was fired for my obvious screwups, and that thankfully the NOC analyst had been there to save them from losing such a large client.
Happy birthday.
Without question, there were things I could have done better in that situation, but could not have predicted in a million years that someone would reformat init files without testing them.
I really don't want you to think I'm criticizing your actions - you were a true hero to the customer in a crazy situation. I think its truly unfortunate that they terminated you for this. Best of luck!
When I'm fully in control I engineer things in a way that make disasters unlikely and easily recoverable. But that is not always possible. I tend to avoid touching any production DB but again that is not always possible.
I always believed I have a developer mindset more than an operational mindset and worked more in roles compatible with the former than the larger. When looking at the devops culture I always wonder whether the lessons of strict control of production environments are getting enough attention. There is value is closing the gap between the two world but please leave a gap.
https://www.youtube.com/watch?v=GwQW3KW3DCc