Literally every engineer who didn't think to have a dev database can also be responsible. This is just a case of 'shit happens' since nobody had the sense to not work on, not back up, the live db.
More responsible, I would say. You expect a junior to make mistakes; the company should be structured to handle that happening.
Though I would look askance at whoever hired a philosophy grad as well, to be perfectly honest. The author admits he didn't have the experience to spot bad practice at the time.
You don't expect a junior to make mistakes, you expect everyone AND everything to make mistakes/fail. Backups and separated environments is the least you could expect from a company earning millions.
Actually even senior developers or architects make mistakes. Philosophy grad or not, it doesn't matter. That's to be expected.
What's more questionable is:
* Developers have access to the production database from their machine, while it should only be accessible to the front machines within the datacenter.
* Junior developers don't need an access to production machine, only sysops and maybe the technical PM.
* No backup of the production database. WTF???
If they had a hardware failure they would have been in the same shit.
Well - to be fair - if the company's practices include developing on the production database and not doing daily/weekly backups, then hiring an inexperienced Philosophy major is the least of their problems.
Also, someone who actually had the development experience and knowledge of better-practice would not have taken that position.
This. Times 1000. I would even go as far as to fire the CTO because if the data is your bread and butter, the you protect the data.
Protecting the data is called a disaster recovery (DR) plan in those big outdated companies that people like to make fun of.
The reason that these companies have a DR plan is to tell the CEO that 'when' (not 'if') all of the data goes away, (a) how long will it take to get it back and (b) how out-of-sync will this data be (2 minutes from freshness? 6 hours?).
Whoever was your boss should have taken responsibility. Someone gave you access to the production database instead of setting up a proper development and testing environment. For a company doing "millions" in revenues, it's odd that they wouldn't think of getting someone with a tiny bit of experience to manage the development team.
We sell middleware to a number of customers with millions of dollars in revenue who don't have backups, don't have testbeds for rolling out to "dev" before pushing to "prod" and don't have someone with any expertise in managing their IT / infrastructure needs.
My experience is that this is the norm, not the exception.
As a college student, this fucking horrifies me. Is there anyway I can guarantee I don't end up at someplace as unprofessional as this? I want to learn my first job not teach/lead.
An interview is a two way street. You too can ask questions, make sure you fit within the team, and that the job is to your liking. That is the time to ask and find out about anything that may be pertinent to your job.
Also, you have a choice of leaving if you don't like the job, and or don't find the practices in place to be any good, or you can fix them.
Ask about their development and staging process during the technical interview. Ask about how someone gets a piece of code into production (listen for mention of different environments).
It doesn't cover every last thing, but a team following these practices is the kind of team you're looking for. Ask these questions at your interviews.
The interview advice here is excellent. Ask questions - in the current climate they're hunting you, not the other way around.
Additionally, start networking now. Get to know ace developers in your area, and you will start hearing about top-level development shops. Go to meetups and other events where strong developers are likely to gather (or really, developers who give a shit about proper engineering) and meet people there.
It's next to impossible to know, walking into an office building, whether the company is a fucked up joke or good at what it does - people will tell you.
Well you should certainly look for a team that gives you a good impression with their dev process.
But, things are not always perfect, even on great teams. Not saying its normal to destroy your production database! But even in good shops it's a constant challenge to stay organized and do great work. Look for a team that is at least trying to do great work, rather than a complacent team.
Wow, that's quite a story.
If your company is ever 1 button press away from destruction. Know that this will eventually happen.
I'm quite surprised stuff like this hadn't happened earlier. When I am doing development with a database I will quite often trash the data when writing code.
Agreed, but if the manager took responsibility for this he or she probably would be fired. Still, it is the only way to be; otherwise, you're not a real leader.
Wow, I'm sorry you had to experience that. I'm sure it was traumatic -- or perhaps you took it better than I would have. It must be of some comfort to look back now and realize that you only bore a small part of the blame, and that ultimately a large potion of the responsibility lies on the shoulders of whomever set up the dev environment like that, as well as whomever cancelled backups.
This is certainly a monumental fuckup, but these things inevitably happen even with better development practices, this is why you need backups, preferably daily, and as much separation of concerns and responsibilities as humanly possible.
Anecdote:
I am working for a company that does some data analysis for marketers aggregated from a vast number of sources. There was a giant legacy MyISAM (this becomes important later) table with lots of imported data. One day, I made some trivial looking migration (added a flag column to that table). I tested it locally, rolled it out to staging server. Everything seemed A-OK until we started migration on the production server. Suddenly, everything broke. By everything, I mean EVERYTHING, our web application showed massive 500-s, total DEFCON1 across the whole company. It turned out we ran out of disk space, since apparently myisam tables are altered the following way: first the table is created with updated schema, then it is populated with data from the old table. MyISAM ran out of disk space and somehow corrupted the existing tables, mysql server would start with blank tables, with all data lost.
I can confirm this very feeling: "The implications of what I'd just done didn't immediately hit me. I first had a truly out-of-body experience, seeming to hover above the darkened room of hackers, each hunched over glowing terminals." Also, I distinctly remember how I shivered and my hands shook. It felt like my body temperature fell by several degrees.
Fortunately for me, there was a daily backup routine in place. Still, several hour long outage and lots of apologies to angry clients.
"There are two types of people in this world, those who have lost data, and those who are going to lose data"
More than once (over the last few years) I've been doing some important update. I tend to do it the same way.
START TRANSACTION;
--run SQL--
--check results--
COMMIT; or ABORT TRANSACTION;
Of course, if you happen to run into the 1 or 2 MyISAM tables that no one knew were MyISAM, abort doesn't do anything. You've screwed up the data and need a backup.
So you always have to make a backup and check that the tables are defined they way they should be. Nothing is quite as much fun as the first time you delete a bunch of critical data off a production system. Luckily the table was small, basically static, and we had backups so it was only a problem for ~5 minutes.
Reading those stories makes me realize how well thought-out the process at my work is:
We have dev databases (one of which was recently empty, nobody knows why; but that's another matter), then a staging environment, and finally production. And the database in the staging environment runs on a weaker machine than the prod database. So before any schema change goes into production, we do a time measurement in the staging environment to have a rough upper bound for how long it will take, how much disc space it uses etc.
And we have a monthly sync from prod to staging, so the staging db isn't much smaller than prod db.
And the small team of developers occasionally decides to do a restore of the prod db in the development environment.
The downside is that we can't easily keep sensitive production data to find its way into the development environment.
When moving data from prod to other environments, consider a scrambler. E.g., replace all customer names with names generated from census data.
I try to keep data having the same form (e.g., length, number of records, similar relationships, looks like production data). But it's random enough so that if the data ever leaks, we don't have to apologize to everybody.
Since your handle is perlgeek, you're already well equipped to do a streaming transformation of your SQL dump. :)
Yep. For x.com I wrote a simple cron job that sterilizes the automated database dump and sends it to the dev server. Roughly, it's like this:
-cp the dump to a new working copy
-sed out cache and tmp tables
-Replace all personal user data with placeholders. This part can be tricky, because you have to find everywhere this lives (are form submissions stored and do they have PII?)
-Some more sed to deal with actions/triggers that are linked to production's db user specifically.
-Finally, scp the sanitized dump to the dev server, where it awaits a Jenkins job to import the new dump.
The cron job happens on the production DB server itself overnight (keeping the PII exposure at the same level it is already), so we don't even have to think about it. We've got a working, sanitized database dump ready and waiting every morning, and a fresh prod-like environment built for us when we log on. It's a beautiful thing.
Remebers me of adifferent story from back the day when I worked around a pretty huge SAP system (as some kind of super user whatever). In one, seemingly trivial update (trivial compared to the complete system upgrade from one version to the higher one that worked without problems) cleansed the database including all purchase orders from the last two days company wide. Ah, and the back-up becamo "unusable", too.
But as far as I know, nobody was fired for this. Because, yes, things like this just can happen. An eventually it got fixed anyway.
I still remember the all-consuming dread I felt as an intern when I ran an UPDATE and forgot the WHERE clause. I consider it part of the rite of passage in the web developer world. Kind of like using an image or text in a dev environment that you never expect a client to see.
Luckily the company I was at (like any rational company) backed up their db and worked in different environments, so it was more of a thing my coworkers teased me for than an apocalyptic event.
I know how you felt. Many years ago when I was a junior working in a casual game company, I were to add a bunch of credits to a poker player (fake money). I forget the where in the SQL clause and added credits to every player in our database. Lucky me it was an add and not a set and I could revert it. Another time I was going to shutdown my pc (a debian box) using "shutdown -h now" and totally forgot that I was in a ssh session to our main game server. I had to call the tech support overseas and tell him to physically turn on the server...
At one job I went with this scheme for terminal background color: green screen for development, blue for testing, yellow for stage / system test, and red for production. This saved a lot of problems because I knew to be very careful when typing in the red.
I have done this on a few servers but found that it always screws up formatting of the lines in bash when they are long and you are hitting up and going back through the history.
Did you change the $PS1 variable? Can you share your config?
Interessting, I have been the only guy in my team who had the exact opposite colours. Green was production and red was testing (i didn't do any dev work, so). I guess that came from me being from produciton. But I should have payed more attention when working with other people on my machine, in hindsight... Luckily, nothing bad ever happened!
I liked a green screen (too much time spent with old terminals) and blue was ok to type on but not as nice . I went with the yellow (warning) and red (serious warning) because they are not very comfortable to type on and most people get the "alert" status given Star Trek.
molly-guard -- looks great. I've always aliased my commands so if I accidentally type 'po' while in an ssh session on a production server then it complains. On my local dev box, 'po' is just quicker for me to power off the box. It has saved me at least once.
Turning off the wrong-server is a thing that bit me before I installed molly-guard. These days that, and similar, is a tweak I apply to all hosts I control.
(molly-guard makes you type in the hostname before a halt/shutdown/reboot command.)
Steve Kemp? Slightly unrelated, but i just want to say i'm (still) a big fan of your old 2004 program window.exe. Very handy for unhiding the odd broken program.
Have a wonderful day! (and i'll definitely look at installing molly-guard on my production debian servers)
One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired."
Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about.
And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.
One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired."
I asked similar, and agree that it's a really useful question.
I think it's an especially great one for startups, as successful candidates are more likely to come into contact with production systems.
For these positions you not only want people capable of recovering accidents, but also people who have screwed up because, conversely, they've been trusted not to screw up. Those who've never been trusted enough to not damage a system are unlikely to be of much use.
This. We all mess up, but only the best ones will deal with it professionally and learn from it. Sounds like the OP is in that group. He didn't try to hide it, blame it on anyone else, or make excuses. He just did what he could to fix his mistake.
When people say "making mistakes is unacceptable - imagine if doctors made mistakes" they ignore three facts:
1. Doctors do make mistakes. Lots of them. All the time.
2. Even an average doctor is paid an awful lot more than me.
3. Doctors have other people analysing where things can go wrong, and recommending fixes.
If you want fewer development mistakes, as a company you have to accept it will cost money and take more time. It's for a manager to decide where the optimal tradeoff exists.
> If you want fewer development mistakes, as a company you have to accept it will cost money and take more time. It's for a manager to decide where the optimal tradeoff exists.
This is absolutely it, of course it is possible to become so risk averse that you never actually succeed in getting anything done and there are certainly organisations that suffer from that (usually larger ones).
However some people seem to take the view that it is impossible to protect oneself from all risks therefor it is pointless protecting from any of them.
The good news is that usually protecting against risks tends to get exponentially more expensive as you add "nines" therefor having a 99% guarantee against data loss is a lot cheaper than a 99.999% guarantee.
Having a cronjob that does a mysqldump of an entire database, emails some administrator and then does rsync to some other location (even just a dropbox folder) is something that is probably only a couple of hours work.
The aviation industry is the same. Most aviation authorities around the world have a no-fault reporting system so that fixes can get implemented without pilots worrying about losing their job.
Part of our culture/personality/team fit questions is like this. We also have one that's something like "tell me about a time you failed a commitment (deadline, etc) and how you handled it". Numerous people with over a decade claim perfect records, frequently blaming all those around them as having failed. It's been a real easy way to eliminate candidates, particularly because almost every team member we have gave an example that was within like a month of when they interviewed.
Interesting, though perhaps they misinterpreted the question. Some companies seem to have a culture where you never admit failure and perhaps they assumed that was the answer you were looking for?
All of the failed responses weren't just not admitting failure, but directly blaming others. One example was a team lead in charge of a project where they said the developers missed their deadlines and would "whine" about not having enough details to do their job. Someone who didn't have an example of failing to deliver, but had examples of near misses they were able to save would even be good examples. Playing the blame game, though, just doesn't fit.
In my first year as an engineer, I shut off the production application server. It was an internally-developed app server built in & for an internally-developed language, with its own internally-developed persistence mechanism, and it was my first day working with it. I had several terminals open, and in one I'd logged on to production to check how the data was formatted, so I could get something comparable for my dev environment. Unfortunately I forgot to log off, and at some point I issued the `stop server` command, thinking I was doing it on dev. A few minutes later the project leader, one of the most senior engineers, came over and started yelling at me. Fortunately not much was lost (other than a few users' sessions--not a high-traffic site). But I was really appalled at my mistake and I certainly wondered if I was about to be fired. Since then I've taken steps to (1) keep myself off production as much as possible (2) make a production terminal visually distinct from a non-production one. As a sometime project leader myself now, I also make sure newcomers can easily obtain an adequate development environment, and I don't give out the production login to every junior dev. :-)
Wow, that's terrible. Mistakes happen, and for the notion of 'blame' to surface requires some monumentally incompetent management... the exact kind that would have their junior programmers developing against an un-backed-up production database.
The immediate takeaway from a disaster should always be 'How can we make sure this doesn't happen again?' not 'Man, I can't believe Fred did that, what an idiot.'
If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?"
If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?"
They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are.
As an aside: Way back in time I caused about ten thousand companies to have to refile some pretty important government documents because I was doubling xml decoding (& became &). My boss actually laughed and was like "we should have caught this a long time ago"... by we he actually meant himself and support.
>If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?"
This is a question that the person in charge of backups needs to think about, too. I mean, rephrase it as "Is there any one person who can write to both production and backup copies of critical data?" but it means the same thing as what you said.
(and if the CTO, or whoever is in charge of backups screws up this question? the 'perfect storm' means "all your data is gone" - dono about you, but my plan for that involves bankruptcy court, and a whole lot of personal shame. Someone coming in and stealing all the hardware? not nearly as big of a deal, as long as I've still got the data. My own 'backup' house is not in order, well, for lots of reasons, mostly having to do with performance, so I live with this low-level fear every day.)
Seriously, think, for a moment. There's at least one kid with root on production /and/ access to the backups, right? At most small companies, that is all your 'root-level' sysadmins.
That's bad. What if his (or her) account credentials get compromised? (or what if they go rogue? it happens. Not often, and usually when it does it's "But this is really best for the company" It's pretty rare that a SysAdmin actively and directly attempts to destroy a company.)
(SysAdmins going fully rogue is pretty rare, but I think it's still a good thought experiment. If there is no way for the user to destroy something when they are actively hostile, you /know/ they can't destroy it by accident. It's the only way to be sure.)
The point of backups, primarily, is to cover your ass when someone screws up, primarily. (RAID, on the other hand, is primarily to cover your ass when hardware fails) - RAID is not Backup and Backup is not RAID. You need to keep this in mind when designing your backup, and when designing your RAID.
(Yes, backup is also nice when the hardware failure gets so bad that RAID can't save you; but you know what? that's pretty goddamn rare, compared to 'someone fucked up.')
I mean, the worst case backup system would be a system that remotely writes all local data off site, without keeping snapshots or some way of reverting. That's not a backup at all; that's a RAID.
The best case backup is some sort of remote backup where you physically can't overwrite the goddamn thing for X days. Traditionally, this is done with off-site tape. I (or rather, your junior sysadmin monkey) writes the backup to tape, then tests the tape, then gives the tape to the iron mountain truck to stick in a safe. (if your company has money; if not, the safe is under the owner's bed.)
I think that with modern snapshots, it would be interesting to create a 'cloud backup' service where you have a 'do not allow overwrite before date X' parameter, and it wouldn't be that hard to implement, but I don't know of anyone that does it. The hard part about doing it in house is that the person who managed the backup server couldn't have root on production and vis-a-vis, or you defeat the point, so this is one case where outsourcing is very likely to be better than anything you could do yourself.
> If there is no way for the user to destroy something when they are actively hostile, you /know/ they can't destroy it by accident.
Which also means they can't fix something in case of a catastrophic event. "Recover a file deleted from ext3? Fix a borked NTFS partition? Salvage a crashed MySQL table? Sorry boss, no can do - my admin powers have been neutered so that I don't break something 'by accident, wink wink nudge nudge'." This is, ultimately, an issue of trust, not of artificial technical limitations.
> one case where outsourcing is very likely to be better than anything you could do yourself.
Hm. Your idea that "cloud is actually pixie dust magically solving all problems" seems to fail your very own test. Is there a way to prevent the outsourced admins from, um, destroying something when they are actively hostile? Nope, you've only added a layer of indirection.
>> If there is no way for the user to destroy something when they are actively hostile, you /know/ they can't destroy it by accident.
>Which also means they can't fix something in case of a catastrophic event. "Recover a file deleted from ext3? Fix a borked NTFS partition? Salvage a crashed MySQL table? Sorry boss, no can do - my admin powers have been neutered so that I don't break something 'by accident, wink wink nudge nudge'." This is, ultimately, an issue of trust, not of artificial technical limitations.
All of the problems you describe can be solved by spare hardware and read only access to the backups. I mean, your SysAdmin needs control over the production environment, right? to do his or her job. but a sysadmin can function just fine without being able to overwrite backups. (assuming there is someone else around to admin the backup server.)
fixing my spelling now.
Yes, it's about trust. but anyone who demands absolute trust is, well, at the very least an overconfident asshole. I mean, in a properly designed backup system (and I don't have anything at all like this at the moment) I would not have write-access to the backups, and I'm majority shareholder and lead sysadmin.
That's what I'm saying... backups are primarily there when someone screwed it up... in other words, when someone was trusted (or trusted themselves) too much.
Okay, now I think I understand you, and it seems we're actually in agreement - there is still absolute power, but it's not all concentrated in one user :)
>Hm. Your idea that "cloud is actually pixie dust magically solving all problems" seems to fail your very own test. Is there a way to prevent the outsourced admins from, um, destroying something when they are actively hostile? Nope, you've only added a layer of indirection.
the idea here is to make sure that the people with write-access to production don't have write-access to the backups and vis-a-vis. The point is that now two people have to screw it up before I lose data.
Outsourcing has it's place. You are an idiot if you outsource production and backups to the same people, though. This is why I think "the cloud" is a bad way of thinking about it. Linode and rackspace are completely different companies... one of them screwing it up is not going to effect the other.
I think that with modern snapshots, it would be interesting to create a 'cloud backup' service where you have a 'do not allow overwrite before date X' parameter, and it wouldn't be that hard to implement, but I don't know of anyone that does it.
>> I think that with modern snapshots, it would be interesting to create a 'cloud backup' service where you have a 'do not allow overwrite before date X' parameter, and it wouldn't be that hard to implement, but I don't know of anyone that does it.
I test backups for F500 companies on a daily basis (IT Risk Consulting) - this would be missing the point really, the business process around this problem is really moving towards live mirrored replication. This allows much faster recall time, and also mitigates many risks with the conventional 'snapshot' method through either tapes, cloud, etc.
I've said it elsewhere, but it bears repeating: RAID is about availability first and foremost. The fact that it happens to preserve your data in the case of one form of hardware failure is a side effect of its primary goal.
This is so true. I had to leave my first junior dev position for similar reasons as the OP, though nothing as monumental.
I was handed a legacy codebase with zero tests. I left a few small bugs in production, and got absolutely chewed out for it. It was never an issue with our processes, it was obviously an issue with the guy they hired who had 1 intro CS class and 1 rails hobby project on his resume. The lead dev never really cared that we didn't have tests, or a careful deploy process. He just got angry when things went wrong. And even gave one more dev with even less experience than I had access to the code base.
It was a mess and the only thing I was "learning" was "don't touch any code you don't have to because who knows what you might break" which is obviously a terrible way to learn as a junior dev (forget "move fast and break things" we "move slowly and work in fear!"). So I quit and moved on, it was one of the better decisions I've ever made.
As a programmer I consider myself very lucky that one of the first advices I got when I was a junior was from one of my senior colleagues (and a very smart guy): "one of the most valuable qualities of a good programmer is courage".
Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger team-mates, a recent hire, "don't be afraid to break things!"
This is correct and is the way it should be. So how come the programmers are always politically gunning for the keys to the production server cabinet, where you do have to be afraid to break things?
Are they? Where I work (a bank) we were more than happy to move to read-only access to prod servers, and pass by a support team when we need to deploy things.
Heh. Another way I would say that is that "discretion is the better part of valor" And, to rip-off Hitchhiker's Guide to the Galaxy, "cowardice is the better part of discretion"...in that if fear makes you judiciously check your backups and write tests, then that's not a bad thing at all.
Paranoia is often useful, but with a good enviorment, and tools it's rarely need. I find bad assumptions often cause the worst problems. Break things regularly and you end up with fewer assumptions about the code base / production environment which is a vary good thing.
This is the right thing to encourage but I just would like to add always have a backup.
"Don't be afraid to break things as long as you have a backup".
It might be a simple version of the previous code, database copy or even the entire application. Do not forget to backup. If everything fails, we can quickly restore the previous working version.
Every production deployment should involve blowing away the prior instance, rebuilding from scratch, and restarting the service; you are effectively doing a near-full "restore" for every deployment, which forces you to have everything fully backed up and accessible...
Any failure to maintain good business continuity practices will manifest early for a product / employee / team, which allows you to prevent larger failures...
Spoken like a man who has maintained applications but never databases.
In the world where data needs to be maintained, this is not necessarily an option. In the bank where I work, we deploy new code without taking any outage (provide a new set of stored procedures in the database, then deploy a second set of middleware, then a new set of front-end servers, test it, then begin starting new user sessions on the new system; when all old user sessions have completed the old version can be turned off). Taking down the database AT ALL would require user outages. Restoring the database from backup is VERY HARD and would take hours (LOTS of hours).
That being said, we do NOT test our disaster-recovery and restore procedures well enough.
slowly and work in fear businesses usually collapse. the technical problem eventually becomes a business problem and no one realizes it can be solved at the source level.
it first builds inefficieny at the technical level then at the business level, finally it causes issues at the cultural level, and thats when the smart people start leaving.
One day Intel's yields suddenly went to hell (that's the ratio of working die on a wafer to non-working, and is a key to profitability). And no matter how hard they tried, they could only narrow it down to the wafers being contaminated, but the wafer supplier swore up and down they were shipping good stuff, and they were. So eventually they tasked a guy to follow packages from the supplier all the way to the fab lines, and he found the problem in Intel's receiving department. Where a clerk was breaking open the sealed packages and counting out the wafers on his desk to make damned sure Intel was getting its money worth....
His point is that you can have a Fortune 500 company, normally thought to be stable companies that won't go "poof" without ample warning, in which there are many more people than in previous kinds of companies who can very quickly kill it dead.
I physically cringed at that. Even the mail clerk should have noticed there was a 'big deal' about clean rooms and had some idea what the company he worked for did...
We are all born naked, bloody, and screaming; the only thing we know is how to work a nipple. Everything else has to be learned.
One of Toyota's mantras is "If the student has failed to learn, the teacher has failed to teach." Their point is that managers are responsible for solving issues that come from employee ignorance, not line workers.
Exactly. In my organization I know of perfectly good hardware that is either being tossed or used for non-critical applications because someone didn't follow the Incoming Inspection process correctly. It doesn't matter that they could simply be inspected now and found to be perfect, the process wasn't followed, so the product is "junk."
We may be quibbling over definitions. But do you believe that no mammal has an instinctual understanding of using nipples? Or that humans are unique among mammals in that regard?
I'm afraid I have no idea about the answer to either of your questions. I assumed the "we" in your original post meant "we humans".
I do understand that humans don't have an instinctual understanding of using nipples, human babies have a physical sucking reflex that kicks in when you put anything near their mouths. They usually quickly learn that sucking a nipple in a particular way gives out lots of yummy milk.
Where a clerk was breaking open the sealed packages and counting out the wafers on his desk to make damned sure Intel was getting its money worth....
I find this hard to believe. At some point a person in a space suit was introducing them into a clean room; she should have noticed that the packages were not sealed.
Maybe the clerk was carefully sealing them back up again? In retrospect they probably should have had tamper-proof seals given the value attached to the wafers being delivered unopened, but then most problems are easily avoidable in retrospect.
Exactly; this happened early enough in the history of ICs that I'm sure they weren't taking such precautions, just like tamper revealing seals for over the counter drugs didn't become big until after the Chicago Tylenol murders.
There's a good chance a tamper revealing seal would have stopped the clerk from opening the containers, and of course it they'd been broken before reaching the people at the fab lines who were supposed to open them that would have clued Intel into the problem before any went into production and would have allowed them to quickly trace the problem back upstream.
Or in this case energy, ignorance, and not learning enough of the big picture, or wondering why these wafers were sealed in air tight containers in the first place.
It's the well meaning mistakes that tend to be the most dangerous since most people are of good will, or at least don't want to kill the company and lose their jobs.
The difference between working with bad bosses and good bosses really shows itself when there's a disaster going on.
A mere few months into my current job, I ran an SQL query that updated some rows without properly checking one of the subqueries. Long story short - I fucked up an entire attribute table on a production machine (client insisted they didn't need a dev clone). I literally just broke down and started panicking, sure that I'd just lost my new job and wouldn't even have a reference to run on for a new one.
After a few minutes of me freaking out, my boss just says: "You fucked up. I've fucked up before, probably worse than this. Let's go fix it." And that was it. We fixed it (made it even better than it was, as we fixed the issue I was working on in the process), and instead of feeling like an outcast who didn't belong, I learned something and became a better dev for it.
I really wish this was at the top. Everyone will fuck up at some point (even your best engineer). Whether you learn anything from fucking up really determines how bad your mistake was.
There are several reasons why you should not feel guilty. The company was asking for trouble and you just happen to be the trigger. These are the three top things that could prevented that incident.
1) A cron job for the manually task you were doing.
2) Not working directly on production.
3) Having daily backups
And this could happened to anybody. After midnight any of us are at junior level and very prone to do this kind of mistakes.
I used to be a web freelance web developer/tech guy with one client, a designer. What made me quit was an incident where his client's Wordpress site hadn't been moved properly to the new hosting. (not by me)
The DB needed to be searched and replaced to remove all the old urls. After doing so, the wp_options cell on the production site holding much of the customizations kicked back to the defaults for the theme, the serialized data format being used was sensitive to brute DB-level changes.
I had talked to my client before about putting together a decent process including dev databases, scheduled backups, everything needed to prevent just such a screwup, but he waffled. Then blamed me when things went wrong.
I'd had enough and told him to do his own tech work, leaving him to fix his client's website himself. Being that I didn't build it, I didn't know which settings to flip back. I left freelance work and never looked back.
People and companies do this all the time, refuse to spend the time and money ensuring their systems won't break when you need them the most, then scapegoat the poor little technician when it does.
I'd like to say the answer is "don't work in such environments," but there's really no saying that it won't be this way at the next job you work, either.
I certainly wouldn't internalize any guilt being handed down, ultimately it's the founders' jobs to make sure that the proper systems are in place, after all, they have much more on the line than you do. Count it a blessing that you can just walk away and find another job.
I really feel sorry for this guy. Accidents happen, which is why development happens in a sandboxed copy of the live system and why back ups are essential. It simple shouldn't be possible (or at least, that easy) for human error to put an entire company in jeopardy.
Take my own company, I've accidentally deleted /dev on development servers (not that major of an issue thanks to udev, but the timing of the mistake was lousy), a co-worker recently dropped a critical table on dev database and we've had other engineers break Solaris by carelessly punching in chmod -R / as root (we've since revised engineers permissions so this is no longer possible). As much as those errors are stupid and as much as engineers of our calibre should know better, it can only takes a minor lack of concentration at the wrong moment to make a major fsck up. Which is doubly scary when you consider how many interruptions the average engineer gets a day.
So I think the real guilt belongs to the entire technical staff as this is a cascade of minor fcsk ups that lead to something catastrophic.
Bold on your part to own up and offer a resignation. (he "higher ups" should have recognized that and not accepted it). From the movie, "Social Network" http://www.youtube.com/watch?v=6bahX2rrT1I
This could've happened anyone. It's a huge shame for those in charge not for you. Any business letting such operations happen without having backups or proper user-right-management should consider why they still exists, if they really make huge amounts of many as you mentioned.
I would love to see some reflection on this story from OP.
What do you think you learned from this experience?
Do you think your response was appropriate?
What would you have done differently?
Are you forever afraid of Prod env?
Many, many , many of us have been in this situation before, whether as 'monumental' or not. So it is interesting to hear how others handle it.
I realize that the dev environment was a recipe for disaster, and I was simply the one to step on the mine .. but I believe my guilt about leaving the company is 'quite right'. Thankfully I'm not forever afraid of Prod env - I still do a lot of risky stuff .. but I always have nightly backups, and other 'recreate the data' strategies in place.
Risk avoidance (decent staging) and risk mitigation (backups) are two mostly orthogonal aspects of risk management. Often, a backup will be a good first step for a totally messed up system.
However, saying that mistakes will aways reach the outer layer to discount the value of risk avoidance is talking about the possibility of risk realisation where what matters is probability.
I'm not trying to discount the value of risk avoidance they are both important and should both be used, but mitigation should always be the priority of the two.
1) When you have neither you should focus on risk mitigation first.
2) Having a great and complex risk avoidance policy in place is a good thing but doesn't mean that you need a lesser mitigation system.
Guilt is a moral concept; when it comes to a run-of-the-mill operations mistake like yours, it does not belong in analysis of its consequences to the business. You are not a robot. You have made and will make mistakes this bad and worse.
Only consequentialist thinking should be the order of the day here; "what do we know that can prevent a similar mistake from hurting our bottom line". In this case backups are the standard, reasonable, well-known practice. Nothing will be improved by a firing or a resignation, by blaming or by shaming.
Insofar as the real root cause of the problem was not addressed, it's a reasonable prediction that any such company eventually joins the deadpool due to similar oversights.
The CEO leaned across the table, got in my face, and said, "this, is a monumental fuck up. You're gonna cost us millions in revenue".
No, the CEO was at fault, as was whoever let you develop against the production database.
If the CEO had any sense, he should have put you in charge of fixing the issue and then making sure it could never happen again. Taking things further, they could have asked you to find other worrying areas, and come up with fixes for those before something else bad happens.
I have no doubt that you would have taken the task extremely seriously, and the company would have ended up in a better place.
Instead, they're down an employee, and the remaining employees know that if they make a mistake, they'll be out of the door.
Last year I worked at a start-up that had manually created accounts for a few celebrities when they launched, in a gutsy and legally grey bid to improve their proposition†. While refactoring the code that handled email opt-out lists I missed a && at the end of a long conditional and failed to notice a second, otherwise unused opt-out system that dealt specifically with these users. It was there to ensure they really, really never got emailed. The result?
These mistakes are almost without fail a healthy mix of individual incompetence and organisational failure. Many things - mostly my paying better attention to functionality I rewrite, but also the company not having multiple undocumented systems for one task, or code review, or automated testing - might have saved the day.
If you want to see monumental screw-up, look at knight capital group (they accumulated a multi billion dollar position in the span of minutes, losing upwards of $440M, and ended up having to accept a bailout and sell itself to GETCO):
423 comments
[ 3.6 ms ] story [ 302 ms ] threadThough I would look askance at whoever hired a philosophy grad as well, to be perfectly honest. The author admits he didn't have the experience to spot bad practice at the time.
What's more questionable is:
* Developers have access to the production database from their machine, while it should only be accessible to the front machines within the datacenter.
* Junior developers don't need an access to production machine, only sysops and maybe the technical PM.
* No backup of the production database. WTF???
If they had a hardware failure they would have been in the same shit.
* No Foreign Keys
Attempting to clear the table should have just thrown a constraint violation error.
"on delete cascade"!
You'd have to use TRUNCATE CASCADE on postgres to avoid the foreign key error.
Also, someone who actually had the development experience and knowledge of better-practice would not have taken that position.
Protecting the data is called a disaster recovery (DR) plan in those big outdated companies that people like to make fun of.
The reason that these companies have a DR plan is to tell the CEO that 'when' (not 'if') all of the data goes away, (a) how long will it take to get it back and (b) how out-of-sync will this data be (2 minutes from freshness? 6 hours?).
My experience is that this is the norm, not the exception.
Also, you have a choice of leaving if you don't like the job, and or don't find the practices in place to be any good, or you can fix them.
It doesn't cover every last thing, but a team following these practices is the kind of team you're looking for. Ask these questions at your interviews.
Additionally, start networking now. Get to know ace developers in your area, and you will start hearing about top-level development shops. Go to meetups and other events where strong developers are likely to gather (or really, developers who give a shit about proper engineering) and meet people there.
It's next to impossible to know, walking into an office building, whether the company is a fucked up joke or good at what it does - people will tell you.
But, things are not always perfect, even on great teams. Not saying its normal to destroy your production database! But even in good shops it's a constant challenge to stay organized and do great work. Look for a team that is at least trying to do great work, rather than a complacent team.
I honestly think the test is a little out of date, but if they say, "Well, instead of X we're doing better thing Y", that's a great answer.
I'm quite surprised stuff like this hadn't happened earlier. When I am doing development with a database I will quite often trash the data when writing code.
Anecdote:
I am working for a company that does some data analysis for marketers aggregated from a vast number of sources. There was a giant legacy MyISAM (this becomes important later) table with lots of imported data. One day, I made some trivial looking migration (added a flag column to that table). I tested it locally, rolled it out to staging server. Everything seemed A-OK until we started migration on the production server. Suddenly, everything broke. By everything, I mean EVERYTHING, our web application showed massive 500-s, total DEFCON1 across the whole company. It turned out we ran out of disk space, since apparently myisam tables are altered the following way: first the table is created with updated schema, then it is populated with data from the old table. MyISAM ran out of disk space and somehow corrupted the existing tables, mysql server would start with blank tables, with all data lost.
I can confirm this very feeling: "The implications of what I'd just done didn't immediately hit me. I first had a truly out-of-body experience, seeming to hover above the darkened room of hackers, each hunched over glowing terminals." Also, I distinctly remember how I shivered and my hands shook. It felt like my body temperature fell by several degrees.
Fortunately for me, there was a daily backup routine in place. Still, several hour long outage and lots of apologies to angry clients.
"There are two types of people in this world, those who have lost data, and those who are going to lose data"
More than once (over the last few years) I've been doing some important update. I tend to do it the same way.
Of course, if you happen to run into the 1 or 2 MyISAM tables that no one knew were MyISAM, abort doesn't do anything. You've screwed up the data and need a backup.So you always have to make a backup and check that the tables are defined they way they should be. Nothing is quite as much fun as the first time you delete a bunch of critical data off a production system. Luckily the table was small, basically static, and we had backups so it was only a problem for ~5 minutes.
PostgreSQL can do DDL inside a transaction though.
We have dev databases (one of which was recently empty, nobody knows why; but that's another matter), then a staging environment, and finally production. And the database in the staging environment runs on a weaker machine than the prod database. So before any schema change goes into production, we do a time measurement in the staging environment to have a rough upper bound for how long it will take, how much disc space it uses etc.
And we have a monthly sync from prod to staging, so the staging db isn't much smaller than prod db.
And the small team of developers occasionally decides to do a restore of the prod db in the development environment.
The downside is that we can't easily keep sensitive production data to find its way into the development environment.
I try to keep data having the same form (e.g., length, number of records, similar relationships, looks like production data). But it's random enough so that if the data ever leaks, we don't have to apologize to everybody.
Since your handle is perlgeek, you're already well equipped to do a streaming transformation of your SQL dump. :)
-cp the dump to a new working copy
-sed out cache and tmp tables
-Replace all personal user data with placeholders. This part can be tricky, because you have to find everywhere this lives (are form submissions stored and do they have PII?)
-Some more sed to deal with actions/triggers that are linked to production's db user specifically.
-Finally, scp the sanitized dump to the dev server, where it awaits a Jenkins job to import the new dump.
The cron job happens on the production DB server itself overnight (keeping the PII exposure at the same level it is already), so we don't even have to think about it. We've got a working, sanitized database dump ready and waiting every morning, and a fresh prod-like environment built for us when we log on. It's a beautiful thing.
But as far as I know, nobody was fired for this. Because, yes, things like this just can happen. An eventually it got fixed anyway.
Luckily the company I was at (like any rational company) backed up their db and worked in different environments, so it was more of a thing my coworkers teased me for than an apocalyptic event.
(The other character is a # or $ depending on whether the user is root or not.)
Did you change the $PS1 variable? Can you share your config?
It means having a slightly difference .bashrc for each machine, but it's trivial.
(molly-guard makes you type in the hostname before a halt/shutdown/reboot command.)
Have a wonderful day! (and i'll definitely look at installing molly-guard on my production debian servers)
Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about.
And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.
I asked similar, and agree that it's a really useful question.
I think it's an especially great one for startups, as successful candidates are more likely to come into contact with production systems.
For these positions you not only want people capable of recovering accidents, but also people who have screwed up because, conversely, they've been trusted not to screw up. Those who've never been trusted enough to not damage a system are unlikely to be of much use.
When people say "making mistakes is unacceptable - imagine if doctors made mistakes" they ignore three facts:
1. Doctors do make mistakes. Lots of them. All the time.
2. Even an average doctor is paid an awful lot more than me.
3. Doctors have other people analysing where things can go wrong, and recommending fixes.
If you want fewer development mistakes, as a company you have to accept it will cost money and take more time. It's for a manager to decide where the optimal tradeoff exists.
This is absolutely it, of course it is possible to become so risk averse that you never actually succeed in getting anything done and there are certainly organisations that suffer from that (usually larger ones).
However some people seem to take the view that it is impossible to protect oneself from all risks therefor it is pointless protecting from any of them.
The good news is that usually protecting against risks tends to get exponentially more expensive as you add "nines" therefor having a 99% guarantee against data loss is a lot cheaper than a 99.999% guarantee.
Having a cronjob that does a mysqldump of an entire database, emails some administrator and then does rsync to some other location (even just a dropbox folder) is something that is probably only a couple of hours work.
This. I don't regret my life's many failures. I regret the times I've flamed out, blamed others, or ran away.
Doing things means making mistakes. You can spot a professional by how they deal with mistakes.
In most companies there's enough process to protect you from the big screw ups.
The immediate takeaway from a disaster should always be 'How can we make sure this doesn't happen again?' not 'Man, I can't believe Fred did that, what an idiot.'
If you are a CEO you should be asking this question: "How many people in this company can unilaterally destroy our entire business model?"
If you are a CTO you should be asking this question: "How quickly can we recover from a perfect storm?"
They didn't ask those questions, they couldn't take responsibility, they blamed the junior developer. I think I know who the real fuckups are.
As an aside: Way back in time I caused about ten thousand companies to have to refile some pretty important government documents because I was doubling xml decoding (& became &). My boss actually laughed and was like "we should have caught this a long time ago"... by we he actually meant himself and support.
This is a question that the person in charge of backups needs to think about, too. I mean, rephrase it as "Is there any one person who can write to both production and backup copies of critical data?" but it means the same thing as what you said.
(and if the CTO, or whoever is in charge of backups screws up this question? the 'perfect storm' means "all your data is gone" - dono about you, but my plan for that involves bankruptcy court, and a whole lot of personal shame. Someone coming in and stealing all the hardware? not nearly as big of a deal, as long as I've still got the data. My own 'backup' house is not in order, well, for lots of reasons, mostly having to do with performance, so I live with this low-level fear every day.)
Seriously, think, for a moment. There's at least one kid with root on production /and/ access to the backups, right? At most small companies, that is all your 'root-level' sysadmins.
That's bad. What if his (or her) account credentials get compromised? (or what if they go rogue? it happens. Not often, and usually when it does it's "But this is really best for the company" It's pretty rare that a SysAdmin actively and directly attempts to destroy a company.)
(SysAdmins going fully rogue is pretty rare, but I think it's still a good thought experiment. If there is no way for the user to destroy something when they are actively hostile, you /know/ they can't destroy it by accident. It's the only way to be sure.)
The point of backups, primarily, is to cover your ass when someone screws up, primarily. (RAID, on the other hand, is primarily to cover your ass when hardware fails) - RAID is not Backup and Backup is not RAID. You need to keep this in mind when designing your backup, and when designing your RAID.
(Yes, backup is also nice when the hardware failure gets so bad that RAID can't save you; but you know what? that's pretty goddamn rare, compared to 'someone fucked up.')
I mean, the worst case backup system would be a system that remotely writes all local data off site, without keeping snapshots or some way of reverting. That's not a backup at all; that's a RAID.
The best case backup is some sort of remote backup where you physically can't overwrite the goddamn thing for X days. Traditionally, this is done with off-site tape. I (or rather, your junior sysadmin monkey) writes the backup to tape, then tests the tape, then gives the tape to the iron mountain truck to stick in a safe. (if your company has money; if not, the safe is under the owner's bed.)
I think that with modern snapshots, it would be interesting to create a 'cloud backup' service where you have a 'do not allow overwrite before date X' parameter, and it wouldn't be that hard to implement, but I don't know of anyone that does it. The hard part about doing it in house is that the person who managed the backup server couldn't have root on production and vis-a-vis, or you defeat the point, so this is one case where outsourcing is very likely to be better than anything you could do yourself.
Which also means they can't fix something in case of a catastrophic event. "Recover a file deleted from ext3? Fix a borked NTFS partition? Salvage a crashed MySQL table? Sorry boss, no can do - my admin powers have been neutered so that I don't break something 'by accident, wink wink nudge nudge'." This is, ultimately, an issue of trust, not of artificial technical limitations.
> one case where outsourcing is very likely to be better than anything you could do yourself.
Hm. Your idea that "cloud is actually pixie dust magically solving all problems" seems to fail your very own test. Is there a way to prevent the outsourced admins from, um, destroying something when they are actively hostile? Nope, you've only added a layer of indirection.
(also, "rouge" is "#993366", not "sabotage")
>Which also means they can't fix something in case of a catastrophic event. "Recover a file deleted from ext3? Fix a borked NTFS partition? Salvage a crashed MySQL table? Sorry boss, no can do - my admin powers have been neutered so that I don't break something 'by accident, wink wink nudge nudge'." This is, ultimately, an issue of trust, not of artificial technical limitations.
All of the problems you describe can be solved by spare hardware and read only access to the backups. I mean, your SysAdmin needs control over the production environment, right? to do his or her job. but a sysadmin can function just fine without being able to overwrite backups. (assuming there is someone else around to admin the backup server.)
fixing my spelling now.
Yes, it's about trust. but anyone who demands absolute trust is, well, at the very least an overconfident asshole. I mean, in a properly designed backup system (and I don't have anything at all like this at the moment) I would not have write-access to the backups, and I'm majority shareholder and lead sysadmin.
That's what I'm saying... backups are primarily there when someone screwed it up... in other words, when someone was trusted (or trusted themselves) too much.
(that rouge/rogue thing is my pet peeve)
the idea here is to make sure that the people with write-access to production don't have write-access to the backups and vis-a-vis. The point is that now two people have to screw it up before I lose data.
Outsourcing has it's place. You are an idiot if you outsource production and backups to the same people, though. This is why I think "the cloud" is a bad way of thinking about it. Linode and rackspace are completely different companies... one of them screwing it up is not going to effect the other.
Does Amazon Glacier offer this?
Redundancy = Reduce the number of component failures that can lead to system failure (RAID, live replication, hot standby).
Backup = Recover from an obvious failure or overwite (Weekly full backups, daily differentials).
Archival = Recover from a non-obvious failure and/or malicious activity (WORM tapes, offsite backup).
As a failsafe against malicious sysadmins is to split up the responsibilities. The guy handling backups isn't handling archival etc...
I test backups for F500 companies on a daily basis (IT Risk Consulting) - this would be missing the point really, the business process around this problem is really moving towards live mirrored replication. This allows much faster recall time, and also mitigates many risks with the conventional 'snapshot' method through either tapes, cloud, etc.
I was handed a legacy codebase with zero tests. I left a few small bugs in production, and got absolutely chewed out for it. It was never an issue with our processes, it was obviously an issue with the guy they hired who had 1 intro CS class and 1 rails hobby project on his resume. The lead dev never really cared that we didn't have tests, or a careful deploy process. He just got angry when things went wrong. And even gave one more dev with even less experience than I had access to the code base.
It was a mess and the only thing I was "learning" was "don't touch any code you don't have to because who knows what you might break" which is obviously a terrible way to learn as a junior dev (forget "move fast and break things" we "move slowly and work in fear!"). So I quit and moved on, it was one of the better decisions I've ever made.
Seven and a half years later I make sure that I pass that knowledge on to my junior colleagues. I'm proud to say that just in the past 2 weeks I've said this twice to one of my younger team-mates, a recent hire, "don't be afraid to break things!"
"Don't be afraid to break things as long as you have a backup".
It might be a simple version of the previous code, database copy or even the entire application. Do not forget to backup. If everything fails, we can quickly restore the previous working version.
Every production deployment should involve blowing away the prior instance, rebuilding from scratch, and restarting the service; you are effectively doing a near-full "restore" for every deployment, which forces you to have everything fully backed up and accessible...
Any failure to maintain good business continuity practices will manifest early for a product / employee / team, which allows you to prevent larger failures...
In the world where data needs to be maintained, this is not necessarily an option. In the bank where I work, we deploy new code without taking any outage (provide a new set of stored procedures in the database, then deploy a second set of middleware, then a new set of front-end servers, test it, then begin starting new user sessions on the new system; when all old user sessions have completed the old version can be turned off). Taking down the database AT ALL would require user outages. Restoring the database from backup is VERY HARD and would take hours (LOTS of hours).
That being said, we do NOT test our disaster-recovery and restore procedures well enough.
The bug is in the part that I though it was so obvious, I missed some check.
it first builds inefficieny at the technical level then at the business level, finally it causes issues at the cultural level, and thats when the smart people start leaving.
In high tech this can get really messy, these are frequently inherently more fragile companies. My favorite example is from Robert X. Cringley in this great book: http://www.amazon.com/Accidental-Empires-Silicon-Millions-Co... ; from memory:
One day Intel's yields suddenly went to hell (that's the ratio of working die on a wafer to non-working, and is a key to profitability). And no matter how hard they tried, they could only narrow it down to the wafers being contaminated, but the wafer supplier swore up and down they were shipping good stuff, and they were. So eventually they tasked a guy to follow packages from the supplier all the way to the fab lines, and he found the problem in Intel's receiving department. Where a clerk was breaking open the sealed packages and counting out the wafers on his desk to make damned sure Intel was getting its money worth....
His point is that you can have a Fortune 500 company, normally thought to be stable companies that won't go "poof" without ample warning, in which there are many more people than in previous kinds of companies who can very quickly kill it dead.
One of Toyota's mantras is "If the student has failed to learn, the teacher has failed to teach." Their point is that managers are responsible for solving issues that come from employee ignorance, not line workers.
I do understand that humans don't have an instinctual understanding of using nipples, human babies have a physical sucking reflex that kicks in when you put anything near their mouths. They usually quickly learn that sucking a nipple in a particular way gives out lots of yummy milk.
I find this hard to believe. At some point a person in a space suit was introducing them into a clean room; she should have noticed that the packages were not sealed.
There's a good chance a tamper revealing seal would have stopped the clerk from opening the containers, and of course it they'd been broken before reaching the people at the fab lines who were supposed to open them that would have clued Intel into the problem before any went into production and would have allowed them to quickly trace the problem back upstream.
Or in this case energy, ignorance, and not learning enough of the big picture, or wondering why these wafers were sealed in air tight containers in the first place.
It's the well meaning mistakes that tend to be the most dangerous since most people are of good will, or at least don't want to kill the company and lose their jobs.
A mere few months into my current job, I ran an SQL query that updated some rows without properly checking one of the subqueries. Long story short - I fucked up an entire attribute table on a production machine (client insisted they didn't need a dev clone). I literally just broke down and started panicking, sure that I'd just lost my new job and wouldn't even have a reference to run on for a new one.
After a few minutes of me freaking out, my boss just says: "You fucked up. I've fucked up before, probably worse than this. Let's go fix it." And that was it. We fixed it (made it even better than it was, as we fixed the issue I was working on in the process), and instead of feeling like an outcast who didn't belong, I learned something and became a better dev for it.
1) A cron job for the manually task you were doing.
2) Not working directly on production.
3) Having daily backups
And this could happened to anybody. After midnight any of us are at junior level and very prone to do this kind of mistakes.
The DB needed to be searched and replaced to remove all the old urls. After doing so, the wp_options cell on the production site holding much of the customizations kicked back to the defaults for the theme, the serialized data format being used was sensitive to brute DB-level changes.
I had talked to my client before about putting together a decent process including dev databases, scheduled backups, everything needed to prevent just such a screwup, but he waffled. Then blamed me when things went wrong.
I'd had enough and told him to do his own tech work, leaving him to fix his client's website himself. Being that I didn't build it, I didn't know which settings to flip back. I left freelance work and never looked back.
People and companies do this all the time, refuse to spend the time and money ensuring their systems won't break when you need them the most, then scapegoat the poor little technician when it does.
I'd like to say the answer is "don't work in such environments," but there's really no saying that it won't be this way at the next job you work, either.
I certainly wouldn't internalize any guilt being handed down, ultimately it's the founders' jobs to make sure that the proper systems are in place, after all, they have much more on the line than you do. Count it a blessing that you can just walk away and find another job.
Take my own company, I've accidentally deleted /dev on development servers (not that major of an issue thanks to udev, but the timing of the mistake was lousy), a co-worker recently dropped a critical table on dev database and we've had other engineers break Solaris by carelessly punching in chmod -R / as root (we've since revised engineers permissions so this is no longer possible). As much as those errors are stupid and as much as engineers of our calibre should know better, it can only takes a minor lack of concentration at the wrong moment to make a major fsck up. Which is doubly scary when you consider how many interruptions the average engineer gets a day.
So I think the real guilt belongs to the entire technical staff as this is a cascade of minor fcsk ups that lead to something catastrophic.
Many, many , many of us have been in this situation before, whether as 'monumental' or not. So it is interesting to hear how others handle it.
I realize that the dev environment was a recipe for disaster, and I was simply the one to step on the mine .. but I believe my guilt about leaving the company is 'quite right'. Thankfully I'm not forever afraid of Prod env - I still do a lot of risky stuff .. but I always have nightly backups, and other 'recreate the data' strategies in place.
You also could set up 20 layers of dev environments and it still doesn't matter, mistakes can still reach the outer layer.
You need to have the ability to recover from any problem quickly and with the data as updated as you need it to be.
1) When you have neither you should focus on risk mitigation first. 2) Having a great and complex risk avoidance policy in place is a good thing but doesn't mean that you need a lesser mitigation system.
Guilt is a moral concept; when it comes to a run-of-the-mill operations mistake like yours, it does not belong in analysis of its consequences to the business. You are not a robot. You have made and will make mistakes this bad and worse.
Only consequentialist thinking should be the order of the day here; "what do we know that can prevent a similar mistake from hurting our bottom line". In this case backups are the standard, reasonable, well-known practice. Nothing will be improved by a firing or a resignation, by blaming or by shaming.
Insofar as the real root cause of the problem was not addressed, it's a reasonable prediction that any such company eventually joins the deadpool due to similar oversights.
No, the CEO was at fault, as was whoever let you develop against the production database.
If the CEO had any sense, he should have put you in charge of fixing the issue and then making sure it could never happen again. Taking things further, they could have asked you to find other worrying areas, and come up with fixes for those before something else bad happens.
I have no doubt that you would have taken the task extremely seriously, and the company would have ended up in a better place.
Instead, they're down an employee, and the remaining employees know that if they make a mistake, they'll be out of the door.
And they still have an empty users table.
http://krugman.blogs.nytimes.com/2011/08/11/academia-nuts/
What a screw up!
These mistakes are almost without fail a healthy mix of individual incompetence and organisational failure. Many things - mostly my paying better attention to functionality I rewrite, but also the company not having multiple undocumented systems for one task, or code review, or automated testing - might have saved the day.
[†] They've long been removed.
http://dealbook.nytimes.com/2012/08/03/trading-program-ran-a...