I think having a day for a whole year is a bit sparse. Some startups start and shutdown within a year. Apart from checking backups after any big code change related to backups, I think backups should be checked quarterly.
It takes no more than couple hours most of the time, and as our wise said, "An ounce of prevention is worth a pound of cure"
Except the reason they had to shut down might be because they never checked their backups and then they lost a critical amount of data and it turned out that the backups were indeed no good.
I don't think any startup knows that they're going to shutdown within the year. No one would take the time if they knew they were going to shut down soon.
It's in there now. Row counts per table and checksums are written out with the backup for every table. If things don't match up, alarm in a loud and noisy way.
My point is that there are sanity checks built into the process. It's not difficult. There doesn't need to be a human verifying things there. Is the row count increasing over the previous backup? Check. Is the checksum different? Check. Is it non-zero? Check. And on and on.
Stop making excuses and automate testing your backups.
so, when your check process does not report because it failed to run, how do you know? Do you have a monitor for your monitor? Does that monitor have a monitor?
There is no reason not to automate your backup tests, but there is no reason not to eyeball the check is actually working from time to time.
Indeed, If one day a year is what it takes for you to remember to care about backups, you probably shouldn't be involved with backups. This could happen to anyone from a person losing their home photo collection, to a hospital missing critical data within a patient management system - I know because I've been involved or seen first hand both of these exact scenarios. It's what we learn from our mistakes that defines our future, not the people telling us we made them.
Oh wow, some interesting stuff in here. Looks like they use MySQL as a queue for scheduling the ORC Peons? Would have loved to hear more about why they did that.
Because it works perfectly fine? I tend to write anyone off who scoffs at simple database-as-queue designs without understanding what the scaling requirements are. You can use a database as a job queue for 10s of thousands of jobs per day without any sweat.
Please forgive me, but I don't understand your hostility.
I understand that there are perfectly legitimate reasons for using a database as a queue. If you frequently need to look at and rearrange your jobs while they're in flight, chances are a pure FIFO structure probably doesn't work that well for you anyways. If the enqueue is contingent on a transaction committing, probably makes a lot of sense for the job to be in the same database. You don't need to tell me -- I've seen more than a few in production.
But an actual message queue also "works perfectly fine" based on the information provided, and I would imagine that a company like Facebook probably already has a few of those lying around. It would have been a conscious choice to use MySQL as a queue.
We use it because it works well for us. We've put a lot of work into making MySQL scale for us to the point where it's a very well supported system and one of the main choices for a lot of storage decisions.
We even use MySQL as a queue for Facebook Messenger. More details about this:
A couple of months ago, in a public chatroom for DBAs, one of GitLab's engineering leads made some very insulting comments about Facebook engineers & eng practices. Pretty ironic.
I want to feel bad for GibLab, but it's really, really hard when they hire people like that.
Would extend this to: "keep your personal data backed up too" - and for low-value or low-security data, it's easily done with consumer commercial services, since they're designed to allow you to be lazy and have no backup discipline.
This actually inspired me to set up my daily / monthly backups, which I had not done since upgrading to Devuan almost half a year ago. Fortunately, I had already written a blog post [1] about backups, so setting myself back up and adding a cron task took under 30 min with a fresh disk.
Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time and even then it could still just stop working one day.
We should probably treat this issue as something more like a disease like high blood pressure, namely, you don't know you have it, but it is probably doing irreparable damage to your internal organs. If we had no name for the disease or understanding of it we would just die at an earlier age without an obvious cause.
Let's identify the diseases of this sort in our industry and work on prevention, diagnosis and treatment of them instead of just saying you should have been more careful.
Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers.
Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it damages it !
Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and no written plan). The solutions above are just guesses at this point, we'll dive into this in the coming days and will communicate what we will do in a blog post.
Good morning (posted from throwaway for reasons Ill describe).
I feel for you greatly here, and I commend your openness about how data restoration caused 6 hours of data loss. I too work in a critical area where even minutes DB lost is bad.
We just had our own test event recently. We make sure that we can fail everything over, and run on all secondaries. I found out how that worked; we failed. The problem with this, is I found out after the fact. Due to the secrecy, not even the teams knew why things failed the way they did. I had to piece it together from disjoint hearsay, and now I believe I have a competent picture.
So yes, when I read your post mortem and RCA, it reminded me greatly of what happened here as well. But we all can learn from your example. As for me, I'm posting it as a throwaway due to likely threats on my job.
I admire the response and quick action, but after reading this, I get the strong feeling Gitlab et al didn't know they were running a real business with real projects and real people who trusted you to be a good custodian of their data.
I also would ask Why are we running our own Postgre setup and not using RDS? And then Why do we not have production Postgre DBAs on staff that would do this rather than an engineer?
Completely agree. This is 1000x better than the typical cooperate PR response that says nothing which we'll usually get. Kudos for their honesty and openness, even in a somewhat embarrassing situation such as that.
it is also very, very risky from a liability perspective, both for insurance purposes and third-party indemnification. That's why most companies will keep as silent as possible.
> I've never heard of anyone successfully suing AWS/Google/Azure over lost data
Those companies have huge legal departments writing iron-clad contracts and stocking a lot of very sharp knives for any such eventuality. Smaller companies make for much easier targets.
> even if your notes are kept private, it'll still come out in discovery
Those notes might have been lost by then - that is, if their existence at any given time is even known. It is perfectly reasonable for people working quickly after an outage, to not actually write down every step or observation they make.
> What are the risks here?
Not prison, but you might be forced to pay a bunch if someone manages to get a ruling against you for negligence or the likes.
Imagine how deep their pockets are. Now imagine how deep your pockets would need to be to go toe to toe with them.
Sure plenty of uber larges Incs are "in the cloud" but with mission critical (read: high value worth sueing over) applications and data? That's a pretty small universe.
I think this is a big slap to company when they need to talk to their enterprise customers. I am not sure how we can really prevent this from happening by implementing an industry standard.
You know, I remember reading years ago that during the launch preparation for Apollo 16(?), they did a routine pressure test of the Command Module atop the rocket a few days before launch. The technician in charge made the simple error of forgetting to open a pressure release valve during the test which lead to an overpressurisation of the CM causing significant damage, including separation of the heat shield.
The ENTIRE Saturn 5 rocket had to be wheeled back to the hangar and the CM dismantled and rebuilt, resulting in a month long delay in the launch.
When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again."
I am willing to bet that Gitlab is now a company that will never slacken off their backup checking in the future.
>When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again."
This same general story appears in different forms with million dollar trading errors, etc. But I have to wonder if it's really that good of a lesson. If you truly take it to heart, you could potentially keep people on the team that are legitimately incompetent which could result in another catastrophic failure.
And if one starts applying your principle at a workplace, you end up with the whole company being afraid of doing anything at all, to avoid being fired.
No, there is a level of competency where once you make mistakes below that, firing is the right move.
If a surgeon keeps killing patients because he keeps forgetting to wash his hands, then he needs to be fired.
If a developer skips all company policies and deploys directly to production without a good reason, then he needs to be fired.
There are reasonable mistakes, then there are errors resulting from wreckless actions indicative of a larger problem with the person's view towards their work.
It is not hard to distinguish between someone who makes a rare, but totally understandable simple mistake, and someone who regularly screws up significantly more often than everyone else. Being tolerant of mistakes makes you a good employer. Being tolerant of regular incompetence makes you a bad one.
That is the wrong question. The real question is what did they do to ensure that this mistake cannot be made again by anybody. In the form this variation is in, the real question is what changes were made to the design so that the system cannot be over pressurized in that way by mistake.
Sometimes you can put in all the checks and balances you wish, but humans will just make a, well, human mistake.
All commercial aircraft have checklists in place, and a two person cross checking system in the cockpit, as well as mechanical means to detect potential problems - but aircraft still every now and then land with their gear still up etc.
Does it make economic sense for an organisation that has invested possibly millions of dollars in training a pilot to immediately sack him/her for a costly mistake like this? Sure, disciplinary action and possible demotion within the ranks is a given, but will you get rid of the one person who will be almost fanatical about never landing with the gear up again?
If they consistently make technical mistake and are sloppy in their airmanship, then sure, fire away. But it is going to be pretty much impossible to design a 'system' that can prevent human errors from creeping in.
Thanks. Just landed in Europe for Git Merge. It is heartwarming to read some of the comments here. We realize that we only get this leniency once and we'll learn from this and communicate our lessons in a few days from now.
The lesson here has been learned over and over again, painfully.
For every backup system you have, test restores, from scratch, periodically. The more critical the data is to your business, the more frequently and more automated you want those backup checks.
Of course you also want procedures to try to prevent tired admins from deleting production databases. To the greatest degree possible, implement systems to prevent manual tampering with production data; often there's an alternative. But databases can get broken or corrupt without admin interference, so preventing manual database removal or corruption is not an ultimate solution.
Why is it you (GitLab) and every other company has to learn from your own mistakes again and again instead of learning from others? Just a cursory glance at a few log files would have told you that your backups are not happening. Nevermind verifying actual backups...
Do you have DBAs? Are they completely inept at their job? Don't answer.
Backup is elementary. It should not need to be learned nor manifest itself as a hindsight revelation. Frankly GitLab seems more and more like a mom and pop shop that learned Rails over a weekend. They did not even know if they had functioning backups before shit hit the fan.
Yours is an entirely appropriate response. While I appreciate the gushing solidarity in the rest of this thread, non working backups really is an elementary IT error, accurately attributable to incompetence.
I could not agree more @soheil, let me also say that Sid aka @sytse is not only a top notch CEO but he's also a hard working, caring individual who truly not only cares about the product but the whole community so, so much.
Incidents are inevitable and it's important to have a proper RCA/Service-Disruption process in place to handle those.
As mentioned on the other thread, maybe Gitlab doesn't have enough operational/SRE expertise in-house yet, but that was the case in every fast-growing company I've worked for last decade.
> even then it could still just stop working one day
This is where automated testing helps. My mail server has a sister VM out in the wild that once a day picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup. If I don't get the message, restoring the backup failed. If the message looks too old then making or transferring the backup failed.
My source control services and static web servers do something similar. None of the shadow copies are available to the world, though I can see them via VPN to perform manual check occasionally and if something nasty happens they are only a few firewall and routing rules away from being used as DR sites (they are slower as in their normal just-testing-the-backups operation they don't need nearly the same resources as the live copies, but slow is better than no!).
This won't catch everything of course, but it catches many things that not doing it would not. The time spent maintaining the automation (which itself could have faults of course) is time well spent if done intelligently. For a system as large in scale as GitLabs then a full restore daily is probably not practical so a more selective heuristic will need to be chosen if you are operating at such a scale. My arrangement still needs some manual checking and sometimes I'm too busy or just forget, so again it isn't perfect, but the risk of making it more clever and inviting failure that way is at this point worse than the risk of my being lazy at exactly the wrong time.
One thing my arrangement doesn't test is point-in-time restores (because sometimes the problem happened last week, so last night's backup is no use) but there is a limit to how much you can practically do.
> The problem of restoring non-existing backups should be treated as a more serious problem in our industry
It is by people that care about it, but not enough people care, and too many people see the resources needed to get it right as an expense rather than an investment for future mental health.
It isn't just non-existent backups. Any backup arrangement could be subject to corruption either through hardware fault, process fault, or deliberate action (the old "they hacked us and took out our backups too" - I really must get around to publishing my notes on soft-offline backups...).
> Let's identify the diseases of this sort in our industry
Apathy mainly.
The people who care most are either naturally paranoid (like me), have lost important data at some point in the past so know the feeling first hand (thankfully not me, though in part to having a backup strategy that worked) or have had to have the difficult conversation with another party (sorry, I can't magic your data back for you, it really is gone) and watch the pitiful expressions as they beg for the impossible.
The only way to enforce the correct due diligence is to make someone responsible for it, it is more a management problem than a technical one because the technical approaches needed pretty much all exist and for the most part are well studies and documented.
Of course to an extent you have to accept reasonable risks. It is usually not practical to do everything that could be done, and understandable human error always needs to be accounted for as do "acts of Murphy". But someone needs to be responsible for deciding what sort of risk to take (by not doing something, or doing something less ideal) rather than them just being taken by general inaction.
> picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup.
This is sensible, but one problem with this is developing "blindness" for messages you receive every day. For some recurring tasks you can automate a bit further. Instead of receiving a success message every day, receive a failure message the first day there's an absence of success message. A number of tools and services exist for setting this up, one is linked in my HN profile (shameless plug ;-)
> This is sensible, but one problem with this is developing "blindness" for messages you receive every day.
This can be an issue, I do get lazy about checking them when I'm busy.
The next step will be to automate checking them a bit more, rather than stopping the positive messages. A simple script that for the mail example logs in via POP/IMAP, checks for the last notification and checks the relevant timestamps. Similar for other services (is the last commit in the repo from more than 24 hours ago?). Then I get a simple "all OK" or not. I still have to run the script, perhaps hooking it to CGI and setting it to my browser home page to further remove my laziness from the equation, but I don't want to just be told when something is wrong as I'll never really trust that something going wrong won't block warning messages - I want to make checking as easy as possible though.
I may play with your service and add it as a secondary set of checks though!
Now that I'm developing email stuff I get all the time the existential crisis of "how to I make sure my testing and alerting infrastructure is working?"
Really, how can you be sure your restoration tests are running correctly and their messages get all the way to you?
I think I'll add periodic positive messages for my restoration procedures. Stuff like (if success and day % 23 == 0 then email_something). Still doesn't guarantee the test is correct.
> Really, how can you be sure your restoration tests are running correctly and their messages get all the way to you?
I get a daily message, so if I know if things aren't getting sent or getting through. I've been wary of setting up anything that only sends problem messages because regularly getting the "all OK" messages proves that at least that part of the infrastructure is working.
The problem with that is when you start getting too many "all OK" messages.
Getting a message must be mostly a surprise. One daily message is OK for me, but if so it may be good to tell about more than a backup. A daily digest of "everything is up and running" may indeed be something very nice to have.
And how do you ensure a team will always have someone surprised not to get a daily message? Maybe sending for may people, or requiring some acknowledge, and creating a warning over the warning channel if there isn't one. (And you certainly aren't going to send those positive messages over the warning channel, because it requires a completely different kind of action.)
That is why I'm thinking of scripting up something to scan the status messages and present a list (in the console or on a web page), colour highlighted (normal for info only, green for all OK, red for error or missed message, big red bold (flashing?) for an issue still not resolved from the previous day. That automates checking but I've still got the individual updates in case that fails. Maybe I'll make the output of this my homepage and/or console login banner.
> And how do you ensure a team will always have someone surprised not to get a daily message?
For a team you have a management problem not a technical one! Somebody needs to be responsible for it in the end, plain and simple.
On a less authoritarian note a technical helper might perhaps be to have a screen, always on, showing a dashboard of statuses, where everyone can see it? Make it beep incessantly when a new warning arrives until someone acknowledges it. If you acknowledge a message either deal with it (if you can) or notify whoever needs to be told so they can take action.
People will still sometimes ignore or miss the warnings. Nothing and no one is perfect.
But I'm only dealing with my own collection of personal data and projects here so it isn't that scale of problem - at work I used to be responsible for infrastructure as well as my actual job but we no longer operate at a scale where that is at all practical so I have to trust someone else is dealing with it (though I'm not that trusting I do sometimes take time out to check things myself where I have access).
> This is where automated testing helps. My mail server has a sister VM out in the wild that once a day picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup. If I don't get the message, restoring the backup failed. If the message looks too old then making or transferring the backup failed.
That's awesome. I've always tested my backups by having shitty, cheap servers so bad that restoring from backup happens about once every few months.
Going to an automated restoration test is a great idea, though also a lot of work for most people.
Just checking, manually or automatically that your backups are occurring and are of a reasonable size is probably sufficient for most operations and would have caught most if not every case in the GitLab instance.
Is it too much to ask for a few days dedicated to this? I mean, pretent that your critical infastructure goes up in smoke - can you recover, and how quickly? For most companies it simply means that you try to restore the machines from your backup. You can do this as everything else continues as normal.
This page did a lot to raise my awareness of my own inadequate backup situation. Any hurt feeling of the CEO caused by this page will be outweighed by the greater good of the message being spread.
> We should probably treat this issue as something more like a disease like high blood pressure, namely, you don't know you have it, but it is probably doing irreparable damage to your internal organs.
Things that aren't tested can pretty much be counted on to fail a non-trivial percentage of the time. If it's going to be business critical, then it needs to be tested. (Lesson hopefully learned here)
By the same token, things which aren't monitored can also be counted upon to fail a non-trivial percentage of the time and even worse, go unnoticed. (Another teachable point here)
...but people being what they are tend to learn these things the hard way unless we've got concrete, real-world incidents like this for them to learn from.
One time at work I accidentally triggered delete on an RDS CloudFormation stack It was not fun. Automatic backup from AWS is useless because automatic snapshots are removed as soon as the RDS instance is removed unless you tell AWS to make a final snapshot. We didn't have that flag in the stack template at the time, so ugh.
Oh how I deleted the stack? I was using the mobile app and was trying to look at the status of the Cfn stack, but the app was laggy my finger pressed the wrong button... sigh The other interesting thing was I checked the status because the previous night I changed my RDS instance to provisioned IOPS (took 8 hours and failed too). I felt sad and guilty, but at the same time I felt whatever because the upgrade didn't go through so perhaps this accident was all meant to be....
For critical stuff IMO there needs to be more than just a confirmation dialog. The user needs to be transitioned into a totally different state of mind from the usual click, click, click, click.
E.g. forcing someone to manually type the characters D E L E T E before allowing deletion of something potentially important.
Either that or everything should have multiple levels of undo. Everything.
There's certainly some stuff in AWS that requires you to type the name of the thing you're deleting into a text field in order to delete it, but I suspect that's purely a UI check -- there wouldn't be so much point in requiring the name twice in the underlying API.
So if you've got a dud client implementation then you're going to lose the check.
One way to do stuff like this is to have separate roles for read-only and read-write access. I pay a lot more attention to what I'm doing on the rare occasions I assume permissions to change things.
The graver problem is programmers — and, humans in general — don't realise the gravity of recklessness until the moment when shit hits the fan. Startups begin with general imprudence towards checks and processes, which is understandable but even in the growing stages, the idea of incorporating them is snubbed because of 'priorities'. "Time is precious and there are more important problems to solve in the way".
Unauthenticated MongoDb on default port? The likelihood of a person port scanning the entire web just doesn't strike as anything adverse, and then, someday someone does exactly that. I guess this is one important reason to bring at least some experienced people on board because there is a likely chance that they can give a better perspective on seriousness of such issues.
When stuff "just works", you don't need to check your backups. I fully trust my iPhone's iCloud backups, my Time Machine backups, and my cloud rsyncs. Time Machine also lets me know if they get corrupted, or if I haven't backed up in a while. That's how backups should work - an adage of "you don't have backups unless you check them" just won't work for most people.
Why should "check your backups day" apply to only enterprise backup? I've seen it repeated as gospel many times on Hacker News to folks who've lost backups that it was their own fault for not checking them. My point is we should focus on a software solution to making this more robust, and not blaming people.
I'm not saying it should apply only to enterprise backup, it's just that as you pointed out, the consumer-oriented backup solutions already work pretty reliably, making a "check your backups day" unlikely to matter to the average consumer (nevermind that given daily interaction with the gamut of Google's and Apple's services, consumers frequently "check their backups" in the course of their daily activities).
I think easy, good server-level backup software would be great. The problem is that enterprise servers are usually highly customized, part of a large, unique architecture, and contain a lot of proprietary, confidential, and potentially personal, legally protected data. That makes it a lot harder to get a one-size-fits-all backup solution set up, which means that the onus of reliable backups will, of necessity, rest upon the company's administrators.
It is very sad when no one checks backups. This bites companies every day and it's usually easy to sympathize, but there's no excuse for it. GitLab needs to perform a serious review of its processes.
I've checked out the GitLab job listings that get published on HN Who's Hiring and other places regularly (I'm currently 100% remote with my current employer and like to track other 100% remote employment opportunities). They have a salary calculator/estimator and personally, I was really underwhelmed with the values it would put out. That calculator makes a city price index adjustment from the base salary and contains a statement that says GitLab prefers to hire people who live in less expensive cities. I also remember feeling that their interview process sounded a little overbearing.
It may be time for GitLab to consider upping the ante on its recruitment procedures and adding some more experienced people to the ranks.
You might want to check your Time Machine backups by hand from time to time using the tmutil[1] tool.
My Time Machine backups in the past have been missing gigabytes of data, without me telling anything about it. And not just volatile data, like temp files or caches, but photos, music and documents.
With something like Borg where you can just mount your backups and look at them normally it's fairly easy to see whether they're ok / include what you wanted.
Of course, backing a whole platform up is more complex, and things like databases normally require custom scripting (dump -> backup dump, eg. pg_dumpall | borg create ... -).
> Our backups to S3 apparently don't work either: the bucket is empty
Ouch. I can't even imagine how that feels. This is why even despite monitoring and paging scripts, I still have an event to check my company's backups weekly. Now I don't feel so paranoid.
No hidding, no eupheminization, their live doc stream actually made me question what things did I do on my systems. Looks like convergent evolution in some parts, like prompt changes.
Make a backup, restore to test environment, run checksums, anonymize, release test environment.
That way each and every backup is tested both for integrity and ability to rebuild a working environment from it.
In my practice insufficient backup is still (unfortunately) a very common occurrence.
On another note: just having stuff stored with triple replication in the cloud is emphatically NOT a backup.
And it also helps if the same people that have access to the live environment do not have write access to the backups, but that's only feasible past a certain team size.
> After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com
Everyone is talking about backups, but why not about this? How is it even possible to delete the production database by accident? Why does he have SSH access there? Why do they test their database replication in production? Why are they fire-fighting by changing random database settings _in production_?
I know that all of this is common practice. I am questioning it in general.
Because when something goes wrong he has to be able to discover what it was.
> Why do they test their database replication in production?
Many places do that. With free databases there is simply no reason to, but the practices are inherited from the non-free databases best practices arsenal.
People, if you are using postgres and have a cluster in production, make sure you also have the script that creates this cluster in a VCS and is able to create an equal cluster on virtual machines in your computer.
> Why are they fire-fighting by changing random database settings _in production_?
Oh, that's because they are fire-fighting. The problem is that when you get a problem you know nothing about, you don't really know what to replicate in another environment to reliably test things there. The most you can do is verify that your changes aren't harmful, the real test is always in production.
151 comments
[ 2.8 ms ] story [ 245 ms ] threadIt takes no more than couple hours most of the time, and as our wise said, "An ounce of prevention is worth a pound of cure"
Stop making excuses and automate testing your backups.
There is no reason not to automate your backup tests, but there is no reason not to eyeball the check is actually working from time to time.
MySQL is also pretty good about not starting if the data is corrupted for whatever reason. The process not starting is a pretty obvious alarm, too.
Checks are cheap. Better to automate your logic and invest in good monitoring and automation.
[0]: http://www.snopes.com/holidays/aprilfools/cleaning.asp
https://blog.codinghorror.com/international-backup-awareness...
https://code.facebook.com/posts/1007323976059780/continuous-...
I understand that there are perfectly legitimate reasons for using a database as a queue. If you frequently need to look at and rearrange your jobs while they're in flight, chances are a pure FIFO structure probably doesn't work that well for you anyways. If the enqueue is contingent on a transaction committing, probably makes a lot of sense for the job to be in the same database. You don't need to tell me -- I've seen more than a few in production.
But an actual message queue also "works perfectly fine" based on the information provided, and I would imagine that a company like Facebook probably already has a few of those lying around. It would have been a conscious choice to use MySQL as a queue.
I swear I'm actually, seriously just curious!
We use it because it works well for us. We've put a lot of work into making MySQL scale for us to the point where it's a very well supported system and one of the main choices for a lot of storage decisions.
We even use MySQL as a queue for Facebook Messenger. More details about this:
https://www.youtube.com/watch?v=eADBCKKf8PA
https://code.facebook.com/posts/820258981365363/building-mob...
https://githubengineering.com/moving-persistent-data-out-of-...
I want to feel bad for GibLab, but it's really, really hard when they hire people like that.
[1] https://thatgeoguy.ca/blog/2013/12/26/encrypted-backups-in-d...
Quite another to test the backup's restorability.
Most of us back up. Very few test that the backup works as intended. I need to do better in the later.
We should probably treat this issue as something more like a disease like high blood pressure, namely, you don't know you have it, but it is probably doing irreparable damage to your internal organs. If we had no name for the disease or understanding of it we would just die at an earlier age without an obvious cause.
Let's identify the diseases of this sort in our industry and work on prevention, diagnosis and treatment of them instead of just saying you should have been more careful.
Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it damages it !
all the love and support in the world towards the team
Thanks for the support, we've received a lot of kind reactions and are very grateful for them.
Thank you for sharing.
I feel for you greatly here, and I commend your openness about how data restoration caused 6 hours of data loss. I too work in a critical area where even minutes DB lost is bad.
We just had our own test event recently. We make sure that we can fail everything over, and run on all secondaries. I found out how that worked; we failed. The problem with this, is I found out after the fact. Due to the secrecy, not even the teams knew why things failed the way they did. I had to piece it together from disjoint hearsay, and now I believe I have a competent picture.
So yes, when I read your post mortem and RCA, it reminded me greatly of what happened here as well. But we all can learn from your example. As for me, I'm posting it as a throwaway due to likely threats on my job.
All outages are blameless. It's always a process failure or lack of a proper system or tool.
And presumably even if your notes are kept private, it'll still come out in discovery if you end up in court?
What are the risks here? I'd be interested to learn more.
Those companies have huge legal departments writing iron-clad contracts and stocking a lot of very sharp knives for any such eventuality. Smaller companies make for much easier targets.
> even if your notes are kept private, it'll still come out in discovery
Those notes might have been lost by then - that is, if their existence at any given time is even known. It is perfectly reasonable for people working quickly after an outage, to not actually write down every step or observation they make.
> What are the risks here?
Not prison, but you might be forced to pay a bunch if someone manages to get a ruling against you for negligence or the likes.
Doesn't mean it never happens.
Sure plenty of uber larges Incs are "in the cloud" but with mission critical (read: high value worth sueing over) applications and data? That's a pretty small universe.
The ENTIRE Saturn 5 rocket had to be wheeled back to the hangar and the CM dismantled and rebuilt, resulting in a month long delay in the launch.
When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again."
I am willing to bet that Gitlab is now a company that will never slacken off their backup checking in the future.
This same general story appears in different forms with million dollar trading errors, etc. But I have to wonder if it's really that good of a lesson. If you truly take it to heart, you could potentially keep people on the team that are legitimately incompetent which could result in another catastrophic failure.
If a surgeon keeps killing patients because he keeps forgetting to wash his hands, then he needs to be fired.
If a developer skips all company policies and deploys directly to production without a good reason, then he needs to be fired.
There are reasonable mistakes, then there are errors resulting from wreckless actions indicative of a larger problem with the person's view towards their work.
Make the same mistake again, you get fired.
All commercial aircraft have checklists in place, and a two person cross checking system in the cockpit, as well as mechanical means to detect potential problems - but aircraft still every now and then land with their gear still up etc.
Does it make economic sense for an organisation that has invested possibly millions of dollars in training a pilot to immediately sack him/her for a costly mistake like this? Sure, disciplinary action and possible demotion within the ranks is a given, but will you get rid of the one person who will be almost fanatical about never landing with the gear up again?
If they consistently make technical mistake and are sloppy in their airmanship, then sure, fire away. But it is going to be pretty much impossible to design a 'system' that can prevent human errors from creeping in.
I kind of hope the CEO of Gitlab isn't reading HN right now
https://news.ycombinator.com/threads?id=sytse
For every backup system you have, test restores, from scratch, periodically. The more critical the data is to your business, the more frequently and more automated you want those backup checks.
Of course you also want procedures to try to prevent tired admins from deleting production databases. To the greatest degree possible, implement systems to prevent manual tampering with production data; often there's an alternative. But databases can get broken or corrupt without admin interference, so preventing manual database removal or corruption is not an ultimate solution.
Do you have DBAs? Are they completely inept at their job? Don't answer.
Incidents are inevitable and it's important to have a proper RCA/Service-Disruption process in place to handle those.
As mentioned on the other thread, maybe Gitlab doesn't have enough operational/SRE expertise in-house yet, but that was the case in every fast-growing company I've worked for last decade.
This is where automated testing helps. My mail server has a sister VM out in the wild that once a day picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup. If I don't get the message, restoring the backup failed. If the message looks too old then making or transferring the backup failed.
My source control services and static web servers do something similar. None of the shadow copies are available to the world, though I can see them via VPN to perform manual check occasionally and if something nasty happens they are only a few firewall and routing rules away from being used as DR sites (they are slower as in their normal just-testing-the-backups operation they don't need nearly the same resources as the live copies, but slow is better than no!).
This won't catch everything of course, but it catches many things that not doing it would not. The time spent maintaining the automation (which itself could have faults of course) is time well spent if done intelligently. For a system as large in scale as GitLabs then a full restore daily is probably not practical so a more selective heuristic will need to be chosen if you are operating at such a scale. My arrangement still needs some manual checking and sometimes I'm too busy or just forget, so again it isn't perfect, but the risk of making it more clever and inviting failure that way is at this point worse than the risk of my being lazy at exactly the wrong time.
One thing my arrangement doesn't test is point-in-time restores (because sometimes the problem happened last week, so last night's backup is no use) but there is a limit to how much you can practically do.
> The problem of restoring non-existing backups should be treated as a more serious problem in our industry
It is by people that care about it, but not enough people care, and too many people see the resources needed to get it right as an expense rather than an investment for future mental health.
It isn't just non-existent backups. Any backup arrangement could be subject to corruption either through hardware fault, process fault, or deliberate action (the old "they hacked us and took out our backups too" - I really must get around to publishing my notes on soft-offline backups...).
> Let's identify the diseases of this sort in our industry
Apathy mainly.
The people who care most are either naturally paranoid (like me), have lost important data at some point in the past so know the feeling first hand (thankfully not me, though in part to having a backup strategy that worked) or have had to have the difficult conversation with another party (sorry, I can't magic your data back for you, it really is gone) and watch the pitiful expressions as they beg for the impossible.
The only way to enforce the correct due diligence is to make someone responsible for it, it is more a management problem than a technical one because the technical approaches needed pretty much all exist and for the most part are well studies and documented.
Of course to an extent you have to accept reasonable risks. It is usually not practical to do everything that could be done, and understandable human error always needs to be accounted for as do "acts of Murphy". But someone needs to be responsible for deciding what sort of risk to take (by not doing something, or doing something less ideal) rather than them just being taken by general inaction.
This is sensible, but one problem with this is developing "blindness" for messages you receive every day. For some recurring tasks you can automate a bit further. Instead of receiving a success message every day, receive a failure message the first day there's an absence of success message. A number of tools and services exist for setting this up, one is linked in my HN profile (shameless plug ;-)
This can be an issue, I do get lazy about checking them when I'm busy.
The next step will be to automate checking them a bit more, rather than stopping the positive messages. A simple script that for the mail example logs in via POP/IMAP, checks for the last notification and checks the relevant timestamps. Similar for other services (is the last commit in the repo from more than 24 hours ago?). Then I get a simple "all OK" or not. I still have to run the script, perhaps hooking it to CGI and setting it to my browser home page to further remove my laziness from the equation, but I don't want to just be told when something is wrong as I'll never really trust that something going wrong won't block warning messages - I want to make checking as easy as possible though.
I may play with your service and add it as a secondary set of checks though!
Now that I'm developing email stuff I get all the time the existential crisis of "how to I make sure my testing and alerting infrastructure is working?"
Really, how can you be sure your restoration tests are running correctly and their messages get all the way to you?
I think I'll add periodic positive messages for my restoration procedures. Stuff like (if success and day % 23 == 0 then email_something). Still doesn't guarantee the test is correct.
I get a daily message, so if I know if things aren't getting sent or getting through. I've been wary of setting up anything that only sends problem messages because regularly getting the "all OK" messages proves that at least that part of the infrastructure is working.
Getting a message must be mostly a surprise. One daily message is OK for me, but if so it may be good to tell about more than a backup. A daily digest of "everything is up and running" may indeed be something very nice to have.
And how do you ensure a team will always have someone surprised not to get a daily message? Maybe sending for may people, or requiring some acknowledge, and creating a warning over the warning channel if there isn't one. (And you certainly aren't going to send those positive messages over the warning channel, because it requires a completely different kind of action.)
> And how do you ensure a team will always have someone surprised not to get a daily message?
For a team you have a management problem not a technical one! Somebody needs to be responsible for it in the end, plain and simple.
On a less authoritarian note a technical helper might perhaps be to have a screen, always on, showing a dashboard of statuses, where everyone can see it? Make it beep incessantly when a new warning arrives until someone acknowledges it. If you acknowledge a message either deal with it (if you can) or notify whoever needs to be told so they can take action.
People will still sometimes ignore or miss the warnings. Nothing and no one is perfect.
But I'm only dealing with my own collection of personal data and projects here so it isn't that scale of problem - at work I used to be responsible for infrastructure as well as my actual job but we no longer operate at a scale where that is at all practical so I have to trust someone else is dealing with it (though I'm not that trusting I do sometimes take time out to check things myself where I have access).
That's awesome. I've always tested my backups by having shitty, cheap servers so bad that restoring from backup happens about once every few months.
Going to an automated restoration test is a great idea, though also a lot of work for most people.
Just checking, manually or automatically that your backups are occurring and are of a reasonable size is probably sufficient for most operations and would have caught most if not every case in the GitLab instance.
That's why we need awareness days.
Things that aren't tested can pretty much be counted on to fail a non-trivial percentage of the time. If it's going to be business critical, then it needs to be tested. (Lesson hopefully learned here)
By the same token, things which aren't monitored can also be counted upon to fail a non-trivial percentage of the time and even worse, go unnoticed. (Another teachable point here)
...but people being what they are tend to learn these things the hard way unless we've got concrete, real-world incidents like this for them to learn from.
Oh how I deleted the stack? I was using the mobile app and was trying to look at the status of the Cfn stack, but the app was laggy my finger pressed the wrong button... sigh The other interesting thing was I checked the status because the previous night I changed my RDS instance to provisioned IOPS (took 8 hours and failed too). I felt sad and guilty, but at the same time I felt whatever because the upgrade didn't go through so perhaps this accident was all meant to be....
Doubly ouch that it seems that there's no confirmation dialog with a 5-second countdown before you can hit Yes, or whatever.
E.g. forcing someone to manually type the characters D E L E T E before allowing deletion of something potentially important.
Either that or everything should have multiple levels of undo. Everything.
So if you've got a dud client implementation then you're going to lose the check.
One way to do stuff like this is to have separate roles for read-only and read-write access. I pay a lot more attention to what I'm doing on the rare occasions I assume permissions to change things.
Unauthenticated MongoDb on default port? The likelihood of a person port scanning the entire web just doesn't strike as anything adverse, and then, someday someone does exactly that. I guess this is one important reason to bring at least some experienced people on board because there is a likely chance that they can give a better perspective on seriousness of such issues.
I think easy, good server-level backup software would be great. The problem is that enterprise servers are usually highly customized, part of a large, unique architecture, and contain a lot of proprietary, confidential, and potentially personal, legally protected data. That makes it a lot harder to get a one-size-fits-all backup solution set up, which means that the onus of reliable backups will, of necessity, rest upon the company's administrators.
It is very sad when no one checks backups. This bites companies every day and it's usually easy to sympathize, but there's no excuse for it. GitLab needs to perform a serious review of its processes.
I've checked out the GitLab job listings that get published on HN Who's Hiring and other places regularly (I'm currently 100% remote with my current employer and like to track other 100% remote employment opportunities). They have a salary calculator/estimator and personally, I was really underwhelmed with the values it would put out. That calculator makes a city price index adjustment from the base salary and contains a statement that says GitLab prefers to hire people who live in less expensive cities. I also remember feeling that their interview process sounded a little overbearing.
It may be time for GitLab to consider upping the ante on its recruitment procedures and adding some more experienced people to the ranks.
My Time Machine backups in the past have been missing gigabytes of data, without me telling anything about it. And not just volatile data, like temp files or caches, but photos, music and documents.
[1]: http://osxdaily.com/2012/01/21/compare-time-machine-backups-...
You might end up disappointed if you do.
[0] https://kb.vmware.com/kb/1013628
Of course, backing a whole platform up is more complex, and things like databases normally require custom scripting (dump -> backup dump, eg. pg_dumpall | borg create ... -).
The CEO should laugh with us a bit and be proud to inspire a day be named after their hiccup, thanks to their transparency.
I'm totally compassionate but we should never lose our sense of humour!
The CEO should laugh with us a bit and be proud to inspire a day be named after their hiccup, thanks to their transparency.
I'm totally compassionate but we should never lose our sense of humour!
Ouch. I can't even imagine how that feels. This is why even despite monitoring and paging scripts, I still have an event to check my company's backups weekly. Now I don't feel so paranoid.
No hidding, no eupheminization, their live doc stream actually made me question what things did I do on my systems. Looks like convergent evolution in some parts, like prompt changes.
Thanks Gitlab.
Make a backup, restore to test environment, run checksums, anonymize, release test environment.
That way each and every backup is tested both for integrity and ability to rebuild a working environment from it.
In my practice insufficient backup is still (unfortunately) a very common occurrence.
On another note: just having stuff stored with triple replication in the cloud is emphatically NOT a backup.
And it also helps if the same people that have access to the live environment do not have write access to the backups, but that's only feasible past a certain team size.
Everyone is talking about backups, but why not about this? How is it even possible to delete the production database by accident? Why does he have SSH access there? Why do they test their database replication in production? Why are they fire-fighting by changing random database settings _in production_?
I know that all of this is common practice. I am questioning it in general.
Because when something goes wrong he has to be able to discover what it was.
> Why do they test their database replication in production?
Many places do that. With free databases there is simply no reason to, but the practices are inherited from the non-free databases best practices arsenal.
People, if you are using postgres and have a cluster in production, make sure you also have the script that creates this cluster in a VCS and is able to create an equal cluster on virtual machines in your computer.
> Why are they fire-fighting by changing random database settings _in production_?
Oh, that's because they are fire-fighting. The problem is that when you get a problem you know nothing about, you don't really know what to replicate in another environment to reliably test things there. The most you can do is verify that your changes aren't harmful, the real test is always in production.