Backups every 24 hours? I would have expected hourly. This is a experience thing, when doing things with data (like wiping db directories) I mv a directory rather than rm it. If you are working on a problem for hours you are likely to make a simple critical mistake like that which then creates a whole new problem to solve.
Best of luck though and kudos for the transparency!
a) YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com
b) YP terminates the removal, but it’s too late. Of around 300 GB only about 4.5 GB is left
Nightmare material. Are there any known cases of DBAs or developers suffering a heart attack due to this sort of thing?
8 comments
[ 1.9 ms ] story [ 32.9 ms ] threadThis is the best communicated outage that I can remember seeing.
Best of luck though and kudos for the transparency!
a) YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com
b) YP terminates the removal, but it’s too late. Of around 300 GB only about 4.5 GB is left
Nightmare material. Are there any known cases of DBAs or developers suffering a heart attack due to this sort of thing?
I'm pretty sure if it was one of our older database people, they would have dropped dead. They were visibly shaking.
Always test yur backups automatically and fail the backup job if the restore fails, because DR/BCP is no guden without viable customer data.