To add to this: having the backups copy somewhere that has write-only permissions can be a life-saver in a crypto-ransom malware situation. Hackers are smart enough to delete or encrypt backups, and this is made far easier for them when admins have full read/write permissions to the backup location.
Offline backups are very important - not only because they can't be modified because they're disconnected, but because they force you to spend time going and getting them and that time can be important. Many backups are successfully destroyed in trying to back them up, and spending some time to think about what is happening can be worth it.
rsync.net is great for this. They keep ZFS snapshots, so even though you have read/write access you will always have the snapshots stored. So you will be safe at least as long as you notice the corruption within the normal 7-day rolling window (there are options for other windows or manual snapshots too IIRC).
This should also lead to the question of how up-to-date your backups need to be and is your solution setup for that (is it 1s, 1m, 1h, 1d, etc.)? Define an actual SLA for how much data you can loose and test for that.
Does "write-only permissions" somehow imply they can't overwrite the backup with an encrypted copy? I don't understand how read permissions would allow them to delete or replace something.
I think the OP means something more like "immutable" or "copy on write", so you would never modify or delete previous data (or at least separate the modifications/deletions to a more-secure system).
At the desktop level, this would result in a whole new class of useless support calls.
At the SOHO server level, this might be useful. But the alternatives are already available: unmount the disk, or unmount and unplug it, or unmount, unplug, and carry over to the safe, or unmount, unplug, and carry to the backup site...
At the datacenter level, nobody has time to physically touch things.
The solution for this that I've been working with is having an S3 bucket with versioning enabled.
The systems that send backups to that S3 bucket don't have permissions to modify previous versions. If one or all of those systems are compromised then they can't overwrite previous backups because the original data will still exist in old versions.
Do you know of a good app for doing checklists of this type? Many TODO-list apps market themselves as "checklist apps" but they really just mean "yeah we've got checkboxes". They don't mean "our app lets you choose from a library of checklists for common tasks and pull a fresh one to tick off when you want to perform one of those tasks."
I'm looking for similar functionality but more IT-oriented (or at least generic, not referring to airplane models, etc.) and preferably with an offline desktop "client" (it is especially critical that you have your checklist library available if the whole corporate network is down, for example) or, barring that, at least a desktop-friendly web UI.
Honestly, if I knew there was a market for this, I would build it. It's either a great opportunity or the market isn't big enough.
I was a flight instructor and am now a software engineer, plus I've made a daily checklist app. So I think I'm at least qualified to throw out some opinions on this.
It's not a software engineering challenge. The minimal features are textbook basic CRUD. The essential UI will be well-supported by any framework. Even the challenges when you branch out are solved problems. Device sync, sharing, backups, etc all have some off-the shelf solution or are solved in another space you can look at.
It's a product design / UX / whatever you want to call it problem. Within the set of choices that are easily available to you in the software engineering world, you have to make the right ones! Most people won't naturally make good checklists. How do you help them? One of the biggest (maybe single biggest) issues with checklist is using them at all. How do you get compliance up? How do you handle "skipping" items? Do you go with read/do or do/confirm? How do you keep checklists up-to-date (critical for emergency checklists). It would be very easy to get this stuff wrong.
It also hits a weird spot in the market. What would people pay for this? I don't think you hit enough users for "free" and then my guess is it's a really weird curve with some people paying $1 and then a minority paying like a lot ($100?). I don't know, that's a total guess. Just bringing this up to cover "why isn't a company already doing this?".
Not to be discouraging! Just don't take it as a software engineering challenge. If you're experienced and working on your home turf, it will be easy. The hard thing will be making the right easy thing. Best way to do that is to solve your own problem btw. Maybe start with a "I crashed the server" checklist or something.
If anyone wants to take a stab, the essay that the book "The Checklist Manifesto" is based on would be a good start. The book itself is a bit verbose, but would be worth it if you get serious. I'd also chat for 30 minutes or answer random questions for anyone who's going to make something. My email is my last name at gmail.
Couldn't have put it better myself. As usual, the hard part isn't building software, it's building software that people want and finding a way to get them to pay you for it.
It's Apple only but I use a grocery list app called Grocery (which uses the Apple Reminders database) and it remembers items, allowing you to easily restore things you've check off previously to an unchecked status.
I've thought of the same idea. The rough workflow for creation would be (1) create a checklist, (2) publish it to the world as a template, (3) let others fork it, send PRs, etc. Then the usage workflow is (1) search for a template, (2) instantiate your own copy, (3) use it. As you say, you'd want the list and the checkbox states available offline, and maybe synced elsewhere.
Throw in collaboration features like sharing a single checklist ("are we bringing everything we need for the roadtrip? mark items that you've packed" "someone please buy milk, or else I will if it's still on the list when I stop at the grocery store tonight," "has everyone grabbed a task for the next sprint?"), or delegating items to other people ("for tomorrow's presentation, make sure someone has set up the projector"), and it could add the magic that I felt back in 2009 when I realized how amazing git became once GitHub was layered on top of it.
The reusable-list idea works for me when I'm traveling. I have a list of stuff I need to do when I leave the house -- turn off the thermostat, make sure the plants are watered, etc. -- and I find it annoying to have to uncheck the old list's items in Google Keep. I'd rather keep the old checked lists and instantiate new ones. The list itself evolves over time, so having the old completed (or not-quite-completed) lists is interesting.
I have a small sticky note pad on my desk. When we have a meeting about features, I jot down my todos. In some cases, I may do a screengrab, or submit a bug report (to myself), but, most times, it’s the PostIt.
When I’m done with a task, I crumple the note, and drag it into the trash can.
And if it's really important, have one person doing what it says in the checklist, and another reading what's to be done next, making sure the first person did it, and _then_ ticking it off. And finally storing the dated checklist in a paper file somewhere
The number of times I've missed a step on a paper checklist because I was on autopilot is ... non-zero
A good trick to convince oneselves to backup and the cost of redoing the work lost.
Sometimes, just a couple of pages of documents, or a setup script, is worth weeks of work, ressources, money, etc. And it's nothing compare to the costs of creating multiple backups.
In an early startup, where developers are also doing ops, and there are periodic manual tasks (like daily/weekly/monthly running certain kinds of backups, eyeballing AWS dashboards and logs to check for anything alerts missed, etc.), using the same board and issue-tracker you use for development tasks helps.
In one company, I used GitLab Board, and had special tags for the recurring and background tasks, which were sticky in Kanban columns right below the generally one-time development tasks. The deadline dates for recurring tasks kept changing to the next time the task had to be done. The Issue description had the concise immediate instructions for the steps of the task (with a link to the wiki for related and background info). And every time you did the task, you added a very quick Comment with any notes, and bumped the due date.
It was very lightweight, and fit the simplicity I was advocating (e.g., the GitLab Kanban board says what everyone is doing, all info goes into GitLab in one form or another, minimal other tools and silos to juggle, etc.).
Yes, it's usually a column-based system for tracking progress of tasks that move through various stages (eg. planning, development, testing, rollout, evaluation).
Exactly. Sometimes my columns aren't waterfall phases, but more: Backlog, Blocked, Active, Done, Abandoned.
The "Blocked" is important to highlight on the board: when someone can't do their thing, no sense floundering quietly -- highlight that, see who can unblock it, and what anyone else can do in the meantime.
"Abandoned" is another one that I think is important, but might be unusual. If the task no longer makes sense, stop immediately, move it to Abandoned, and move on to a more worthwhile task. Just don't abandon anything quietly; acknowledge it on the board, communicating to everyone immediately we're not doing that anymore. Later, we can reflect on what info we had, when, and what we can learn from that.
(I usually really-really want a Gantt model as well, on the same database that the board view is coming from, but the tools aren't quite there yet.)
> Just don't abandon anything quietly; acknowledge it on the board, communicating to everyone immediately we're not doing that anymore.
Even on personal projects I try to not remove TODO's or tasks without writing why it is no longer a goal. The amount of knowledge that is thrown away in many organizations or by people by discarding tasks without saying why is astounding, the why for discarding it can be as productive as a task completed.
We use Google Calendar, and it is generally the best solution in my experience. But not just a daily event; an actual time, with a meeting invite that anyone relevant joins, to run each task together. It helps knowledge transfer and helps guarantee that it happens. Documentation, like with most meetings, is thrown into Notion or whatever; just quick and dirty.
Example: test our database backup restore process. Everyone hops on; leader logs into the cloud provider; confirms that the backup strategy is what we expect (which is documented in the description of the meeting invite); confirms that database backups exist in line with that policy; confirms they're of a reasonably expected size; we click one for staging -> restore to a new deployment -> wait for the cloud provider to set everything up, use that 5 minutes to talk with one another about stuff -> set up the database username/password; log in to AWS Secrets Manager -> update credentials -> roll the deployment to pull in the new credentials; confirm everything looks good; and tear the old (again: staging) deployment down. A second person on the call is tasked with documenting everything we did, taking screenshots of the screen share, and getting it all into a wiki. For us, it takes less than a half hour, do it every quarter or so, usually scheduled for a Friday afternoon where we can also talk about weekend plans and enjoy some drinks.
Eyeballing dashboards is a harder one; a similar thing we do, similar to the last example, is more of a Performance Review, which is bi-weekly. We've got a collection of dashboards, database index metrics, etc; we all get together for a half hour and review all of them similarly. If anything looks unacceptable, we generally backlog two tickets: one to adjust our Slack/Datadog alerts for it so the next time we don't find out during the review, and another to remediate.
If you are backing up users data that is not yours to own and do with as you please, yes you absolutely can have to many backups. If you are required by law or policy to erase user data and it's in your backup(s), you now have a real problem on your hands.
Note: Sometimes the law/policy will make a carve-out exception for backups, if so, yay for you! I'm not currently aware of a law that does this.
Essentially yes. I can't know all laws for all jurisdictions(nor would I want to), but laws could make a carve-out provision. I'm not currently aware of any that matter in my jurisdiction(s). That doesn't mean there aren't some out there somewhere.
Policies are of course a different thing, and they may or may not apply to you, from industry regulations or cyber security insurance or just company or trade group policies that might apply.
If you decide to go down this route, you should bear in mind that other supervisory authorities might be stricter and that you must be able to demonstrate that it’s impractical to delete backup data.
This will require, at the very least, a risk assessment, business impact assessment and data protection impact assessment.
There's two aspects to this 1) Holding backups for too long that eventually contain user data you no longer need and 2) backing up user data that you no longer needed in the first place.
Working in security I have been advising clients to make sure that they consider their obligations for retention periods. Not just so that you release user information you no longer need at the point you no longer need it, but also to reduce litigation if there were to be a data breach.
Early in one of my companies, we had a cloud hosted database provider that only provided (at our meager level of spend) 7 days of daily backups for the main db. We had a specific customer, one of maybe ten, and a higher percentage of our revenue, who made an big bulk import oopsie-whoopsie; and given the specifics of this software product, did not notice for weeks. They wanted everything back to the way it was.
We ended up being saved by our traditionally "crappy" local dev process; me and the other dev had copies of the prod db on our development machines. And, naturally, we rarely updated them; the last time I had sync'ed it down was, conveniently, about four weeks ago. So they had some data loss between that "backup" and the oopsie-whoopsie, but far less major than their issues with the current state.
The most interesting thing to really come out of that was a set of tooling to precision restore an individual customer's data from a backup of this multi-tenanted database. That was something I see as pretty rare for engineers to consider in their backup strategy; you may have the backup, it may be easily restored, but it will take the world with it and maybe you don't want the whole world. We ended up using that a half dozen or more times over the next few years... against a far more comprehensive set of backups of course. Yeah, maybe it makes sense to store a history of every change made to every row; but for our needs it made a lot more sense to leverage backups and some glue scripts.
One of the important considerations of database design is “what might I want to restore” and there are often way more answers than just “how it was at a point in time”.
And some of those restores should happen in the application layer not the db backup.
100% of my restores have been due to user/developer fuckups.
0% has been because of hardware failures.
This is why I love that PostgreSQL backup files are plain text files. It is so easy to extract a single table (or parts of a table) using a simple perl/python/whatever script.
If you're going to pay for 7 backups I wonder if it'd be better space them out a bit. You could keep 1x 12 hourly, 2x daily, 2x weekly, 1x monthly etc. Or just use some form of event sourcing so your DB becomes a materialized view you can restore. These days devs often aren't allowed to have copies of sensitive customer data.
Once upon a time, a Japanese megacorp was designing a new factory. The master plans for the factory were stored on a USB thumb drive owned by the Chief Engineer, who was the only person authorized to make changes to the master plan.
The Chief Engineer and crew went to inspect another factory for inspiration. There was a direct flight to the city, but by corporate policy they had to use the preferred airline, requiring a transfer. The Chief Engineer didn't want to lug his heavy laptop around, so he checked it in, with the thumbdrive still plugged in.
Chief Engineer and crew arrived successfully at the factory. His suitcase did not. Enquiries with the airline told them the suitcase had been shipped to the wrong place and would be returned very soon. And that was the last they or the airline ever saw of it. There was no central server, no versioning, no backups -- the plans were now gone.
Fortunately, since this was a Japanese company, the plans were printed out weekly for review at the status update meeting. So the plans were manually reconstructed from paper printouts, page by page, line by line, character by character, and one of the many, many people typing it in was the friend who told me this story. Unsurprisingly, they no longer work at Megacorp.
Something about that story reminds me of toy story being deleted and a single animator had taken a copy of the _entire_ project home to work on it from home.
Another good one is Maersk. When they suffered a ransomware attack that deleted all access to all backups/replication servers, the only surviving backup was a replication server that happened to be offline during the attack, because it was situated in a country with very unstable internet and power supply.
> the Chief Engineer was the only person authorized to make changes to the master plan.
> the plans were printed out weekly for review at the status update meeting.
This is so quintessentially Japanese. Have a single point of failure in a rigid chain of command structure, but bake in a ton of paper redundancy. Nonetheless, I bet the lost laptop was an extremely advanced model.
In the early 2000s, my dad worked with some Japanese collaborators. They brought over some impossibly thin laptops that seemed a decade ahead of what you could get in the US at the time…and proceeded to use them to send faxes.
I got my sysops backups monitored in Nagios. A script that runs in NSClient++ checks the backup date, size, and the backup log for errors. Still, you also need to inspect the backup manually from time to time - just to be sure.
- write all commands, ever entered to any shell, down. Of course, this also includes the procedure to create backups
- do not treat all data equal. There's 'fast' (but small) data that needs more backups more often and there is 'slow' (but large) data that is infrequently updated, thus less often backed up.
- 3-2-1 rule is a minimal setup; extend it wherever possible. Also, use 'migration periods' to replace one backup system with another; during this period, you use both backup systems until you gained enough trust in the new one to deprecate the old one. This should include actually testing restoring systems from backups in VMs at least once.
My private 3-2-1+:
- immediate sync of files to my internal nextcloud server, hosted offsite. That means any files exist at least once in to different locations
- daily ZFS automatic snapshots at the server side and deprecation of old ones
- weekly, my server starts another Odroid CS2 at another offsite location and rsync's any changes; the Odroid is shutdown afterwards
- yearly, I attach a 16TB usb drive and send ZFS snapshots.
- in addition to these 'basic' 3-2-1, the '+' means: daily, Nextcloud's document folder is duplicated to a different ZFS dataset on the server which is not actively write-referenced in any VM; Also, for some very important files, I keep more versions: password manager, dotfiles, config files etc. are a) versioned in my private Gitlab, also b) Nextcloud will keep versions and c) some of these files are synced to my phone; I also print out the most precious keys yearly and store those at a 'secure' location
- here's a special tip: Backup your OTP codes! I realized this recently: Google Authenticator will not create backups. You can migrate your OTP codes to Aegis Authenticator [1], which allows you to create automatic, encrypted backups that can be automatically synced using Nextcloud
I’ve taken to keeping as much of my info as possible in flat text files and flowing that into git. That in turn is easy to replicate and protect against ransomware and the like
63 comments
[ 4.4 ms ] story [ 130 ms ] threadThis should also lead to the question of how up-to-date your backups need to be and is your solution setup for that (is it 1s, 1m, 1h, 1d, etc.)? Define an actual SLA for how much data you can loose and test for that.
I have no idea why major drive users don't demand this from the makers.
At the SOHO server level, this might be useful. But the alternatives are already available: unmount the disk, or unmount and unplug it, or unmount, unplug, and carry over to the safe, or unmount, unplug, and carry to the backup site...
At the datacenter level, nobody has time to physically touch things.
Besides, for the bozos, ship the drives with the write-enable turned on. They won't notice the difference.
The systems that send backups to that S3 bucket don't have permissions to modify previous versions. If one or all of those systems are compromised then they can't overwrite previous backups because the original data will still exist in old versions.
Another comment mentioned an offline backup. And I agree. But it's also easy to go from having a month old offline backup to a year old to...
One that seems close is ForeFlight (demo: https://www.youtube.com/watch?v=d05_ZQMm6Zo&t=23s), but it's just one feature of a much larger app, is mobile only, and obviously targeted at aviation.
I'm looking for similar functionality but more IT-oriented (or at least generic, not referring to airplane models, etc.) and preferably with an offline desktop "client" (it is especially critical that you have your checklist library available if the whole corporate network is down, for example) or, barring that, at least a desktop-friendly web UI.
Honestly, if I knew there was a market for this, I would build it. It's either a great opportunity or the market isn't big enough.
It's not a software engineering challenge. The minimal features are textbook basic CRUD. The essential UI will be well-supported by any framework. Even the challenges when you branch out are solved problems. Device sync, sharing, backups, etc all have some off-the shelf solution or are solved in another space you can look at.
It's a product design / UX / whatever you want to call it problem. Within the set of choices that are easily available to you in the software engineering world, you have to make the right ones! Most people won't naturally make good checklists. How do you help them? One of the biggest (maybe single biggest) issues with checklist is using them at all. How do you get compliance up? How do you handle "skipping" items? Do you go with read/do or do/confirm? How do you keep checklists up-to-date (critical for emergency checklists). It would be very easy to get this stuff wrong.
It also hits a weird spot in the market. What would people pay for this? I don't think you hit enough users for "free" and then my guess is it's a really weird curve with some people paying $1 and then a minority paying like a lot ($100?). I don't know, that's a total guess. Just bringing this up to cover "why isn't a company already doing this?".
Not to be discouraging! Just don't take it as a software engineering challenge. If you're experienced and working on your home turf, it will be easy. The hard thing will be making the right easy thing. Best way to do that is to solve your own problem btw. Maybe start with a "I crashed the server" checklist or something.
If anyone wants to take a stab, the essay that the book "The Checklist Manifesto" is based on would be a good start. The book itself is a bit verbose, but would be worth it if you get serious. I'd also chat for 30 minutes or answer random questions for anyone who's going to make something. My email is my last name at gmail.
Perfect for do-check lists.
[0]: https://apps.apple.com/de/app/daily-checklist-checkyourlist/...
Throw in collaboration features like sharing a single checklist ("are we bringing everything we need for the roadtrip? mark items that you've packed" "someone please buy milk, or else I will if it's still on the list when I stop at the grocery store tonight," "has everyone grabbed a task for the next sprint?"), or delegating items to other people ("for tomorrow's presentation, make sure someone has set up the projector"), and it could add the magic that I felt back in 2009 when I realized how amazing git became once GitHub was layered on top of it.
The reusable-list idea works for me when I'm traveling. I have a list of stuff I need to do when I leave the house -- turn off the thermostat, make sure the plants are watered, etc. -- and I find it annoying to have to uncheck the old list's items in Google Keep. I'd rather keep the old checked lists and instantiate new ones. The list itself evolves over time, so having the old completed (or not-quite-completed) lists is interesting.
Also see this recent item https://news.ycombinator.com/item?id=32552782 proposing a spec for to-do lists.
1. Type it into a file.
2. Print it onto a piece of paper
3. Put a mark by each item with a pen as you finish it.
I have a small sticky note pad on my desk. When we have a meeting about features, I jot down my todos. In some cases, I may do a screengrab, or submit a bug report (to myself), but, most times, it’s the PostIt.
When I’m done with a task, I crumple the note, and drag it into the trash can.
The number of times I've missed a step on a paper checklist because I was on autopilot is ... non-zero
You can create an issue with a checklist in it by pasting in the following:
If you put that in the issue description itself, checking those off will be reflected in anything else that displays that issue (3/5 complete).And if you want to duplicate a checklist, you can copy and paste in a full block of markdown.
https://www.amazon.com/Checklist-Manifesto-How-Things-Right/...
Sometimes, just a couple of pages of documents, or a setup script, is worth weeks of work, ressources, money, etc. And it's nothing compare to the costs of creating multiple backups.
If your RTO is very short, then full backups with fast restores are best.
In one company, I used GitLab Board, and had special tags for the recurring and background tasks, which were sticky in Kanban columns right below the generally one-time development tasks. The deadline dates for recurring tasks kept changing to the next time the task had to be done. The Issue description had the concise immediate instructions for the steps of the task (with a link to the wiki for related and background info). And every time you did the task, you added a very quick Comment with any notes, and bumped the due date.
It was very lightweight, and fit the simplicity I was advocating (e.g., the GitLab Kanban board says what everyone is doing, all info goes into GitLab in one form or another, minimal other tools and silos to juggle, etc.).
Does it have specific meaning in software engineering? Isn't Kanban just billboard in Japanese?
See here: https://en.wikipedia.org/wiki/Kanban_board
The "Blocked" is important to highlight on the board: when someone can't do their thing, no sense floundering quietly -- highlight that, see who can unblock it, and what anyone else can do in the meantime.
"Abandoned" is another one that I think is important, but might be unusual. If the task no longer makes sense, stop immediately, move it to Abandoned, and move on to a more worthwhile task. Just don't abandon anything quietly; acknowledge it on the board, communicating to everyone immediately we're not doing that anymore. Later, we can reflect on what info we had, when, and what we can learn from that.
(I usually really-really want a Gantt model as well, on the same database that the board view is coming from, but the tools aren't quite there yet.)
Even on personal projects I try to not remove TODO's or tasks without writing why it is no longer a goal. The amount of knowledge that is thrown away in many organizations or by people by discarding tasks without saying why is astounding, the why for discarding it can be as productive as a task completed.
Example: test our database backup restore process. Everyone hops on; leader logs into the cloud provider; confirms that the backup strategy is what we expect (which is documented in the description of the meeting invite); confirms that database backups exist in line with that policy; confirms they're of a reasonably expected size; we click one for staging -> restore to a new deployment -> wait for the cloud provider to set everything up, use that 5 minutes to talk with one another about stuff -> set up the database username/password; log in to AWS Secrets Manager -> update credentials -> roll the deployment to pull in the new credentials; confirm everything looks good; and tear the old (again: staging) deployment down. A second person on the call is tasked with documenting everything we did, taking screenshots of the screen share, and getting it all into a wiki. For us, it takes less than a half hour, do it every quarter or so, usually scheduled for a Friday afternoon where we can also talk about weekend plans and enjoy some drinks.
Eyeballing dashboards is a harder one; a similar thing we do, similar to the last example, is more of a Performance Review, which is bi-weekly. We've got a collection of dashboards, database index metrics, etc; we all get together for a half hour and review all of them similarly. If anything looks unacceptable, we generally backlog two tickets: one to adjust our Slack/Datadog alerts for it so the next time we don't find out during the review, and another to remediate.
Just curious - how often do you test this?
I write an MD5 checksum for each file so they can be verified at any time in the future.
Note: Sometimes the law/policy will make a carve-out exception for backups, if so, yay for you! I'm not currently aware of a law that does this.
I am confused by these sentences. Sometimes the law does it, but you're not aware of a law that does it?
Policies are of course a different thing, and they may or may not apply to you, from industry regulations or cyber security insurance or just company or trade group policies that might apply.
If data is on backup tapes you seem to be able to inform them of how long your retention policies are.
If you decide to go down this route, you should bear in mind that other supervisory authorities might be stricter and that you must be able to demonstrate that it’s impractical to delete backup data.
This will require, at the very least, a risk assessment, business impact assessment and data protection impact assessment.
Working in security I have been advising clients to make sure that they consider their obligations for retention periods. Not just so that you release user information you no longer need at the point you no longer need it, but also to reduce litigation if there were to be a data breach.
Until a court or politician says otherwise, "Deleting data" means inaccesible, not physically destroyed.
AFAIK GDPR, CCPA don't specify deletion means physical destruction.
You can kill one or one-thousand backups with the same effort.
We ended up being saved by our traditionally "crappy" local dev process; me and the other dev had copies of the prod db on our development machines. And, naturally, we rarely updated them; the last time I had sync'ed it down was, conveniently, about four weeks ago. So they had some data loss between that "backup" and the oopsie-whoopsie, but far less major than their issues with the current state.
The most interesting thing to really come out of that was a set of tooling to precision restore an individual customer's data from a backup of this multi-tenanted database. That was something I see as pretty rare for engineers to consider in their backup strategy; you may have the backup, it may be easily restored, but it will take the world with it and maybe you don't want the whole world. We ended up using that a half dozen or more times over the next few years... against a far more comprehensive set of backups of course. Yeah, maybe it makes sense to store a history of every change made to every row; but for our needs it made a lot more sense to leverage backups and some glue scripts.
And some of those restores should happen in the application layer not the db backup.
0% has been because of hardware failures.
This is why I love that PostgreSQL backup files are plain text files. It is so easy to extract a single table (or parts of a table) using a simple perl/python/whatever script.
There is even a tower-of-Hanoi tape rotation scheme:
https://www.techtarget.com/searchdatabackup/tip/An-introduct...
* https://en.wikipedia.org/wiki/Backup_rotation_scheme#Tower_o...
3 recent copies of important data, in 2 different media formats, with 1 copy off-site
The Chief Engineer and crew went to inspect another factory for inspiration. There was a direct flight to the city, but by corporate policy they had to use the preferred airline, requiring a transfer. The Chief Engineer didn't want to lug his heavy laptop around, so he checked it in, with the thumbdrive still plugged in.
Chief Engineer and crew arrived successfully at the factory. His suitcase did not. Enquiries with the airline told them the suitcase had been shipped to the wrong place and would be returned very soon. And that was the last they or the airline ever saw of it. There was no central server, no versioning, no backups -- the plans were now gone.
Fortunately, since this was a Japanese company, the plans were printed out weekly for review at the status update meeting. So the plans were manually reconstructed from paper printouts, page by page, line by line, character by character, and one of the many, many people typing it in was the friend who told me this story. Unsurprisingly, they no longer work at Megacorp.
Another good one is Maersk. When they suffered a ransomware attack that deleted all access to all backups/replication servers, the only surviving backup was a replication server that happened to be offline during the attack, because it was situated in a country with very unstable internet and power supply.
> the plans were printed out weekly for review at the status update meeting.
This is so quintessentially Japanese. Have a single point of failure in a rigid chain of command structure, but bake in a ton of paper redundancy. Nonetheless, I bet the lost laptop was an extremely advanced model.
In the early 2000s, my dad worked with some Japanese collaborators. They brought over some impossibly thin laptops that seemed a decade ahead of what you could get in the US at the time…and proceeded to use them to send faxes.
- write all commands, ever entered to any shell, down. Of course, this also includes the procedure to create backups
- do not treat all data equal. There's 'fast' (but small) data that needs more backups more often and there is 'slow' (but large) data that is infrequently updated, thus less often backed up.
- 3-2-1 rule is a minimal setup; extend it wherever possible. Also, use 'migration periods' to replace one backup system with another; during this period, you use both backup systems until you gained enough trust in the new one to deprecate the old one. This should include actually testing restoring systems from backups in VMs at least once.
My private 3-2-1+:
- immediate sync of files to my internal nextcloud server, hosted offsite. That means any files exist at least once in to different locations
- daily ZFS automatic snapshots at the server side and deprecation of old ones
- weekly, my server starts another Odroid CS2 at another offsite location and rsync's any changes; the Odroid is shutdown afterwards
- yearly, I attach a 16TB usb drive and send ZFS snapshots.
- in addition to these 'basic' 3-2-1, the '+' means: daily, Nextcloud's document folder is duplicated to a different ZFS dataset on the server which is not actively write-referenced in any VM; Also, for some very important files, I keep more versions: password manager, dotfiles, config files etc. are a) versioned in my private Gitlab, also b) Nextcloud will keep versions and c) some of these files are synced to my phone; I also print out the most precious keys yearly and store those at a 'secure' location
- here's a special tip: Backup your OTP codes! I realized this recently: Google Authenticator will not create backups. You can migrate your OTP codes to Aegis Authenticator [1], which allows you to create automatic, encrypted backups that can be automatically synced using Nextcloud
[1]: https://f-droid.org/en/packages/com.beemdevelopment.aegis/
Big media files are a bit trickier