Wow that really sucks. I'm sorry this happened to you guys. Hopefully you've got a backup on a USB somewhere. If not, all the best with working on a brand-new site! =)
Production can push as many backups as it wants, it shouldn’t be able to overwrite the previous backups. Deletion and expiry has to happen separately. There’s no reason for it to even be able to list the backups and that restriction combined with a randomized backup ID (in addition to any date/time tagging) limits the ability to corrupt the backups.
Seems to me it would be better to have the backup pull from production so the scripts cannot be reverse engineered. Maybe it's just 6 in one and a half dozen in the other though.
This is a pokemon fansite that's old enough to drink that was just running on xenforo and wordpress. There probably is no "production vs staging" or immutability or ZFS snapshots, or large amounts of engineering going on.
They had offsite backups, so I'm just saying however permissions are managed, they should have had them reduced.
Idk if they were just pushing to object storage on AWS/Azure/GCS or had another server, but my comment stands.
Now, do I blame them? No, like you said it was a fansite. I'm just offering advice on how to avoid this sort of thing in the future for them and sites like them.
I'm kind of curious though - what can you do as a non-technical user to prevent this? Let's say you're on shared hosting - you don't know what object storage is, and you upload your PHP themes via FTP (or at best, FTP+implicit SSL, but the default is plaintext).
Your backup is probably downloading something over FTP, or creating a backup in cPanel/DirectAdmin/whatever shared hosting panel.
Your host doesn't allow SSH. This is the hypermajority of shared hosting.
(My apologies, I didn't mean that directed toward you, it was more of a general "this is HN, not everybody will self host GPG backups on immutable redundant cloud storage")
Hmm, in this case I might have a local script running on a cronjob or Scheduled Task to FTP(S) into the server and copy the files on the server locally. You can run this on a desktop or a raspi if one knows how to use it.
It sounds from the blog like they already had automated off-sites, which makes me feel like they do have (perhaps limited) SSH access? Maybe it's just a cPanel deal though.
No troubles, didn't take it personally and hope I didn't come off that way :)
I remember as a teenager subscribing to dirt cheap hosting that only allowed uploads via a portal (:S) and trying to upload phpbb file by file, so I get it ahaha... passion often outstrips means/ability :P
Wow, pokemon cards as ransom, that's a new one for me. The post doesn't say how much later the message was sent, so it could possibly be a troll who noticed the website was missing data. But still that's quite a story. Sorry that happened to you guys.
>The team first reverse-engineered our backup script to delete all of our offsite backups.
I guess the permission should be one way. The production server can only push data to backup server, never to delete or overwrite anything even if it wanted to.
what software introduced the vulnerability? it seems sorta odd to write an article about this and not warn others how to prevent it or document exactly what went wrong
21 comments
[ 2.9 ms ] story [ 42.0 ms ] threadIt's not a panacea, because hackers will be able to push arbitrary things, but you're greatly reducing your surface.
*Please point out how wrong this is.
https://docs.aws.amazon.com/whitepapers/latest/aws-security-...
Further, we also use Veeam to backup Linux servers.. a high level read on how it works with immutable storage:
https://www.veeam.com/blog/v11-immutable-backup-storage.html
Idk if they were just pushing to object storage on AWS/Azure/GCS or had another server, but my comment stands.
Now, do I blame them? No, like you said it was a fansite. I'm just offering advice on how to avoid this sort of thing in the future for them and sites like them.
Your backup is probably downloading something over FTP, or creating a backup in cPanel/DirectAdmin/whatever shared hosting panel.
Your host doesn't allow SSH. This is the hypermajority of shared hosting.
(My apologies, I didn't mean that directed toward you, it was more of a general "this is HN, not everybody will self host GPG backups on immutable redundant cloud storage")
It sounds from the blog like they already had automated off-sites, which makes me feel like they do have (perhaps limited) SSH access? Maybe it's just a cPanel deal though.
No troubles, didn't take it personally and hope I didn't come off that way :)
I remember as a teenager subscribing to dirt cheap hosting that only allowed uploads via a portal (:S) and trying to upload phpbb file by file, so I get it ahaha... passion often outstrips means/ability :P
The paypal switcheroo was dumb too
I guess the permission should be one way. The production server can only push data to backup server, never to delete or overwrite anything even if it wanted to.