64 comments

[ 2.9 ms ] story [ 116 ms ] thread
Overall, cool work! Especially the search space reduction part.

I haven't dive into the code, but only 818000/60=13633 attempts per second for 64 SHA-256 rounds plus one AES-256 decryption on a 2080 doesn't sound right. Learn some GPU and tuning the code can likely increase the throughput a lot.

Mind you, that's only 25x naive Python implementation on (supposedly) 1 core.

Also, hashcat does >1M hash/s for even higher number of SHA256 iterations, and hundreds of million attempts per second on single block AES-256.

Edit: An optimized version of this is very likely deployable on consumer-grade hardware. May still not very useful due to the forensics requirement, though.

Thanks! I wonder if the Python number is correct, I remember Python being prohibitively slow in comparison. But assuming it is:

There are 256 sha256 iterations on average, so the number is a bit better - but there's probably still a lot to improve (it's much more optimised than the naive version, but it was written by reverse-engineers, not GPGPU specialists). The PoC was also opensourced [0], it would be great if someone experienced could spot any obvious problems.

One major issue was that the key scheduling is basically

    do {
        key = sha256(key)
    while (key[0] != '\0')
So on average there are 256 SHA256s per key, but worst case it takes thousands of iterations. Coding this in a GPU-friendly way was non-trivial, and there are still some GPU cycles wasted.

> Edit: An optimized version of this is very likely deployable on consumer-grade hardware. May still not very useful due to the forensics requirement, though.

I think speeding the code three or four orders of magnitude more would go a long way towards pracical usage. I think the biggest issue was getting TID and precise enough time range. Brute-forcing a suspected TID values and widening the time search range would help a lot.

Disclaimer: I worked on that research, but I'm not an employe anymore.

[0]: https://github.com/CERT-Polska/phobos-cuda-decryptor-poc

One other thing. The article states pythons performance at keys-per-second, then all the other numbers are given in keys-per-minute. That means that pythons number looks really small in comparison, but if we take it (500 kps) and multiply it by 60, we get 30k keys-per-minute, or about 50% faster than the first CUDA baseline.

Now I'm quite fond of python, but largely I read this as the CUDA implementation having quite some room for improvement. Having almost no CUDA experience of my own, that is just a hunch, which I'm glad rfoo supports with (surely) a lot more experience than me.

I can't offer "four orders of magnitude", but "four" ;-)

PIDs on windows are always a multiple of four.

(I didn't see that mentioned in the blog post, but didn't check the code so maybe that's already in there.)

Edit: nvm I guess that's why it says 2^30 not 2^32...

phobos is not a moon !!!

phobos is the god and personification of fear and panic in Greek mythology.

this is crucial in understanding ransomware ;)

I imagine it's the root of the term "phobia" in this way.
It is, yeah.
This little thread was actually more interesting/informative than the linked article
Nearly all my personal photos were encrypted by the helprecover@foxmail.com ("HELP") variant of Phobos. I've been holding onto the encrypted copies for a while in hopes that some people were working on a crack, and I'm excited to read this update.

Sidecar question: when automating your backups, what's a good way to make sure your rolling backups aren't simply backing up malware-encrypted files? I found out too late that all my backups were encrypted. I only had 1 week retention to save space. Is longer retention and manual checks the only sane strategy? Or has some backup software built in sanity detection for crypto attacks by now?

> what's a good way to make sure your rolling backups aren't simply backing up malware-encrypted files?

Compute checksums. Also, if storing diffs check out how many files the back-up think changed. All of your photo library getting re-uploaded should be a red flag.

I do this using an `rsync --dry-run` with `--stats` enabled, you can get the number of changed/deleted files from the stats and then decide whether you want to proceed without `--dry-run`. This is for off-site backups. For on-site, I prefer ZFS Snapshots.
I do an off-line, off-site backup once a year that never gets overwritten. ZFS snapshots can help here, too, for online stuff, especially if you have monitoring about changeset size.
zfs server where you can only ssh into it with 1 user, and that user cannot zfs destroy, only zfs receive
Don't save just up to 1 week old snapshots. Also save a 2 week old, 3 week old and 1 month old one.
I store two weeks of daily snapshots, and then 12 months of monthly snapshots. It gives me a year of pretty good coverage for only about 2x the cost of just two weeks of backups.
One technique would be to place unchanging bait files that you pre-check before allowing the backup to proceed.
That’s a nifty and cheap idea. Now I am wondering if I should make the standard juicy targets (eg ~/Documents, .config, .ssh) complete decoys and put all of my real data just off to the side. Could still be hit by a generic attack, but targeted data extraction attempts would initially fail.
Hmmm, settings things like `~/.ssh` to non standard locations too would probably block a lot of the standard dependency-chain-malware coming around as well.
don't backup file names. Backup checksums.
I agree. And for some stuff you get cryptographic checksums for free.

Backup of Git repositiories:

    ... #  git fsck --full
    error: unable to unpack contents of .git/objects/a2/cf1a9631658799733f43c3b3f0a799696a4b21
    error: a2cf1a9631658799733f43c3b3f0a799696a4b21: object corrupt or missing: .git/objects/a2/cf1a9631658799733f43c3b3f0a799696a4b21
Oops... No matter if it's a malware, the lack of ECC which by bad luck induced a bit flip that wasn't detected (on an otherwise okay Git repo) or a disk failing, it's trivial to detect if the repo is corrupted.

Same for my ripped archive of Audio CDs. The rippers save lots of information and the rips are bitperfect, cross checked with other people's rips' checksums. And the checksums are all there.

For family pictures, I add a checksum to the pictures myself.

Backups aren't really backups until they've been verified :)

what do you mean adding checksum to the picture, do you add the checksum as a filename suffix eg IMG0001_<checksum>.jpg, something like that? Or do you tuck it into the exif data and have a tool that computes the checksum of the file minus the checksum part.
Yup exactly just adding a suffix. I'm not only backing .jpg files. For example I also backup a few screenshots (some are in .png and some are in .webp format).

So I don't care about the different pictures (or short family movies) format.

I just wrote some Clojure / babashka code to do that. I also truncate the checksum so that the filename doesn't become gigantic: it's not sensitive content, it's just to detect corruption.

Then I can use another computer and generate, say, all the thumbnails of the pictures and do a quick eyeball verification. If it looks correct, later on I can just automatically have the checksums verified.

Funnily enough I got a few old JPG pictures who were corrupt but I ended finding the correct version on older backups.

Checksum then helps too: otherwise you have two files with the same name (say on different HDD), but only one is correct and you don't know which one without manually opening them.

It's not super advanced and maybe a bit overkill but it's not complicated and works fine for my use case.

P.S: I take it another way would be to use a fs that use content-based addressing or does checksumming for me.

yah ZFS is supposed to alert somehow, I've been curious about the actual end user experience for that workflow and how it feels. Restoring from backup for disturbed crcs is excellent, I've been hoping to get into that action myself once I discovered various low priority files had bit rot on them.
People normally say don’t store binaries in git. Is this a big issue if the files don’t change very often? From what I understood the biggest problem is they don’t diff well. With photos not changing very often, can it work?

Anyone tried using git for 500G of photos?

I would love to if it worked, I have my photo collection spread out on multiple computers and merging the edits to the master backup is always a pita. “Was this file removed from copy A or added to copy B”? All those problems just solve themselves with a clear DVCS git history.

Not having the possibility of ever removing photos, to free up space, is of course another issue of git.

I guess that 500GB repository would be barely usable.

You should check out Git LFS if you want to do that, as it sounds like a good idea in the first place!

Git simply wasn't designed for that and so the key issue with storing binaries in it is what you mentioned last - that the way git works, a full clone has the full history of all the files. Deleting a file in git then doesn't actually delete the file from git history, so a fresh full clone of 500G of photos isn't going to be 500G, it's going to be that, times however many copies exist in history. A shallow clone solves that, and shallow clones supposedly work better these days in latest version of git, but fundamentally you're using a hammer on screws, as it were.

If you're open to new tools, git annex is what you're looking for. The other two options are Subversion, which has some DVCS features these days, or Perforce Helix Core (paid), though I can't vouch for it as I've never used it.

We've been working on some open source tooling called "oxen" that was built for large datasets of images, video, audio, text etc. We wanted to solve the exact problem you're flagging here with git.

Feel free to check it out here https://github.com/Oxen-AI/oxen-release#-oxen would love any feedback!

I've been playing around with beyond compare snapshot. I've done whole drive snapshots. I'm pretty close to running a diff to see how things have evolved on my drives and see where all the file system activity has shifted around. The files sizes are pretty small, in the MB range, maybe 6 or 10MB I forget.

https://www.scootersoftware.com/v4help/index.html?snapshots....

My backups got attacked by ransomware, but I only caught it about 6-8 months after it occurred. Thankfully, my backup drive is copied to another backup which never deletes files, only copies them, so the renamed files that were encrypted were eventually copied over to my second drive, but the originals remained. The attacker wasn't aware of the second drive.
For something like photos you could just store every revision.
Sorry that happened to you. Using a de-duplicating backup solution (such as rdiff-backup or restic), will let you keep daily increments indefinitely with very little overhead.
Don't do simple rolling backups, use something with deduplication like borg backup or ZFS/btrfs if you want to do it at the FS level. The backup size should not increase by much more than the actual size of any new files, so if suddenly, you need twice as much backup space because all your files seem to have changed, you should get suspicious.
Seconded, I have an external 2tb HDD that stores 1 years worth of daily backups from my 256gb (~180gb used at any time) laptop hard drive. My backup script (https://gist.github.com/Jeffrey-P-McAteer/7d4b9052825914b5e0...) takes maybe 30 minutes for a full backup, 5 minutes for most deltas. Files which are the same get hard-linked to the previous days backups, new files are copied over and content-de-duped by btrfs.
If nothing else, external hard drives are cheap and robust enough that I think more people should invest in having an offsite backup. Annually make a full backup, write the date on the outside, and leave it at the parents house. Make that your family holiday ritual.
As of recently, my setup for backups consists of 2x4 TB HDDs (from different manufacturers) with BTRFS in RAID 1, plugged into a small 2-drive USB3 docking station. With both checksumming and mirroring, feels pretty safe from HW-failure/bit rot standpoint (if one disk fails, you can still mount the other in "degraded" mode).
Agreed. I'll shill rsync.net (no affiliation, just a happy customer) and their ZFS VM backup service. It's basically just a lightweight freebsd VM with a big ZFS volume attached, so you can `zfs send` incremental backups to it, and they support meta-snapshotting of your backup machine on their end. I wrote https://github.com/wyager/zfs-backup to manage my automatic incremental backups, and there are a number of other tools like this.
Also ensure your client does not have access to the backup server share so that ransomware can't encrypt backups on a network drive etc.

My backup solution (backuppc/other syncs + zfs + sanoid/syncoid plus offsite server with zfs) means the backup server pulls files from the clients using backuppc/rsync. The backup server volume is zfs snapshoted regularly using sanoid. The offsite server pulls these from the backup server via syncoid/zfs send.

I'm not using rsync.net since I have my own infrastructure, but would definitely choose it as the offsite server if needed.

Yes this. I use Borg via ssh to an off-site server. With the proper ssh config (force-command, no pty, no forwarding etc) you can lock it down pretty well, especially since you can add an "append only" switch to the serve command that will refuse any modifications or deletions to existing snapshots.
> Nearly all my personal photos were encrypted by the helprecover@foxmail.com ("HELP") variant of Phobos.

I can't assist you with recovery, and without a lot of logs and forensics data (or a significant performance improvement), the described method is likely unfeasible. But I'll try to find a matching sample and let you know if it's vulnerable.

> Sidecar question: when automating your backups, what's a good way to make sure your rolling backups aren't simply backing up malware-encrypted files?

Lots of good responses, I like incremental backups without ovewriting anything (supported OOTB by all copy-on-write filesystems, like ZFS or BTRFS). Not sure how to configure this on Windows.

I use borgbackup to a server and once a month upload it to backblaze b2 with a 90 day retention policy on the bucket. The policy doesn't let you modify or delete the files for 90 days even if you have access to the account. Costs something like 5$ for the 3x500GB of backups I have.
> when automating your backups, what's a good way to make sure your rolling backups aren't simply backing up malware-encrypted files?

Maybe you could check the level of entropy (measure of randomness) of files before backing up - very high entropy could suggest encrypted data?

This is good. Some antivirus programs run this check, but some ransomware adapted by encrypting 16-byte AES blocks every so often in the file, so that the file becomes useless without entropy increasing too much.

Also, JPEG, PNG, .jar, .xlsx, etc. are already compressed, so pretty high entropy to begin with.

As others have pointed out, the growth rate of your de-duplicated backup size is probably the best way to detect ransomware.

I use restic which has very flexible retention policies. Rather than retain simply "x days", one can have it "keep the last x hourlies" all the way up to "keep the last x annuals", and XOR many criteria in one policy. Since it uses snapshotting, compression and deduplication it's very storage-efficient. They also offer a simple REST server which allows one to backup with append-only permissions, which can alleviate a ransomware attack.
I use rsync manually, and always with "--dry-run" first. Unless the ransome-ware is smart enough to rot my backup very, very, very slowly, I should be able to detect any problem by simply reading which files are to be overwritten. 99.99% of the files I back up rarely change.

I also have most of my non-sensitive data on Onedrive, which keeps old versions of files.

I worked in backup software for 22 years until a few years ago. My suggestion is to make a backup and store it for long term on an external drive. The to the cloud full followed incremental backup daily so you can go back to any point along the way at any time. Currently I use Acronis and it works well. I backup to their cloud to avoid having my local backups also encrypted preventing restoring in the event of failure or malware. Good backup strategy would not overwrite old backups until you must and backup regularly, so you don't lose any more data than you can afford to lose. Remember recovery time as well.
So, if you're a victim and you don't want to pay ransom

- hire an expert to find out the missing data (timestamp, is it the vulnerable version?, ...), then

- rent a GPU server and keep the fingers crossed.

I've looked for cheap GPU servers yesterday, the cheapest i found was Ultrarender at 200€ per week for a Dual RTX 3080Ti remote workstation. Which is probably overkill, a single RTX 3080Ti can do it in 33 hours or so (assuming the "12 Nvidia GPUs" they mentioned in the article are of similar speed).

Not sure if you could run that on vast.ai but they do offer more granular pricing, so that might be the better option.
Yes, that's the "almost" part of the blog post (and what makes it interesting, IMO). Not really something a regular person can do. Since CERT is not profit-driven, that kind of help was offered for governmental institutions for free (but it's hard, because government and companies need to recover ASAP and this kind of research and computation takes time).

If you want success stories, the same organisation has also published decryptors for other ransomware families (Mapo, Crypromix, Flotera) and they were just broken in a straightforward way. Usually the vulnerability is not explained for successful decryptors, to make bad guys life a bit harder.

coreweave.com has pretty cheap GPUs
I know I'm in minority and that this view is not empathetic one, but I really like that ransomware is around.

More secure data storage at companies where otherwise it would be just silently stolen and sold. More backups. Even some incentive to research security of encryption methods.

We won't get more secure systems without some proper incentives.

I love that ransomware is around because it forces companies to take security seriously or pay the price. You could even say ransomers are the good guys in this regard, contributing to a safer world.
Robber barons have always shaped the world in mysterious ways that history does not fully grasp.
This is like saying it’s a good thing that burglars exist so it forces us to invest in stronger locks/doors
An unhardened system allows the first attacker the maximum benefit. A hardened system reduces incentive for the first and every follow on attacker.

Good things can come from bad things. That doesn't make the bad things not bad.

If you really want burglars and locks analogy it would be invisible burglars, that steal secrets and things you don't notice from your home. Then some not invisible burglars appear and you realize you need better locks. Invisible burglars are more dangerous.

You may prefer world without burglars at all, but that is not an option. It's just wishful thinking.

I think this would be a stupid idea but you can buy an Antminer for $2458 and it does 104 TH/S for SHA256. I think doing a bit of hacking to make it crack the ransomware might be feasible?
There have been tons of projects that tried to repurpose bitcoin hardware. Even if it's not the latest that's a lot of computational power! So far though none of the project I follow have yielded any real results.

Here's a interesting read that gives a bit of info on the differences and why it doesn't really work. https://rya.nc/asic-cracking.html

Interesting, I wonder if you could (hypothetically) develop some hardware that does the bits the CPU needs to do and essentially have a cracker ASIC? And if it would yield improvements that require a reaction.
I'm sure you could build an asic for assistance, but I tend to see more FPGA style tools for these kinds of things given changes / variables that you wouldn't want hardcoded. There have definitely been a few tools built, the article I linked had reference to DES hardware crackers.

Anyone that comes up with a way to repurpose all those bitcoin miners into something productive will be pretty cool in my book.

I'm glad people are still finding that post useful.
Thanks for writing it! Most of your posts were interesting reads.