I've recently learned that OpenZFS has effective counter-measures for ransomware and data corruption in general. Quote from here [0]:
> As a copy-on-write file system, OpenZFS provides efficient and consistent snapshots of your data at any given point in time. Each snapshot only includes the precise delta of changes between any two points in time and can be cloned to provide writable copies of any previous state without losing the original copy. Snapshots also provide the basis of OpenZFS replication or backing up of your data to local and remote systems. Because an OpenZFS snapshot takes place at the block level of the file system, it is immune to any file-level encryption by ransomware that occurs over it. A carefully-planned snapshot, replication, retention, and restoration strategy can provide the low-level isolation you need to enable your storage infrastructure to quickly recover from ransomware attacks.
They're pretty automatic on any system I get my grubby hands on. The added advantage is that backups done properly will protect you against the machine going up in smoke, where CoW won't.
That's why proper backups are done in a pull-configuration and not the easy way of push. I.e., the backup system has access to the production system, and grabs the data itself. The production system can at worst serve bad data to the backup system, against which the protections are not that hard. Basically just a protocol client that can handle malicious servers, like a HTTP(S) server/client or such.
Many (most?) backup systems seem to mount the backup filesystem locally, allowing ransomware to access and encrypt the backup files too. For example, not too long ago I set up a networked drive that can be accessed only with a password over Apple File Protocol. I gave that password ONLY to Time Machine hoping it might do something smart and mount the drive in a way that only it could access. Of course it mounts it under /Volumes, allowing it to be accessed via Terminal without having to enter the password...
I guess because of this you see "protect yourself from ransomware attack" guides recommending keeping your backup drive/filesystem attached for as short a period as possible. But even then you're left hoping that the ransomware doesn't encrypt your backups first - perhaps because it's smart enough to - or that it doesn't get too many of them before you notice the infection.
I wish all backup software would avoid making the remote filesystem available system-wide and raise the bar by requiring ransomware to know how to extract access to the remote filesystem from the particular backup software you're using before it could encrypt your backups. Unfortunately, as far as I could tell last time I looked this issue seems to be ignored by backup software vendors.
This is a completely valid point. Why would you give access to the backup storage area to the production system?
Where I have implemented it, the only access the production system has to the backup system is to connect to an ssh server with a private key, and shove a tar down the pipe, which the backup system then squirrels away somewhere safe. This is a push configuration, but it's a safe one - the backup system doesn't have access to the production system either, and the production system cannot overwrite/delete any data on the backup system.
I suppose that snapshotting will cause old versions of the data to be thrown away, once the old versions start taking up too much disk space. Then I suppose the ransomware can take advantage of this and simply perform so many writes that the unencrypted versions of the data are thrown away. And isn't this what happens anyway when a disk is nearly full and you start encrypting its contents?
If there are no automatic snapshots by default then the snapshots are just like backups - i.e. an active measure the user has to take? Then it doesn't really protect against ransomware (Assume that we are talking about users who don't have backup - they wouldn't have zfs snapshot either?)
ZFS snapshots can be automated. I have ZFS periodic snapshots enabled on my FreeNAS system, where I can select when snapshots will occur (frequency and time) as well as how long the snapshots will be kept.
Backups can also be automated. I think that parent's point is that, like backup, if it isn't the default then users are probably not doing it. It is my experience that most people, even people who should know better like you or me, think of data loss risks as theoretical until they've experienced it first hand once or twice.
For better or worse, ZFS isn't really targeted at "average users" as of right now.
See also: There is no stable ZFS implementation for Windows as of right now, and the implementation for macOS (while surprisingly painless in general) requires at least some command line use to set up.
If anyone actually develops a ZFS implementation designed for average users, I would agree that it should include automated backups by default.
It's actually not allowing you to write new data at all. So it prevents the user from assuming there is more space to store his data, as there is not sufficient space with his indicated/configured data retention wishes.
Typically these systems are set up to automatically snapshot on a periodic basis (say nightly) and automatically pruned after another period (say monthly) - if there was massive activity due to ransomware causing a full disk in most such systems it'd have to be manually investigated, it wouldn't just decide to delete snapshots to free space.
It certainly should help against ransomware in that an admin who was investigating an unexpectedly full disk, who's hopefully aware of such things, would probably notice that all the files on the drive were no longer readable and all had recent modification timestamps. And that's if a user doesn't happen to notice it first and report issues with their files to IT.
As long as those snapshots last more than a few days as to provide sufficient time to catch and investigate the issue, ransomware damage is usually revertable much faster than backups with simple snapshots.
> so many writes that the unencrypted versions of the data are thrown away
I don't know about this specific system, but typical behaviour of NILFS (a continuously shapshotting log file system) is to report the filesystem as full, requiring manual intervention or waiting until the data ages-out and gets garbage collected.
Export zvols from a central fileserver via iscsi, and install the exposed operating system on it. Snap the zvols on the fileserver. Unless your fileserver is compromised, there's no way the ransomeware can persist in your network.
Zvols on zfs exhibit poor disk utilisation unless your block size, stripe size and file size line up just right. They work, but I've found major space savings sticking with zfs directly, if possible.
>I've recently learned that OpenZFS has effective counter-measures for ransomware and data corruption in general.
Sort of. The data corruption part yes, that's a core transparent part of the filesystem as it should be. But for ransomware ZFS only provides helpful tools for doing so, it's not designed around that specifically (though it could provide the foundation for something that was). Very low cost atomic snapshots and ease of replication definitely can help if they're used, particularly combined with ZFS permission delegation system. You can set a user to have snapshotting permissions separately from permission to destroy snapshots, as well as rollbacks, sends, etc. Of course that won't help if ransomware gets root (unless in turn the storage has hardware separation, loaded over a NAS say), but it's still quite handy.
It would be interesting actually if ZFS had some limited active content monitoring capabilities, such as detecting changes in entropy or a sudden massive spike in snapshot delta size (indicating everything in the file system had been touched). At any rate though note the "a carefully planned" bit there, ZFS is a fantastic foundation for a better data protection system and makes it easier to do well. But it's not install-and-forget by itself either right now, though someone could build something on top if they wanted to (or maybe already has? it has been a while since I surveyed ZnapZend and the like).
>spent lavishly on a fleet of luxury cars, two French villas, and a large yacht ... lives ... on the Black Sea, where Russian officials have declined for years to arrest him
It's a shame the Russians aren't a little more cooperative with stopping this kind of stuff.
The article even mentions that allegedly the bot network was used to launch attacks beneficial to the Russian government.
If there is any criminal organization powerful enough in Russia, and their leaders haven't been assassinated or imprisoned, it's safe to assume they work closely with the government.
I've always felt the US government should just issue a modern day Marque and Reprisal against countries who don't play ball with hackers like this. Russia won't arrest their cyber criminals? Okay, just announce that it's legal for American hackers to attack Russian targets.
Depends on if you care more about the benefits of getting at your enemies than the consequences of legitimizing criminal behavior and/or the inevitable "friendly fire" incidents.
>The FBI worked with a number of security researchers to reverse engineer the DGA and figure out how it generated the list of domain names it provided to the infected computers every week. That way, the FBI would know which thousand domains were being selected every week in advance. Then, right before the takedown, they acquired a temporary restraining order that required domain registries in the U.S. to redirect any attempts to contact those thousand domains to a substitute, government-run server.
So the answer is, yes, there was a way, and so eventually that's what law enforcement did.
Why would taking down the C&C for one week kill the botnet? Or for that matter even a few months. I assume the bots can just keep trying to find the C&C until it finally gets through.
From the article:
"on July 11, 2014, the Justice Department reported that the number of computers infected with GOZ malware had been reduced by 31 percent thanks to law enforcement intervention."
Also they were blocking the autogenerated domains on a weekly basis, but that would only work if the infected machine was in the US I believe.
"If Bogachev had been more careful about not using his real name when registering for accounts, he might not now be the FBI’s most wanted cybercriminal."
I'm not sure he cared about being discovered given this information..."Bogachev lives in the resort town of Anapa on the Black Sea, where Russian officials have declined for years to arrest him or extradite him to the United States. In fact, the Russian government has benefited from his criminal activity."
Friends in high places never hurts, especially if they are corrupt. He basically gets to live life like a Bond villain so I'm reasonably confident he never intended to hide his name or activity for the long haul.
Quite possibly, but he's probably still a millionaire. Not a bad deal for him. He still gets to make millions and commit fraud scot free, with the protection of the Russian government. He also gets to have fun working on sophisticated malware and novel C2 infrastructure architectures, continuously honing his skills.
If I were him (and also had no morals) I'd probably be pretty happy with my life.
An interesting read, thanks for sharing. At the end ransomware WannaCry was mentioned. Is it in any way related to Bogachev's operations or is it confirmed to be an effort by North Korea? The part where he uses his real name for a yahoo account does sound stupid, but maybe he really felt safe that Russian authorities won't cooperate with an international arrest order.
Another thing to look at is dedupe percentage...if things all of a sudden stop compressing really well, it's an indicator of text files being converted to encrypted, random looking, noise.
Something I've long wondered about—shouldn't Ransomeware be relatively easy to detect with a heuristic? There aren't all that many† legitimate use cases for rewriting 30%+ of the data on your hard disk. Seems like a good time for the OS to pause the process and notify the user before continuing. If ZFS snapshots or similar are in use, there are even fewer cases where you'd want to rewrite 30% of data and delete all snapshots.
† Note, "not many" ≠ zero. I realize this would sometimes result in false positives, but I imagine the trade-off would be worthwhile, especially if the protection was user controlled.
Windows admins have been running heuristics with File Server Resource Manager for a few years. We had users disabled and SMB access immediately denied for users that created certain file types, and email alerts get sent. It was surprisingly effective.
54 comments
[ 3.0 ms ] story [ 86.9 ms ] thread> As a copy-on-write file system, OpenZFS provides efficient and consistent snapshots of your data at any given point in time. Each snapshot only includes the precise delta of changes between any two points in time and can be cloned to provide writable copies of any previous state without losing the original copy. Snapshots also provide the basis of OpenZFS replication or backing up of your data to local and remote systems. Because an OpenZFS snapshot takes place at the block level of the file system, it is immune to any file-level encryption by ransomware that occurs over it. A carefully-planned snapshot, replication, retention, and restoration strategy can provide the low-level isolation you need to enable your storage infrastructure to quickly recover from ransomware attacks.
[0] https://www.ixsystems.com/blog/combating-ransomware/
I guess because of this you see "protect yourself from ransomware attack" guides recommending keeping your backup drive/filesystem attached for as short a period as possible. But even then you're left hoping that the ransomware doesn't encrypt your backups first - perhaps because it's smart enough to - or that it doesn't get too many of them before you notice the infection.
I wish all backup software would avoid making the remote filesystem available system-wide and raise the bar by requiring ransomware to know how to extract access to the remote filesystem from the particular backup software you're using before it could encrypt your backups. Unfortunately, as far as I could tell last time I looked this issue seems to be ignored by backup software vendors.
Where I have implemented it, the only access the production system has to the backup system is to connect to an ssh server with a private key, and shove a tar down the pipe, which the backup system then squirrels away somewhere safe. This is a push configuration, but it's a safe one - the backup system doesn't have access to the production system either, and the production system cannot overwrite/delete any data on the backup system.
See also: There is no stable ZFS implementation for Windows as of right now, and the implementation for macOS (while surprisingly painless in general) requires at least some command line use to set up.
If anyone actually develops a ZFS implementation designed for average users, I would agree that it should include automated backups by default.
It certainly should help against ransomware in that an admin who was investigating an unexpectedly full disk, who's hopefully aware of such things, would probably notice that all the files on the drive were no longer readable and all had recent modification timestamps. And that's if a user doesn't happen to notice it first and report issues with their files to IT.
As long as those snapshots last more than a few days as to provide sufficient time to catch and investigate the issue, ransomware damage is usually revertable much faster than backups with simple snapshots.
I don't know about this specific system, but typical behaviour of NILFS (a continuously shapshotting log file system) is to report the filesystem as full, requiring manual intervention or waiting until the data ages-out and gets garbage collected.
Sort of. The data corruption part yes, that's a core transparent part of the filesystem as it should be. But for ransomware ZFS only provides helpful tools for doing so, it's not designed around that specifically (though it could provide the foundation for something that was). Very low cost atomic snapshots and ease of replication definitely can help if they're used, particularly combined with ZFS permission delegation system. You can set a user to have snapshotting permissions separately from permission to destroy snapshots, as well as rollbacks, sends, etc. Of course that won't help if ransomware gets root (unless in turn the storage has hardware separation, loaded over a NAS say), but it's still quite handy.
It would be interesting actually if ZFS had some limited active content monitoring capabilities, such as detecting changes in entropy or a sudden massive spike in snapshot delta size (indicating everything in the file system had been touched). At any rate though note the "a carefully planned" bit there, ZFS is a fantastic foundation for a better data protection system and makes it easier to do well. But it's not install-and-forget by itself either right now, though someone could build something on top if they wanted to (or maybe already has? it has been a while since I surveyed ZnapZend and the like).
It's a shame the Russians aren't a little more cooperative with stopping this kind of stuff.
If there is any criminal organization powerful enough in Russia, and their leaders haven't been assassinated or imprisoned, it's safe to assume they work closely with the government.
It would have to be for crimes defined in Russian law, but yes.
As a hypothetical, I wonder whether the US could carry out a private prosecution in Russia under Russian law.
He gave a lecture about it in one of the classes I followed at one point.
Wouldn't there be a seed on each infected machine used to determine the current C&C domains to use? Why couldn't we predict them?
>The FBI worked with a number of security researchers to reverse engineer the DGA and figure out how it generated the list of domain names it provided to the infected computers every week. That way, the FBI would know which thousand domains were being selected every week in advance. Then, right before the takedown, they acquired a temporary restraining order that required domain registries in the U.S. to redirect any attempts to contact those thousand domains to a substitute, government-run server.
So the answer is, yes, there was a way, and so eventually that's what law enforcement did.
From the article: "on July 11, 2014, the Justice Department reported that the number of computers infected with GOZ malware had been reduced by 31 percent thanks to law enforcement intervention."
Also they were blocking the autogenerated domains on a weekly basis, but that would only work if the infected machine was in the US I believe.
I'm not sure he cared about being discovered given this information..."Bogachev lives in the resort town of Anapa on the Black Sea, where Russian officials have declined for years to arrest him or extradite him to the United States. In fact, the Russian government has benefited from his criminal activity."
Friends in high places never hurts, especially if they are corrupt. He basically gets to live life like a Bond villain so I'm reasonably confident he never intended to hide his name or activity for the long haul.
If I were him (and also had no morals) I'd probably be pretty happy with my life.
† Note, "not many" ≠ zero. I realize this would sometimes result in false positives, but I imagine the trade-off would be worthwhile, especially if the protection was user controlled.