Well that is scary. What anti virus / security tools do others on here run. On Linux I run nothing. I'm concerned that this might not be okay any more.
> The malware is still active and has a zero-detection rate in all major anti-virus systems.
Anti virus wouldn't really help in this case. I should also point out that it appears that HiddenWasp is targeting systems that are already compromised in some form.
The "achilles heel" of these sorts of CnC attacks tends to be, and is apparently so in this case, that they do need to phone home.
In this case, the IP addresses are known, and blackholing them with your firewall (try something like ufw [1], if you don't have one), or your hosts file, is enough. Using something like pi-hole you can also deploy these protections for every device on your network.
There are quite a few hosts lists maintained out in the wild [0], and tools like fail2ban to protect against brute-force attacks.
But, as usual, by far and away the safest way to protect your system is to not run untrusted code. You probably have a package manager, going outside it tends to be less safe.
Even using the official repos doesn’t guarantee you to be safe. The project developer might have had their development machine hacked and/or their source repository hacked and malicious code inserted. So when Arch / Ubuntu / etc package maintainers pull the latest source, unless they’re doing very thorough diffs of each source file, they could easily miss the injected malicious code and subsequently repackage that for the distro. What has actually happened before.
Also if your Linux / UNIX machine has an addressable WAN IP then you have all the usual risks of involved with hosting services on the public internet. A few years ago there was a spate of compromised web servers with the malware running as Apache modules.
So there aren’t really any guarantees when running Linux.
If you don't expect your users to receive shell script email attachments from untrusted sources (which could be the initial attack vector here), then blocking the SH extension would be one simple step. Of course, there are a few more things you could do: blocking shell script shebang signatures, zipped files with the executable bit set etc.
If this is something you want from your email provider, along with some other simple defenses [1][2], then please drop me an email at joran@ronomon.com, I work on an email startup in private beta.
The problem with this strategy is that IP Address/Domains/Hashes that are publicly available for free are typically dated and are rarely useful for catching anything other than mass scanning / ssh brute forcing or for doing retroactive searching and analysis.
You're also assuming that the linux server is the initial attack vector.
It is not uncommon for an org to be breached, and for the adversary to pivot to an admin box, and for the adversary to then gain control of a server using the admins legitimate credentials. All the while dropping persistence mechanisms along the way. Secure configurations and setups are always recommended and the above information is great advice. BUT it is also dated advice.
If you're not actively monitoring these servers and the activity on these servers they will eventually be popped. Logging all process/CLI activity is an awesome way to get started on monitoring, but if you're going that route, deploying Wazuh or an EDR tool can only add extra value by allowing you to create alerts off of specific values.
Don’t use it myself, but heard good things about Sandfly Security. Doesn’t use signatures but rather looks for odd things and detects HiddenWasp due to simple things like processes running from /tmp and users with uid 0
The AntiVirus industry is a snakeoil industry. Their products barely work and if they do, they do while sucking a lot of performance out of your device. If you absolutely want one, install ClamAV. Then use uBlock Origin + HTTPS Everywhere on your browser and PiHole on the network. Install all updates on devices, devices with no updates are isolated into a separate network. Use a password manager and SSH keys, disable password auth on all SSH servers you connect to. Close all ports to the outside that are not necessary and could be accessed with a VPN instead (keep one SSH port open to repair said VPN). Do not download email attachments you did not prompt for. If you need to open email attachments (ie you're HR and you received a Resume), make sure it's a real file and not a fake (a .pdf.exe) or ask for a share link from a popular storage provider (Nextcloud, Google Drive, Dropbox, etc.).
The above notes should get you protected better than 99.9% of people on the internet.
I work as a detection engineer for a Security vendor and I'm going to go against the grain and say that there ARE tools you should run to monitor your linux infrastructure that are similar to AV but are NOT AV. I write detection's for these tools all day long for Linux systems and there's a BUNCH you can detect and alert on that's abnormal activity that should be investigated and having tools in place such as an EDR tool can make those investigations easier.
Looking at the original post - some of those would be monitoring the the bash history environment variables. This is commonly modified by threat actors as a defense evasion technique. (I see it a bunch with insider threats to)
There's a bunch of others - but having just that one alert trigger would have likely been enough to start an investigation and catch this malware should the protective controls others have recommended fail.
That said the tools to do the more necessary modern monitoring do typically require more maintenance and configuration and it's not typically a set and forget type of solution.
Options I'd recommend:
LimaCharlie - https://limacharlie.io - Cloud based EDR tool, it has a cost, but is not expensive ($1 per endpoint monitored, supports all Linux flavors). It's a lot easier to deploy and manage, but you'll have to write your own rules. This though will also allow you to remotely look at all processes/pull files off of a machine remotely/isolate hosts on the network etc
There are a bunch of other EDR tools out there like Carbon Black or CrowdStrikes/FireEye's tooling, but they run a high premium ($20-40 per end point, and typically only support Ubuntu or Redhat or both)
Wazuh - https://wazuh.com/ - It's a better maintained fork of OSSEC that makes mass deployments easier/has more default rules etc. It'll give you FIM/Notify you of insecure configurations etc - This one will require the most work to setup
Security Onion - https://securityonion.net/ - this one is probably a little overkill, but modern versions include Wazuh (as seen above) as well as OSQuery/Network IDS's (which are it's core feature)
Look no further than the chain of xors that amount to ... ^ 0x22. These people really don't know much.
ISTM, the only interesting thing is the "user-mode rootkit" mentioned in the beginning. Anything installed after that is moot and completely irrelevant. If anyone gets root on your machine, it's already game over.
However, if your attack vector is getting me to execute a random .sh from you, then I'll roll my eyes and get on with life.
If your email firewall allows random .sh attachments, then you are vulnerable to at least two attacks:
1. Someone targets you specifically, knowing you are a software developer, by impersonating a trusted third-party contact and sending you a ZIP attachment with the contents of an open source project you are known to work on. ZIP attachments preserve the executable bit, and you wouldn't see any .sh extension because it would be an extension-less shell script. It would probably just look like a binary you are used to clicking.
2. Someone impersonates a trusted third-party contact, sending you a ZIP attachment with a symlink traversal to overwrite your .bashrc as soon as you unzip the attachment with a double-click, or as soon as unpatched middleware unzips it for you.
1. They're mixing a Bourne-ism (backquotes) with a Bash-ism (dollar-parens). I don't know why.
2. This could be an attempt to normalize and reduce whitespace, including eliminating newlines (though you should prevent injection of arguments to `echo`):
bash$ Temp=$(echo " a b c ")
bash$ echo "[${Temp}]"
[ a b c ]
bash$ Temp=`echo $(echo " a b c ")`
bash$ echo "[${Temp}]"
[a b c]
bash$ Temp=$(echo $(echo " a b c "))
bash$ echo "[${Temp}]"
[a b c]
Thank you, I was mistaken to call it a Bash-ism. Bash's documentation said it's a Posix-ism:
> The POSIX $() form of command substitution is implemented (see Command Substitution), and preferred to the Bourne shell’s `` (which is also implemented for backwards compatibility).
A shell script with an SH extension, possibly (my guess) delivered as an email attachment.
The article does not provide details, but does list a few steps to the exploit, the first being Linux users (as opposed to Linux servers) with poor antivirus.
For example, ZIP email attachments can contain extension-less files with the executable bit set. This would be one easy way of getting the initial payload bootstrapped.
> The malware will attempt to find the dynamic linker binary within these paths. The dynamic linker filename is usually prefixed with ld-<version number>.
'In addition, in order to check if your system is infected, you can search for “ld.so” files — if any of the files do not contain the string ‘/etc/ld.so.preload’, your system may be compromised.'
While developing our antivirus system, I was surprised to find that most email providers do not block ZIP attachments containing SH shell script files, nor ZIP attachments containing extension-less files with the executable bit set, nor ZIP attachments containing symlink traversal paths, nor any attachments which start with the shebang "#!" shell script signature, nor any which start with ELF or MACH O signatures.
In addition, for a recent sample of 15,000 attachments, we found 700 viruses, of which less than 50% were detected by the top 30 engines on VirusTotal. In particular, ClamAV, which is used by most Linux systems, had a less than 1% success rate. Ironically, most email providers know to block EXE attachments, even in ZIP attachments, but now its Linux and Mac users that are being left behind.
Is it really that strange? Windows email clients tend to run code in attachments just by clicking them (or, in some cases, just viewing them). So the risk factor in letting these emails in is quite large.
That's not as common with other platforms. Even the common GUI platforms on Linux would require the user to manually save the file and explicitly marking the file as executable. Nobody would conflate user friendliness with the ease of running untrusted code.
Defaults does make a difference. Even as these users are "left behind" in email filtering, we're seeing limited effects of malware with the more popular Linux platforms such as Android and the various desktops.
On windows, when you (double)click on an executable file it runs, because that's the default action for executables. On Unixes you need to mark the file as executable (with `chmod`) for this to happen.
Also, for many years already, Windows won't let you run an executable that you just downloaded from the internet just by double clicking, without first clicking through a warning dialog.
Yes, but then again warning dialogs don't have much track record in preventing malware outbreaks.
For example, recent versions of Word won't run VBA macros by default without the user clicking through a warning dialog, and yet people fall for it all the time.
Well, then I would disagree with you there. There are plenty of ways to get Windows to run executable code just by clicking an attachment, and this even without warning dialogs:
- PDFs with JS code. It's not a joke, someone actually titled a paper on PDF exploits alone, "PDF, Let Me Count the Ways".
- Hostile RTFs (with or without OLEs) disguised as Word documents (which Office will gladly open).
- Hostile HTML disguised as Excel spreadsheets (likewise).
- New unknown variants of the old Windows metafile exploit (just by viewing, as xorcist suggested).
I've literally never used a mail client that behaves this way. If the EXE even makes it past attachment scanners, I get a 'save'/'run' prompt and then if it's not signed and trusted, SmartScreen usually prompts me again to confirm that I really want to run it.
The app is usually in an installer package or ZIP which adds a couple more execute barriers.
The real threat is malware hidden in document formats like docx or pdf, which usually have fewer clicks before open + get exploited.
"Linux would require the user to manually save the file [...] explicitly marking the file as executable."
ZIP attachments preserve executable flags.
Also, symlink traversal attacks only require a ZIP attachment to be clicked. Apple patched macOS for this a little while ago, but it's probably still a thing on many systems - since most email providers know to detect ZIP directory traversals, but not ZIP symlink traversals.
There's also a range of other containers for macOS and Linux which will preserve executable flags if you ask them to.
And we have also seen simple character encoding attacks which can crash Apple Mail just on viewing the email.
> Even the common GUI platforms on Linux would require the user to manually save the file and explicitly marking the file as executable.
If someone can find a way to drop a text file in my "~/.config/autostart" dir, next time I log in my Gnome desktop will happily execute the command present in the "Exec" line and it doesn't require the execute bit.
I think something similar could be done using "~/.local/share/applications", overriding an installed application next time that application is launched, but it probably would show two icons; the autostart at the moment is more stealth.
I think the point is -- do to that you would have to have the email client do it.
While once it is out side the control of the email client -- as the quote suggest, it's up to the user -- not the email client to do the dirty work.
The notion is that on windows, even a all knowing user who knew the email contained a virus after viewing it would be helpless as the email client already did the steps required to make the virus active. Where as on Linux, the user would still need to be the facilitator -- not the email client blindly destroying your system.
As a example. You can email me virus all day long, and it is not likely I will get infected because of how my email client handles files and attachments. If I were to switch to a windows based system using a windows client then all bets are off.
tl;dr your exception would mean the bad thing we are talking about already happened. If they could get your email client to write to that file they probably already had enough control to do whatever you are suggesting they put in autostart.
> The notion is that on windows, even a all knowing user who knew the email contained a virus after viewing it would be helpless as the email client already did the steps required to make the virus active. Where as on Linux, the user would still need to be the facilitator -- not the email client blindly destroying your system.
OK, I see your point, normal usage against exploits, and usually security goes against convenience.
I don't know if I'm overly paranoid, but in this specific case I've started long time ago to keep my autostart dir read only. At the same time I've dirs in my command path that are writable by my user (but those need the execute bit tho)...
Keeping your autostart directory read-only is a great idea.
After that, a ZIP symlink traversal could still drop a script in your command path directories that are writable by your user. The execute bit is not a hurdle, unzip utilities will typically preserve the execute bit on Unix systems.
Antivirus is always a "closing the barn door after the horse has left"... it's completely reactionary/inadequate as it almost always relies on other people reporting a problem, so it's always behind and a Tragedy of the Commons in that very little of actual malware is ever reported. HIDS isn't much better. NIDS is slightly more useful as it's independent of the endpoint, but can also be evaded to a degree. Both HIDS/NIDS aren't great on their own because they're really the last lines of defense to tell you you're pwned.
In previous life, I inherited a bunch of Win2k database servers that were running on unfiltered public IPs (yuck!).. and several of them had undetectable, stealth rootkit malware that wasn't economically-feasible to remove (Microsoft and Winternals were in the loop).. and because they "couldn't be reimagined," network rules were used to disconnect it from bot operations. Yuck. Don't put unsecured or fragile anything on public IPs because odds are they will be pwnes with variants of malware no AV has ever seen, and it will be a persistent threat (APT) that won't be evident until later.
The best defenses include, but aren't limited to:
- Never run untrusted code.
- Never allow untrusted write and execute.
- Don't do email / browser operations with admin credentials. Do it in a VM maybe or a separate machine as a normal, unprivileged user is best.
- Least privilege, all the things.
- Verify cryptographic signatures enforcing chain-/web-of-trust of source/binaries/packages.
- Holistic defense-in-depth.
- Minimalism/minimal attack surfaces.
- Don't run fragile, unconfigured OSes on unfiltered public IP's. (Please.)
41 comments
[ 26.1 ms ] story [ 1699 ms ] threadAnti virus wouldn't really help in this case. I should also point out that it appears that HiddenWasp is targeting systems that are already compromised in some form.
The "achilles heel" of these sorts of CnC attacks tends to be, and is apparently so in this case, that they do need to phone home.
In this case, the IP addresses are known, and blackholing them with your firewall (try something like ufw [1], if you don't have one), or your hosts file, is enough. Using something like pi-hole you can also deploy these protections for every device on your network.
There are quite a few hosts lists maintained out in the wild [0], and tools like fail2ban to protect against brute-force attacks.
But, as usual, by far and away the safest way to protect your system is to not run untrusted code. You probably have a package manager, going outside it tends to be less safe.
[0] For example: https://github.com/mitchellkrogza/Ultimate.Hosts.Blacklist
[1] https://wiki.archlinux.org/index.php/Uncomplicated_Firewall#...
Also if your Linux / UNIX machine has an addressable WAN IP then you have all the usual risks of involved with hosting services on the public internet. A few years ago there was a spate of compromised web servers with the malware running as Apache modules.
So there aren’t really any guarantees when running Linux.
If this is something you want from your email provider, along with some other simple defenses [1][2], then please drop me an email at joran@ronomon.com, I work on an email startup in private beta.
[1] https://blog.cotten.io/ghost-emails-hacking-gmails-ux-to-hid...
[2] https://snyk.io/blog/how-to-crash-an-email-server-with-a-sin...
You're also assuming that the linux server is the initial attack vector.
It is not uncommon for an org to be breached, and for the adversary to pivot to an admin box, and for the adversary to then gain control of a server using the admins legitimate credentials. All the while dropping persistence mechanisms along the way. Secure configurations and setups are always recommended and the above information is great advice. BUT it is also dated advice.
If you're not actively monitoring these servers and the activity on these servers they will eventually be popped. Logging all process/CLI activity is an awesome way to get started on monitoring, but if you're going that route, deploying Wazuh or an EDR tool can only add extra value by allowing you to create alerts off of specific values.
The above notes should get you protected better than 99.9% of people on the internet.
Looking at the original post - some of those would be monitoring the the bash history environment variables. This is commonly modified by threat actors as a defense evasion technique. (I see it a bunch with insider threats to)
There's a bunch of others - but having just that one alert trigger would have likely been enough to start an investigation and catch this malware should the protective controls others have recommended fail.
That said the tools to do the more necessary modern monitoring do typically require more maintenance and configuration and it's not typically a set and forget type of solution.
Options I'd recommend:
LimaCharlie - https://limacharlie.io - Cloud based EDR tool, it has a cost, but is not expensive ($1 per endpoint monitored, supports all Linux flavors). It's a lot easier to deploy and manage, but you'll have to write your own rules. This though will also allow you to remotely look at all processes/pull files off of a machine remotely/isolate hosts on the network etc
There are a bunch of other EDR tools out there like Carbon Black or CrowdStrikes/FireEye's tooling, but they run a high premium ($20-40 per end point, and typically only support Ubuntu or Redhat or both)
Wazuh - https://wazuh.com/ - It's a better maintained fork of OSSEC that makes mass deployments easier/has more default rules etc. It'll give you FIM/Notify you of insecure configurations etc - This one will require the most work to setup
Security Onion - https://securityonion.net/ - this one is probably a little overkill, but modern versions include Wazuh (as seen above) as well as OSQuery/Network IDS's (which are it's core feature)
VER=`echo $(uname -a)`
Versus just:
VER=$(uname -a)
or
VER=`uname -a`
ISTM, the only interesting thing is the "user-mode rootkit" mentioned in the beginning. Anything installed after that is moot and completely irrelevant. If anyone gets root on your machine, it's already game over.
However, if your attack vector is getting me to execute a random .sh from you, then I'll roll my eyes and get on with life.
1. Someone targets you specifically, knowing you are a software developer, by impersonating a trusted third-party contact and sending you a ZIP attachment with the contents of an open source project you are known to work on. ZIP attachments preserve the executable bit, and you wouldn't see any .sh extension because it would be an extension-less shell script. It would probably just look like a binary you are used to clicking.
2. Someone impersonates a trusted third-party contact, sending you a ZIP attachment with a symlink traversal to overwrite your .bashrc as soon as you unzip the attachment with a double-click, or as soon as unpatched middleware unzips it for you.
2. This could be an attempt to normalize and reduce whitespace, including eliminating newlines (though you should prevent injection of arguments to `echo`):
> The POSIX $() form of command substitution is implemented (see Command Substitution), and preferred to the Bourne shell’s `` (which is also implemented for backwards compatibility).
https://www.gnu.org/software/bash/manual/html_node/Major-Dif...
But xor 1-2 can be combined, so does xor 3-4.
The article does not provide details, but does list a few steps to the exploit, the first being Linux users (as opposed to Linux servers) with poor antivirus.
For example, ZIP email attachments can contain extension-less files with the executable bit set. This would be one easy way of getting the initial payload bootstrapped.
Perhaps it's a Debianism, but it's been '/lib/ld-linux.so.$ver' for ages:
> The malware will attempt to find the dynamic linker binary within these paths. The dynamic linker filename is usually prefixed with ld-<version number>.
While developing our antivirus system, I was surprised to find that most email providers do not block ZIP attachments containing SH shell script files, nor ZIP attachments containing extension-less files with the executable bit set, nor ZIP attachments containing symlink traversal paths, nor any attachments which start with the shebang "#!" shell script signature, nor any which start with ELF or MACH O signatures.
In addition, for a recent sample of 15,000 attachments, we found 700 viruses, of which less than 50% were detected by the top 30 engines on VirusTotal. In particular, ClamAV, which is used by most Linux systems, had a less than 1% success rate. Ironically, most email providers know to block EXE attachments, even in ZIP attachments, but now its Linux and Mac users that are being left behind.
That's not as common with other platforms. Even the common GUI platforms on Linux would require the user to manually save the file and explicitly marking the file as executable. Nobody would conflate user friendliness with the ease of running untrusted code.
Defaults does make a difference. Even as these users are "left behind" in email filtering, we're seeing limited effects of malware with the more popular Linux platforms such as Android and the various desktops.
Citation please? This is not my experience.
Which email client is that?
Also, for many years already, Windows won't let you run an executable that you just downloaded from the internet just by double clicking, without first clicking through a warning dialog.
For example, recent versions of Word won't run VBA macros by default without the user clicking through a warning dialog, and yet people fall for it all the time.
Malware authors don't mind either way.
- PDFs with JS code. It's not a joke, someone actually titled a paper on PDF exploits alone, "PDF, Let Me Count the Ways".
- Hostile RTFs (with or without OLEs) disguised as Word documents (which Office will gladly open).
- Hostile HTML disguised as Excel spreadsheets (likewise).
- New unknown variants of the old Windows metafile exploit (just by viewing, as xorcist suggested).
- Image "stegosploits" (likewise).
The list goes on...
The app is usually in an installer package or ZIP which adds a couple more execute barriers.
The real threat is malware hidden in document formats like docx or pdf, which usually have fewer clicks before open + get exploited.
ZIP attachments preserve executable flags.
Also, symlink traversal attacks only require a ZIP attachment to be clicked. Apple patched macOS for this a little while ago, but it's probably still a thing on many systems - since most email providers know to detect ZIP directory traversals, but not ZIP symlink traversals.
There's also a range of other containers for macOS and Linux which will preserve executable flags if you ask them to.
And we have also seen simple character encoding attacks which can crash Apple Mail just on viewing the email.
If someone can find a way to drop a text file in my "~/.config/autostart" dir, next time I log in my Gnome desktop will happily execute the command present in the "Exec" line and it doesn't require the execute bit.
I think something similar could be done using "~/.local/share/applications", overriding an installed application next time that application is launched, but it probably would show two icons; the autostart at the moment is more stealth.
While once it is out side the control of the email client -- as the quote suggest, it's up to the user -- not the email client to do the dirty work.
The notion is that on windows, even a all knowing user who knew the email contained a virus after viewing it would be helpless as the email client already did the steps required to make the virus active. Where as on Linux, the user would still need to be the facilitator -- not the email client blindly destroying your system.
As a example. You can email me virus all day long, and it is not likely I will get infected because of how my email client handles files and attachments. If I were to switch to a windows based system using a windows client then all bets are off.
tl;dr your exception would mean the bad thing we are talking about already happened. If they could get your email client to write to that file they probably already had enough control to do whatever you are suggesting they put in autostart.
OK, I see your point, normal usage against exploits, and usually security goes against convenience.
I don't know if I'm overly paranoid, but in this specific case I've started long time ago to keep my autostart dir read only. At the same time I've dirs in my command path that are writable by my user (but those need the execute bit tho)...
After that, a ZIP symlink traversal could still drop a script in your command path directories that are writable by your user. The execute bit is not a hurdle, unzip utilities will typically preserve the execute bit on Unix systems.
ZIP directory traversals are as old as the hills, yet they are still a thing: https://snyk.io/research/zip-slip-vulnerability.
ZIP symlink traversals are even more powerful, yet they are hardly spoken of.
Ideally, you want your email provider to be blocking ZIP attachments with symlink traversals or execute bits (but I am not aware of any which do).
In previous life, I inherited a bunch of Win2k database servers that were running on unfiltered public IPs (yuck!).. and several of them had undetectable, stealth rootkit malware that wasn't economically-feasible to remove (Microsoft and Winternals were in the loop).. and because they "couldn't be reimagined," network rules were used to disconnect it from bot operations. Yuck. Don't put unsecured or fragile anything on public IPs because odds are they will be pwnes with variants of malware no AV has ever seen, and it will be a persistent threat (APT) that won't be evident until later.
The best defenses include, but aren't limited to:
- Never run untrusted code.
- Never allow untrusted write and execute.
- Don't do email / browser operations with admin credentials. Do it in a VM maybe or a separate machine as a normal, unprivileged user is best.
- Least privilege, all the things.
- Verify cryptographic signatures enforcing chain-/web-of-trust of source/binaries/packages.
- Holistic defense-in-depth.
- Minimalism/minimal attack surfaces.
- Don't run fragile, unconfigured OSes on unfiltered public IP's. (Please.)