HN Explorer

Help:We Found a Bitcoin Mining Prog / Email Server Running on Our Server

26 points by gopi_ar ↗ HN
We saw that load spiked on one of Ubuntu servers yesterday and found this on the proc list using all our cores:

statd 7680 690 0.0 743976 58492 ? Sl Nov26 20914:53 ./yam -c 1 -M stratum+tcp://binyu.crypto%40gmail.com:x@xmr.pool.minergate.com:45660/xmr

This was running under the statd user.

What do we do? We checked firewall, SSH, all seem OK. How do we go about investigating this breach? Do help!

25 comments

[ 2.5 ms ] story [ 60.5 ms ] thread
throwbsidbdk
Hahahahaha that's a pretty creative way to monetize an attack.

Hopefully you're taking regular VM snapshots so you've got some logs they can't delete. Otherwise good luck, someone Bitcoin mining is probably clever enough to cover their tracks.

Realistically an breach bad enough that they have server control is probably through the web. The most common way I've seen is through various CMS code execution exploits. If your web apps allow file upload that's a really common way to get code running on the server as well

gopi_ar
Funny thing is we don't do uploads anywhere and there's no CMS whatsoever.. Which leads us to believe it's an OS vulnerability.

Would you how we could hire professionals to investigate this for us? And report it to appropriate groups..?

PS: These are dedicated servers :-/

throwbsidbdk
You could hire a firm that specializes in this sort of thing, but it's going to be expensive. Look for the guys that build security tools, like a company that worked on metasploit or has submitted multiple bug bounties. I know some of the makers of anti virus software will investigate this kind of thing for a (steep) price.

Basic computer forensics needs a copy of the drive as unaltered as possible so you should start with that before running or installing anything. Basically don't run the server if you want to be able to get anything our of it.

If it's not a user data breach its par the course to reintsall and sweep it under the rug lol.

Next time make sure the server takes snapshots and dumps logs to an external place where they can't be deleted.

flukus
>Which leads us to believe it's an OS vulnerability.

Have you ruled out an internal source that decided to use some of your "spare" capacity? Or a previously internal source that might not have had all their privileges revoked?

stevekemp
Is SSH available to the outside world? Many many compromised hosts spend all their time trying to login to remote hosts with dictionaries of usernames/passwords to try.

If your (root) password is weak then I'd not be surprised if that was the source of the infection.

You might see logins via "last", or via the system logs.

nodesocket
Can you answer the following for us?

    Is ssh only allowed by public key? (in /etc/ssh/sshd_config => PasswordAuthentication no)
    Is Apache or NGINX running on the server?
    Is PHP/Ruby/Node/Python running apps?
    What ports are open in iptables (iptables -L)
    What does /var/log/auth.log say?
gopi_ar
Thank you for responding.

We searched the whole system for authorized_keys files and found one created in a /var/lib/redis/ of a staging container (with no firewall) on this host. We then came across the redis vulnerability https://kevinchen.co/blog/postmortem-server-compromised/ . A junior dev had spawned this container without help from dev-ops and hence left ports open.

What doesn't make sense to us is how this daemon (yam) was running under a statd username when the container doesn't have such a user, but the host does? Are LXC containers able to run daemons on the host?

arpa
Well, it is always possible that attacker broke out of the container, as the container is still running under the same kernel, only its process(es) are chrooted + namespaced. Containerized "root" supposedly has its' privileges cut down, but if the kernel is exploitable...
jmgao
> What doesn't make sense to us is how this daemon (yam) was running under a statd username when the container doesn't have such a user, but the host does? Are LXC containers able to run daemons on the host?

This is because usernames don't exist, as far as the kernel's concerned. ps is resolving the process's UID to the corresponding name for the outside context, not the one inside the container.

gopi_ar
This makes sense, we can rest easy knowing they didn't break out of the container. Thanks!
matiasb
Hi, I wrote a Redis security tool once and this attack is familiar to me.

I might be able to help you on investigating this issue. Contact details are on my profile.

mcfrankline
Nope. Not bitcoin.

It's pointed at xmr.pool.minergate.com:45660/xmr

XMR is Monero. It has a lower hashrate so i see how the attacker can make something out of this.

Are you sure it's not an inside job? Cause anyone with access to run this under statd basically owns you right now

gopi_ar
Yes, it's a Redis vulnerability (caused by bad config on our part) in one container where the firewall was down.

Strange thing if we run 'top' from the main host, all containers running redis say 'statd' as their user; inside the container the user showed 'redis'. We removed nfs and all related files, and now it shows a user ID number. Is this something we should worry about?

shostack
On a related note... What stops companies from doing things like adding a background mining script to a website that uses people's browsers to do the mining?

I'm not very familiar with the tech behind BTC mining so there may be some obvious reason that isn't feasible. I was always surprised some evil company like EA hasn't added it to wait screens when matchmaking for various games. They could install whatever they want, target machines likely have solid GPUs, and they are sitting around waiting.

I also wonder about the legal aspects of this. Would someone need to opt in this? Is the energy cost of something like this legally distinguishable from sites that say... load a bazillion tracking tags and eat up your data cap?

r1ch
ESEA (a counter-strike matchmaking service) tried this. It didn't go well.

https://www.google.nl/search?q=esea+bitcoin+scandal

shostack
Interesting. So my takeaway from that is that they weren't really up front with getting consent and opt-in for it, and that it is still legal and valid if they were to have gotten that?
huehehue
There used to be a site that let you (voluntarily) do this. Can't find it now, and wouldn't be surprised if it shut down.

Anyway, it made the average computer sound like a jet engine. That, and the fact that the average computer is nearly useless compared to a dedicated mining rig, makes browser-based mining probably not worth it.

alexginzburg
we are in the same situation. Early morning today found one host with a high cpu usage. Turned out it was running `./yam` process as a `redis` user. I shut the host down for now. Before shutting it down I did a strace and saw json stream clearly stating that it is a monero app. Looks like the cpu spiked about 12 hours ago. We do have redis on a host but it should be behind the iptables rules. Other hosts look ok.
gopi_ar
We were able to get in touch with the hacker and he told us he was just mining and not stealing stuff. We're still cleaning the whole system; might even pay him/her a bounty for this though.
gopi_ar
Update: I sent an email to the email on that script. And the person at the other end replied and mentioned that he/she is doing it for extra pocket money and was only mining on the server. We aren't going to pursue any legal charges, might even pay the person a bounty for pointing out this vulnerability. I'd like to thank all of you, with special mention to some folks over at reddit for all your help!