This is also a good reminder why running services as root is not recommended. If he had used a separate non-root account for Redis, the damage would have been limited to that single user account. That's not a very good situation either, but at least the attacker would not have been able to modify init scripts and replace system files.
> If he had used a separate non-root account for Redis, the damage would have been limited to that single user account.
Probably not. The attacker would have gotten access to the machine. As there is a lot of software installed locally some probably have holes, which allows for full root access.
So I don't really believe it makes much difference for serious attacker if the attacker gets root access or normal access.
After understanding how much this attack was used, I modified the blog post in order to prevent script kiddies from using it without a few changes, but still it caused a lot more harm than I expected. Before Redis 3.2 I'll add security features to prevent this from happening, while adding them in a form that should prevent also people from getting annoyed by the security features. It's an hard mix to get right.
Unfortunately that excludes your whole site from the Wayback Machine, but you might try modifying the Disallow. (I haven't a clue if their rules are granular or strictly site-wide.)
You posted a tutorial describing how to use your software to exploit servers, and you're surprised that people used it? I really hope this was a one time lapse in judgement and not indicative of how the redis team thinks about security.
This is the full disclosure vs antisec debate... if they posted it after they had a patched version out, the best thing to do is explain the issue and get people to upgrade, in my opinion.
Granted, I didn't read the original blog post. Were they distributing tools to make it easy? Or were they explaining how it worked?
In any case, if he hadn't written the blog post, someone else would have.
Popping antirez quote on the lunacy of running this "publicly accessible":
> "From time to time I get security reports about Redis. It’s good to get reports, but it’s odd ... in a software which is designed (as we explain in our security page here http://redis.io/topics/security) to be totally insecure if exposed to the outside world."
Regardless of the readme, the Lessons Learned should include both "know the security posture of the services you run" and "don't run anything publicly facing unless you are forced to, and only then if hardened well".
The first paragraph says he made a mistake when uninstalling Redis, he didn't have it running on purpouse.
Everyone is fallible, things tend to go wrong. Insecure defaults in software hurt even expert devs (which this guy is if you look at his blog).
If your software is "designed to be totally insecure if exposed to the outside world" you should not ship it with defaults exposing it to the outside world. IOW you should not design software to be totally insecure by default.
According to http://www.antirez.com/news/96 it binds to 0.0.0.0 by default, though in latest versions there is an example config file provided for binding to 127.0.0.1
Of course everyone is fallible, and this is the reason why careful admins put safeguards and layers of security.
In a well-prepared system, the administrator would:
#1 block input traffic on all the ports, except the required ones. as routine, after uninstalling any service, open ports are reviewed; even if a service is forgotten running, it won't be able to communicate.
#2 not run redis or any service as root user.
Not doing so is, to say the least, reckless, and with this assumption/context, it doesn't make much sense to theorize about software design.
This should have been run as a non privileged user, on private IPs, a local socket or localhost along with a long and strong password, so only authenticated connections would be able to be able to use it.
Since none of these memory store solutions (memcached, redis, etc.) should be able to be connected to on public addresses it would be a nice feature to only allow them to listen on private internal IP space or sockets by default and fail to run if anything else is used.
This would help reduce everyone's attack surface by default and help prevent misconfiguration by the user or 3rd party integration software.
Install things through the package manager when possible. This makes it easier to uninstall the software cleanly.
Of course one uses the system package manager when possible, and a specialized package manager like npm in most other cases, but not all is lost when one needs a recent version of some autotools monstrosity. In that case just install it using GNU stow. Then uninstalling is just an invocation of "stow -D" (plus deleting a single directory).
But that only makes sense for a box one physically possesses. Why on earth would one reuse a VPS? Nevertheless, I'm glad that TFA did, because the investigation was very interesting.
The most important thing to do after a server setup is to enable firewall. By default block all ports on public IP and then open ssh, http and https. There are tools like ufw, just few commands and your server is secure. This is much better approach than securing all services and system deamons run on a server.
>This is much better approach than securing all services and system deamons run on a server.
Um, no. You should absolutely be doing both. Defense in depth is the only way to remain secure in failure of any other component. A firewall WILL NOT PROTECT YOU FROM THAT.
For example, you just put on the firewall and leave a daemon running on localhost as root. The firewall allows connections from localhost by default (as most do). An attacker finds a hole in your webserver and is able to run scripts in the httpd context. The script makes a connection to the daemon on localhost, exploits it, and now has root privileges. With root the attacker modifies the firewall rules to allow access. You are now fully exploited.
Had your daemon been secured they would have only been a takeover of httpd and daemon accounts and not a full system compromise. Privilege separation is extremely important.
Is there a Saas that helps detect these open instances for you? Sort of like the opposite of pingdom. I want an email when something is accessible because I might have changed a firewall setting that might have accidentally opened the port.
In addition to what others say - first firewall, then lock down services in turn - consider running OSSEC or another intrusion detection system if you are concerned about intrusion.
But really this is the OP's problem: he ran a system open to the internet without any apparent analysis of attack vectors. Server was compromised due to incompetence - not developer incompetence, but operations incompetence, a different discipline.
Can't agree more with the OSSEC recommendation ( http://ossec.net ) . Been using it extensively and gives an early warning system if something is going on (both in terms of files changes, backdoors and log trails).
>How can you make a software intentionally insecure and then also bind to all interfaces by default?
In the last 20 years I've been working on systems, that is the default on way too much. It's only been in very recent history that things come more secured by default. In the past popularity was strongly correlated with ability to set up. 'Works by default with no security' is really easy to setup and reduces the number of complaints that "product is too hard to use, when with $x product instead".
Considering the skills demonstrated in this analysis, you might discover getting started with configuration management (puppet/chef) is easier and might take less time then you expect (even in a fast moving early startup). Might have totalled less time then this hack took up.
As it was said in the "background section" this is not a problem with redis but a configuration/ops problem instead.
Either way, you should always check the open ports and listening services in a given machine. In this case for a quick check you could use: https://www.isredisallsafe.com/
>this is not a problem with redis but a configuration/ops problem instead.
Default system wide failure is a redis problem. Any modern daemon should come with configuration files that a) don't allow the application to run as root unless uncommented by user action, and b) bind to localhost. Further configuration should be necessary to break the system as thoroughly as this user did.
But I am puzzled: a running process like redis is supposed to run in a chroot, and /tmp is supposed to always be mounted with no exec, the chroot and process should have been runned with seteuid/guid redis...
I normally have backend technologies confined in a jails with pf authorizing only inbound & outbound traffic on the legitimate interface+port, and you need no ssh access in a jail.
Looks to me there is a cascade of sysadmins wrong practice.
Devops and modern "full stack devs" seems to have lost their basics.
I wear a "sysadmin" hat (among many others) but not a "dev" hat. Every sysadmin I know who's worth their paycheck does the things you've mentioned above (and others), every time.
Perhaps this is what happens when you take a "dev" with no real sysadmin experience and make him/her responsible for these machines? It's like giving my neighbor's kid with a small dirt bike the key to my Harley and sending him out on the highway -- bad things are bound to happen.
It reminds me of the common solution that many "amateur sysadmins" have when they encounter SELinux issues: turning it off. :/
Selinux is crap. To complex to configure, and poorly configured by default.
process poorly packaged crash and nothing in the logs.
You disable it, see there was a warning about a poor permission and shout at selinux for making you lose days worth of works by poorly logging.
Btw, I am a dev. But I have to sometimes go in production see servers because I need to help production sysadmins troubleshoot the weirdness of todays mis-performing easy to use solutions (selinux, systemd, rsyslogd, packaging).
When you have to recompile centos or debian packages to have them support vanilla features and you are a dev, it is boring.
As always I think the answer is in the middle. You can't have a piece of software that defaults to insecure and expect everyone to have the common sense and skill to secure it. You rarely can release a piece of software with hard security defaults and expect it to have enough usability to gain critical mass / adoption / be popular.
I remember when I first learned web development and I absolutely had a belief that the defaults would protect me from royally fucking up. Postgres security defaults for example. And although this is obviously not true in reality, we can't let ourselves overlook it just because.
> "So we’ll just delete these files, kill the running process, and be done with it, right?"
No, no, no, never! You can poke around the server to figure out how you got attacked, but afterwards the only way to recover is to rebuild from scratch. Someone had root on the system. For all you know, you're currently running a recompiled version of kernel/init, which will recreate all the original scripts when you're not looking.
> Lessons Learned
I'd also add: learn about Apparmor / Selinux and use them in production. Even if redis was running as root, you could still prevent most of the attack (no file writing, no execution, etc.)
I.e. services listening on various ports. Knowing which services are running plus a firewall as an extra precaution goes a long way to avoid issues like this.
50 comments
[ 2.8 ms ] story [ 82.4 ms ] threadProbably not. The attacker would have gotten access to the machine. As there is a lot of software installed locally some probably have holes, which allows for full root access.
So I don't really believe it makes much difference for serious attacker if the attacker gets root access or normal access.
[1] - https://github.com/matiasinsaurralde/evilredis
Granted, I didn't read the original blog post. Were they distributing tools to make it easy? Or were they explaining how it worked?
In any case, if he hadn't written the blog post, someone else would have.
> "From time to time I get security reports about Redis. It’s good to get reports, but it’s odd ... in a software which is designed (as we explain in our security page here http://redis.io/topics/security) to be totally insecure if exposed to the outside world."
Regardless of the readme, the Lessons Learned should include both "know the security posture of the services you run" and "don't run anything publicly facing unless you are forced to, and only then if hardened well".
Everyone is fallible, things tend to go wrong. Insecure defaults in software hurt even expert devs (which this guy is if you look at his blog).
If your software is "designed to be totally insecure if exposed to the outside world" you should not ship it with defaults exposing it to the outside world. IOW you should not design software to be totally insecure by default.
In a well-prepared system, the administrator would:
#1 block input traffic on all the ports, except the required ones. as routine, after uninstalling any service, open ports are reviewed; even if a service is forgotten running, it won't be able to communicate.
#2 not run redis or any service as root user.
Not doing so is, to say the least, reckless, and with this assumption/context, it doesn't make much sense to theorize about software design.
Since none of these memory store solutions (memcached, redis, etc.) should be able to be connected to on public addresses it would be a nice feature to only allow them to listen on private internal IP space or sockets by default and fail to run if anything else is used.
This would help reduce everyone's attack surface by default and help prevent misconfiguration by the user or 3rd party integration software.
Of course one uses the system package manager when possible, and a specialized package manager like npm in most other cases, but not all is lost when one needs a recent version of some autotools monstrosity. In that case just install it using GNU stow. Then uninstalling is just an invocation of "stow -D" (plus deleting a single directory).
But that only makes sense for a box one physically possesses. Why on earth would one reuse a VPS? Nevertheless, I'm glad that TFA did, because the investigation was very interesting.
Um, no. You should absolutely be doing both. Defense in depth is the only way to remain secure in failure of any other component. A firewall WILL NOT PROTECT YOU FROM THAT.
For example, you just put on the firewall and leave a daemon running on localhost as root. The firewall allows connections from localhost by default (as most do). An attacker finds a hole in your webserver and is able to run scripts in the httpd context. The script makes a connection to the daemon on localhost, exploits it, and now has root privileges. With root the attacker modifies the firewall rules to allow access. You are now fully exploited.
Had your daemon been secured they would have only been a takeover of httpd and daemon accounts and not a full system compromise. Privilege separation is extremely important.
[1] https://detectify.com/
But really this is the OP's problem: he ran a system open to the internet without any apparent analysis of attack vectors. Server was compromised due to incompetence - not developer incompetence, but operations incompetence, a different discipline.
This isn't very difficult, just make it bind to localhost by default.
In the last 20 years I've been working on systems, that is the default on way too much. It's only been in very recent history that things come more secured by default. In the past popularity was strongly correlated with ability to set up. 'Works by default with no security' is really easy to setup and reduces the number of complaints that "product is too hard to use, when with $x product instead".
Either way, you should always check the open ports and listening services in a given machine. In this case for a quick check you could use: https://www.isredisallsafe.com/
Default system wide failure is a redis problem. Any modern daemon should come with configuration files that a) don't allow the application to run as root unless uncommented by user action, and b) bind to localhost. Further configuration should be necessary to break the system as thoroughly as this user did.
But I am puzzled: a running process like redis is supposed to run in a chroot, and /tmp is supposed to always be mounted with no exec, the chroot and process should have been runned with seteuid/guid redis...
I normally have backend technologies confined in a jails with pf authorizing only inbound & outbound traffic on the legitimate interface+port, and you need no ssh access in a jail.
Looks to me there is a cascade of sysadmins wrong practice.
Devops and modern "full stack devs" seems to have lost their basics.
Perhaps this is what happens when you take a "dev" with no real sysadmin experience and make him/her responsible for these machines? It's like giving my neighbor's kid with a small dirt bike the key to my Harley and sending him out on the highway -- bad things are bound to happen.
It reminds me of the common solution that many "amateur sysadmins" have when they encounter SELinux issues: turning it off. :/
process poorly packaged crash and nothing in the logs.
You disable it, see there was a warning about a poor permission and shout at selinux for making you lose days worth of works by poorly logging.
Btw, I am a dev. But I have to sometimes go in production see servers because I need to help production sysadmins troubleshoot the weirdness of todays mis-performing easy to use solutions (selinux, systemd, rsyslogd, packaging).
When you have to recompile centos or debian packages to have them support vanilla features and you are a dev, it is boring.
I remember when I first learned web development and I absolutely had a belief that the defaults would protect me from royally fucking up. Postgres security defaults for example. And although this is obviously not true in reality, we can't let ourselves overlook it just because.
[1] https://www.shodan.io/search?query=6379
We're professionals, we don't need to RTFM.
No, no, no, never! You can poke around the server to figure out how you got attacked, but afterwards the only way to recover is to rebuild from scratch. Someone had root on the system. For all you know, you're currently running a recompiled version of kernel/init, which will recreate all the original scripts when you're not looking.
> Lessons Learned
I'd also add: learn about Apparmor / Selinux and use them in production. Even if redis was running as root, you could still prevent most of the attack (no file writing, no execution, etc.)