I'm just curious, what thid he/she install (listed the php files in the github repo).
It would be nice to know what those php files do. Any help demistifying/decoding/ the php files is much appreciated.
haha no, the 1-5.php I found a long time ago (7 Yrs or something like that). I just kept it becaus I thought it was cool you could'n mail them (gmail thinks they are viruses).
The drupal 7.22 hack was for a client who against my advice wanted to choose own hosting server (from a friend). So I'm not really bothered they got hacked.
It's just cool dissecting this stuff.
I'm at work otherwise I would be wasting a good few hours deobfuscating that code. So far I've decoded two files which for the most part seem the same but the line counts are different
Edit: So far php_display seems to allow the attacker the ability to download a file. In common.php at at least.
After checking en.php (Sorry didn't have time to check more files) I found two ip's. You could use unphp to deobfuscate the code a bit but You'll have to do a lot of it by hand which should not be that hard.
I had dealt with a similar hack recently and documented it [1]. The difference is mine was in Wordpress. There was a simple file called post.php that evaled anything that was sent in the post var. Have you found out how your server was hacked in the first place ? Check your Apache logs for errors hackers are usually careless when it comes to errors or warnings.
I took the first function and decoded the first bytes of hex, which gave the infamous eval(gzinflate(base64_decode( function. Then I used http://www.whitefirdesign.com/tools/deobfuscate-php-hack-cod... to decode rest and got a group of variables with hex data that were being grouped together like this eval($xwq2ay . $xq9mar . $xb4jym . $xm0hy3); (full version available here - http://pastebin.com/7V951cRK
Decoding this hex gave me another set of preg_replace functions, which were doing the same thing pretty much. And then again the same, except two preg_replace were being called. Eventually I got something like this http://pastebin.com/JP1eukca
The hex stored in $a and $b are just a clever way of masking gzinflate(base64_decode( so I took the rest of the data, put it into the decoder and finally got to some proper code - http://pastebin.com/A0G290cE
The "IE9 Bugfix" and "IE 4 compatible" comments made me chuckle a little.
However, it looks like the page is somehow referer or IP-sensitive, since Google's cache of it goes to something intended to show popups while curling from my machine gets a fake Adobe Flash page with a nice binary to download - only 13.5KB (I only wish the real plugin was so small!) but packed and obfuscated. Nevertheless it's a pretty dismal obfuscation as I can see some strings like "qemu" and "vbox" which suggest it has VM detection. Google doesn't know its SHA-1 so there's no other public analysis of this one yet.
I don't have time right now but looks like this rabbit hole gets deeper and deeper...
Rather than trying to undo the damage (will you ever be 100% sure you caught everything?) why not create a new site (edit: as in a new VM/box/image from scratch) and import your data fresh.
If I was hacked and files were placed on my server, including a 'web shell' I would be very afraid I don't catch everything and it just gets re-hacked.
Unless this is just a pure curiosity adventure in deobfuscation... then nevermind :)
That is assuming the cracker did not go further than the website's folder.
From my own experience, when one of my sites with username www-data was hacked (default apache installation), the client-side malware JS was injected into .htaccess file and added to ALL folders www-data had write access to.
What I am saying is, assume the worst, what other data could the cracked unix account do on the system.
Sorry, I should have clarified. A new VPS/image/machine. I would not simply create a "new site" on the box that got hacked as I too would assume evil is lurking outside of apache.
Douse it with gasoline and toast some marshmallows as I spin up a new instance imo :)
Aren't tool like that vulnerable to being replaced with a version that runs the real afick and simply removes all references to anything that the attacker wants to hide?
The attacker could then arrange for any activity during the time they were active to be filtered - including the change to afick itself...
Do attackers ever try a double bluff and make an attack look like a "standard" script-kiddie attack - which might be regarded as something that can be recovered from without scrubbing the server and starting again, leaving the more sophisticated main attack in place?
[NB Been reading a lot of John le Carré recently, which probably explains the paranoia].
In all honesty, I'm not particularly sure most of the script kiddie type hackers attacking Wordpress and Drupal have the skillset to figure this out. They're picking off low hanging fruit IMO. This doesn't ring true for all of them, i.e. those expanding a global botnet, but most of them are skilled in the art of running a script and uploading files to a web directory with basic `chown` and `chmod` perms.
Oh all that is certainly possible. As a rule most of these attackers don't really know what afick is or if they do they simply don't care, most wordpress attacks are for the purposes of malware/SEO spam, so it's really a numbers game, figuring out how to quietly subvert afick is a waste of time for them.
More to the point I don't use afick as a detection system but more as a cleanup tool. Typically in most wordpress hacks (I've probably dealt with about 8-10) you'll find that the attackers will target the theme because typically if you "replace all the files on the site" you can't replace the theme, it's unique, unless you've got a clean copy from a backup (assuming you know when the hack took place) then you can't easily replace it with known good code.
But the theme is also the part of the site that changes least, so even an afick database from the first day of the site is sufficiently useful in seeing what files (php, js) have been altered.
Typically, I end up installing afick after I get called in to clean up an existing hack. If it's been hacked once, it may well get hacked again, so I install afick to make the cleanup job easier the second time around.
One problem is that a truly full cleanup of a hacked website should in theory include manual cleanup of all clients' caches (not really practical). Otherwise, malicious index.html (for example with JavaScript that sends cookies to an attacker) could remain cached by the clients forever.
That's a really interesting point, I'd never thought of that. Worse still, you could do some trickery with app cache manifests to prevent cache cleans/page refreshes from fixing it, maybe even forever.
Follow your access log and find out how those files got written/requested. in order to solve the problem you need to identify it first. you probably need to check your permissions as well, prevent apache process from writing to your website root, it should only be able to write to "sites/default/files".
Thanks for your comment.
I will advise my client to take a proper hosting instead. "Getting hacked" was only one of the many trouble we had (and I advised for other hosting in the past already).
This is fairly straightforward ratware. Is there anything in particular you wish to know about how it operates?
browser.php is an amusing one for reversing obfuscation tricks, if anyone wants practice.
You should treat the server as compromised and rebuild from metal, by the way. I know that is annoying as heck but they clearly got code execution and you can therefore assume they had root if they wanted it and that any attempts to detect whether they did are useless because their rootkit makes the box lie to you about its current state.
Running old versions of anything (unless you backport security updates of course, as the Debian project do to keep Stable and OldStable secure without potentially introducing breaking changes with a full package upgrade), particularly popular apps/libs, has this risk. As soon as a project releases a security update there is a good chance that someone out there starts looking for a way to exploit what has been fixed so that they can try use it to abuse sites that have not updated yet.
"unless you backport security updates of course, as the Debian project do to keep Stable and OldStable secure without potentially introducing breaking changes with a full package upgrade"
And that is why you need to run production systems on large well supported stable distributions, like Debian, and not DudeOS or FunkyNameOS created 18 months ago by two dudes and never updated since.
I agree, although it is worth noting that running something obscure will make you less susceptible to automated, untargeted attacks. Potentially quite a lot more vulnerable to anything targeted, though.
Definitely. And why if you roll your own packages for any reason (i.e. you need something in a more more up-to-date form than your chosen stable well-supported distribution includes) you simply must keep a close eye on the relevant projects to make sure that you don't miss an important hole filling fix.
FYI: I run Debian/Stable where I have a choice and stick with the provided versions of everything as a general rule, though I currently have nodejs, npm, and some related modules compiled from other sources.
There's a difference in severity between "will get hacked" and "has this risk". The popular web apps are targeted, and very shortly after any exploits are found, automated drive-by attacks are live across the web. The Googles are great for quickly compiling target lists as these web apps are typically pretty easy to identify - exact phrasing on the login screens etc.
Yep I last year had a locked down aws instance running just a single purpose node.js sever and I saw loads of attempts to access common web based admin pages eg phpmyadmin WordPress.
Naturally these where from Chinese or Russian ip address ranges
I've been building Drupal sites for 6 years, and I've never had a single one get hacked, even after I stopped updating them. It's far more likely that FTP with a weak password was the attack vector.
Not sure about Drupal, but Joomla gets targeted all the time. I don't think FTP is the attack vector. Yes, it's possible the password leaked, but it's more likely a bug in Drupal was exploited.
Disclaimer: I work at a hosting company, and this is my personal experience with hacked websites.
On Joomla I've seen exploits via the site search feature and the admin login (I too work at a company that does hosting). I haven't seen Drupal sites get taken out.
There is substantially more functionality, most rendered moot by code which allows the attacker to run arbitrary instructions on the server. Typically after observing that (called a "shell" in hacker parlance) one doesn't really spend much time looking at the rest of the code.
> You should treat the server as compromised and rebuild from metal, by the way. I know that is annoying as heck but they clearly got code execution and you can therefore assume they had root if they wanted it and that any attempts to detect whether they did are useless because their rootkit makes the box lie to you about its current state.
Is privilege escalation that easy/common? Thinking esp of the number of shared hosting providers out there, if a user account is compromised they don't assume the entire server is compromised.
Now ok, www-data isn't any old user account, but the same principle applies?
In theory you might be right, but in practice patio11 definitely is.
Either way, the big hassle is going to be reinstalling your site, pulling a copy of the database from backup (you have that right?). Might as well go all the way and install everything fresh.
Yes, definitely! Drupal (like Wordpress and others) stores all the "content" of your site in a database so there could be all manner of nastiness hidden in your pages.
And then, obviously, the attacker could have added an admin user account or, less obviously, altered settings stored in the database to make the site insecure.
My advice would be to restore completely from backup, if possible.
The price of restoring from old backup could be steep: some data loss. Or at least time consuming manual data merge with data from more recent, but compromised database.
It might be easier to check database for common vulnerabilities (admin accounts, suspicious content for web pages).
If you must go down that road, I'd suggest pulling the content off the compromised site and copying it manually and very carefully to the recovered site. You can never really be sure you got rid of everything otherwise.
It depends what you are running: if I have exec then I can drop to root on a lot of systems. But with these kind of scripts its usually an automated process which infects the system and that usually means nothing else was done besides infect all www files. But most likely they could have rooted and thus you need to reinstall.
> Is privilege escalation that easy/common? Thinking esp of the number of shared hosting providers out there, if a user account is compromised they don't assume the entire server is compromised.
Privilege escalation is easier than getting the initial shell. I would certainly reinstall any machine where someone has a shell.
As for these web shells, this agin demonstrates the important of blocking outbound connections with a firewall or some type of networking design.
Shared hosting providers stay in business with the principle that somehow they provide a relatively secure platform for their users, so a few of them do harden their systems enough to prevent most priv escalation. But one buffer overflow and a lack of mandatory access control later and you've got gold. Priv escalation exploits have been so common in Linux in the past decade that I would consider local access the same as root access [without MAC like grsec or selinux enabled].
Well apparmor is like the baby cousin to selinux, or the big brother to the traditional unix DAC. Selinux is more robust but also more complex. Grsec is an attempt at both a MAC and various exploit-prevention techniques. In that sense, grsec will be more effective at preventing "traditional" exploit techniques.
But modern software development is so bad, even after using high-level languages and abstractions, much of the webapp and backend development is rife with security holes. You don't need to use things like buffer overflows anymore to simply extract data or take over accounts. Servers are so easily accessible and botnets are so widespread that owning a server isn't really the point anymore; once you have all their data, who needs root?
Ya, I guess I'm thinking of it more from a hosting/cloud perspective, in that even though one user/tenant's site has been hacked, it would be nice not to have to wipe the machine for all other users.
You don't need to bother with old-school stuff like grsec, iptables, IDS, chrooted applications or any stack-protection technologies.
Get a WAF, audit your web-app source-code and use a pen-test tool regularly instead.
SQL-injections walk right in, through the front door. They stuff their pockets full of data and then leave the same way they came, unnoticed most of the time.
I am definitely going to reinvestigate putting every user's data in their own container. When I last looked the options were FreeBSD Jails (not appealing) or Solaris Zones (interesting but a whole new OS); now with LXC maybe that can assist? There must be a way to harden things further...
Docker is not designed to isolate the root user from the rest of your system, and containers/capabilities are not MAC-based security. Docker strips 16 capabilities when it runs a container, leaving 23 additional capabilities to be abused by root, and whatever other vulnerabilities are left that capabilities don't cover. To prevent this you need guest isolation via virtualization.
I think you meant "someone gets privileged code execution," which is a sensible assumption. Even still, app-permission (less than privileged) code execution can still do damage like host malware, IRC dumpsites/bot control, diodes, tor relays, vandalize web properties, etc.
The only way to know that a system is no longer owned for certain is to reimage it to a known good state. Doing anything less is tons of work, and unlikely to catch everything (rootkits, backdoors, hidden services, replaced system files, etc.). Even when running HIDS, HIDS cant be trusted because rootkits can hide things from it because it's running from the system with a possibly infected kernel. So, it turns out reimaging is less work and more trustworthy if the box is rebuilt and the 'sploit can be mitigated before bringing it online to the outside world (build and patch offline to avoid getting re-owned).
While I agree in principle that when owned one should start from scratch, my advice would be to learn something. Often I am asked to analyse attacks and I now have a collection of about 15 malware scripts that not only show me the intent but they are also useful (and remarkably well coded) for my daily admin tasks.
So, fresh start but at least get something out of it!
"and the 'sploit can be mitigated before bringing it online to the outside world"
You should read more carefully.
Also, keeping people waiting without an ETA for a down service because you're learning isn't going to result in happy customers.
Furthermore, whomever is running these boxes needs to deploy NIDS and HIDS and properly secure their boxes, because clearly they don't understand what an attack surface is.
Disagree, both in spirit and to the letter; for starters, I'm pretty sure there's still validity in a very long blog post I wrote about NIDS back in 1998:
This is partly because shared hosting providers act like Internet slumlords. Their unwillingness to spend sufficiently to secure their hardware and users' data is not sufficient evidence to conclude that they are secure. (Honestly, at $4 or whatever a month, I sort of understand.)
Anything short of formatting all the drives and reinstalling the OS will not result in a trustable machine. As they said, the attackers most likely have root access and rootkits make the machine lie about it's state. So even if you remove all your website code and replace it, they probably have a backdoor to get in and compromise it again. And you can't find that backdoor, because they modify utilities like 'ls', etc.. so that you can't detect files that were installed or modified by the rootkit.
To be truly sure you're back to normal, you would want to re-flash firmwares, as well. Which is truly a pain. Your NIC has firmware, your Motherboard has firmware, even SATA disks have firmware...
Here's a gentleman who put malicious firmware on a hard disk to bypass linux security by serving a neutered /etc/passwd file.
http://spritesmods.com/?art=hddhack
Generally, you have to choose the level of rebuild that you can live with given your likely attacker. Usually, flashing firmware is dangerous and likely to alert the operator to the infection, so most attackers interested in spam/phishing wouldn't try that approach. That is probably some three-letter-organization level stuff.
Yeah, like you hint at--this gets near Dragos levels of paranoia. Firmware based hacks are highly non-trivial; particularly considering the sophistication of the scripted attacks that target Wordpress, Drupal, et al. installations en-masse. Unless it's a very high-value target, most people are not going to go through the trouble of coming up with a custom attack for every bit of hardware they encounter (not to mention the amount of functionality you can squeeze into flash while still maintaining its usefulness as operating firmware). In the amount of time it'd take to do that, you could manually seize tons of other insecure installations.
Also, if you happen to know of any Linux utilities that can flash a live OS's HDD firmware without the system going to shit, I'd be curious to learn more.
No, having root on the VM doesn't necessarily let you own the host (I don't think I implied that), but it's been known to happen: http://en.wikipedia.org/wiki/Virtual_machine_escape. If we're dealing with attacks as sophisticated as others mentioned in the thread, I wouldn't put that outside of the realm of possibility.
edit: at the very least I'd recreate the VM from scratch.
I have been there too. When it happened to one of our clients I Googled the site URL and found some script kiddie had boasted of his antics on Twitter. A bit more Googling and I had his mum's house (he lived there, presumably there was a basement). I was wanting to take matters further, as in get the police to arrest the guy and get him prosecuted for criminal damage. However, my 'superiors' told me to just restore the backup and leave it at that.
I have to say that I was impressed by the way the hack worked, in this incident and others, I felt that I was up against a far superior adversary.
Aside from decoding the escaped characters, there's a bunch of simple regex replacements to remove all the random variable usage and then a pass through PHP_Beautifier to fix the formatting.
I used to work for a web hosting company and we saw this kind of attacks ALL the time.
Most of the cases was because of old CMS versions, but in same others the computer uploading the files was infected and the FTP credentials were stolen (Change your user/password and analyze ftp logs).
I would also check the database and do a clean install of the CMS.
The server could be compromised but I don't think this is the case.
Best answer I've seen so far. The takeaway from the guys on the front lines is usually that a full server compromise is rare and that FTP creds were stolen from a client via malware. The result is a simple drive-by that is relatively easy to clean up.
Our website and 2 of our client websites have been compromised like this in the last couple of weeks and they are all across different hosting providers (Zen Hosting and Unlimited Web hosting)
Here is a link to the code we found injected into the index page on our FTP and my attempt at decoding it.. interestingly enough it does relay to javaterm.com as the authors comprimsed site does as well..
We are fairly certain it wasn't achieved through our code as one of the sites is literally 6-7 pages of static html content.
From what we can tell it only ever effects the index page in the root of a servers FTP. In my case all of the shells were deleted(Looking from the FTP logs there were 2-3 uploaded all with different names)
104 comments
[ 0.38 ms ] story [ 215 ms ] threadHere is what main.php does: http://www.unphp.net/decode/3aaa2bc88be0e162fc3ca8786a2f8f82...
Found the following url somewhere: 78.138.127.174/2701dfbvcxff.php
Use http://ip-lookup.net/index.php to get to some abuse email addresses and inform them that the ip is involved in hacking.
Anyway this was just my quick glance, good luck!
Edit: So far php_display seems to allow the attacker the ability to download a file. In common.php at at least.
Edit : https://github.com/icambridge/help_me_clear_this_up/blob/mas... what I've deobfuscated.
125.89.44.28 <- Chinese 62.122.75.2 <- Polish
[1]: http://idiallo.com/blog/2013/11/fixing-3-year-old-hack
I took the first function and decoded the first bytes of hex, which gave the infamous eval(gzinflate(base64_decode( function. Then I used http://www.whitefirdesign.com/tools/deobfuscate-php-hack-cod... to decode rest and got a group of variables with hex data that were being grouped together like this eval($xwq2ay . $xq9mar . $xb4jym . $xm0hy3); (full version available here - http://pastebin.com/7V951cRK
Decoding this hex gave me another set of preg_replace functions, which were doing the same thing pretty much. And then again the same, except two preg_replace were being called. Eventually I got something like this http://pastebin.com/JP1eukca
The hex stored in $a and $b are just a clever way of masking gzinflate(base64_decode( so I took the rest of the data, put it into the decoder and finally got to some proper code - http://pastebin.com/A0G290cE
The code uses a curl to [removed]
html source code of the link shows another obfuscated javascript code: http://pastebin.com/1WLYMp0E
EDIT: I removed the curl link to as it might be some unpatched exploit
http://pastebin.com/zYgcjtK1
And Googling the URL there gets us to something familiar, which someone else has written up before:
http://tweetypage.com/wordpress-hacked/
The "IE9 Bugfix" and "IE 4 compatible" comments made me chuckle a little.
However, it looks like the page is somehow referer or IP-sensitive, since Google's cache of it goes to something intended to show popups while curling from my machine gets a fake Adobe Flash page with a nice binary to download - only 13.5KB (I only wish the real plugin was so small!) but packed and obfuscated. Nevertheless it's a pretty dismal obfuscation as I can see some strings like "qemu" and "vbox" which suggest it has VM detection. Google doesn't know its SHA-1 so there's no other public analysis of this one yet.
I don't have time right now but looks like this rabbit hole gets deeper and deeper...
I find it fun to reverse-engineer these sorts of things when I have the time; it's almost like a multilayered adventure game.
If I was hacked and files were placed on my server, including a 'web shell' I would be very afraid I don't catch everything and it just gets re-hacked.
Unless this is just a pure curiosity adventure in deobfuscation... then nevermind :)
From my own experience, when one of my sites with username www-data was hacked (default apache installation), the client-side malware JS was injected into .htaccess file and added to ALL folders www-data had write access to.
What I am saying is, assume the worst, what other data could the cracked unix account do on the system.
Douse it with gasoline and toast some marshmallows as I spin up a new instance imo :)
http://afick.sourceforge.net/
Then you do have a reasonable idea of what's actually been modified after the fact.
The attacker could then arrange for any activity during the time they were active to be filtered - including the change to afick itself...
Do attackers ever try a double bluff and make an attack look like a "standard" script-kiddie attack - which might be regarded as something that can be recovered from without scrubbing the server and starting again, leaving the more sophisticated main attack in place?
[NB Been reading a lot of John le Carré recently, which probably explains the paranoia].
More to the point I don't use afick as a detection system but more as a cleanup tool. Typically in most wordpress hacks (I've probably dealt with about 8-10) you'll find that the attackers will target the theme because typically if you "replace all the files on the site" you can't replace the theme, it's unique, unless you've got a clean copy from a backup (assuming you know when the hack took place) then you can't easily replace it with known good code.
But the theme is also the part of the site that changes least, so even an afick database from the first day of the site is sufficiently useful in seeing what files (php, js) have been altered.
Typically, I end up installing afick after I get called in to clean up an existing hack. If it's been hacked once, it may well get hacked again, so I install afick to make the cleanup job easier the second time around.
I spent a bit of time messing around with that approach, and came up with this: https://github.com/sgentle/hackcache
browser.php is an amusing one for reversing obfuscation tricks, if anyone wants practice.
You should treat the server as compromised and rebuild from metal, by the way. I know that is annoying as heck but they clearly got code execution and you can therefore assume they had root if they wanted it and that any attempts to detect whether they did are useless because their rootkit makes the box lie to you about its current state.
And that is why you need to run production systems on large well supported stable distributions, like Debian, and not DudeOS or FunkyNameOS created 18 months ago by two dudes and never updated since.
FYI: I run Debian/Stable where I have a choice and stick with the provided versions of everything as a general rule, though I currently have nodejs, npm, and some related modules compiled from other sources.
Naturally these where from Chinese or Russian ip address ranges
Disclaimer: I work at a hosting company, and this is my personal experience with hacked websites.
I am interested in how it operates. Is it as simple as : "It runs the remote php file and adds whatever html?" or is there more to it?
I find similar functions in all the files (error_404/http_request_custom/getUseragent/getReferer/convertIpToString/getIp)
Is privilege escalation that easy/common? Thinking esp of the number of shared hosting providers out there, if a user account is compromised they don't assume the entire server is compromised.
Now ok, www-data isn't any old user account, but the same principle applies?
Either way, the big hassle is going to be reinstalling your site, pulling a copy of the database from backup (you have that right?). Might as well go all the way and install everything fresh.
Can trojan be hiding in the database?
And then, obviously, the attacker could have added an admin user account or, less obviously, altered settings stored in the database to make the site insecure.
My advice would be to restore completely from backup, if possible.
It might be easier to check database for common vulnerabilities (admin accounts, suspicious content for web pages).
Privilege escalation is easier than getting the initial shell. I would certainly reinstall any machine where someone has a shell.
As for these web shells, this agin demonstrates the important of blocking outbound connections with a firewall or some type of networking design.
But modern software development is so bad, even after using high-level languages and abstractions, much of the webapp and backend development is rife with security holes. You don't need to use things like buffer overflows anymore to simply extract data or take over accounts. Servers are so easily accessible and botnets are so widespread that owning a server isn't really the point anymore; once you have all their data, who needs root?
You don't need to bother with old-school stuff like grsec, iptables, IDS, chrooted applications or any stack-protection technologies.
Get a WAF, audit your web-app source-code and use a pen-test tool regularly instead.
SQL-injections walk right in, through the front door. They stuff their pockets full of data and then leave the same way they came, unnoticed most of the time.
https://wiki.ubuntu.com/LxcSecurity http://www.infoq.com/news/2013/09/docker-container-security http://s3hh.wordpress.com/2013/07/19/creating-and-using-cont...
I think you meant "someone gets privileged code execution," which is a sensible assumption. Even still, app-permission (less than privileged) code execution can still do damage like host malware, IRC dumpsites/bot control, diodes, tor relays, vandalize web properties, etc.
The only way to know that a system is no longer owned for certain is to reimage it to a known good state. Doing anything less is tons of work, and unlikely to catch everything (rootkits, backdoors, hidden services, replaced system files, etc.). Even when running HIDS, HIDS cant be trusted because rootkits can hide things from it because it's running from the system with a possibly infected kernel. So, it turns out reimaging is less work and more trustworthy if the box is rebuilt and the 'sploit can be mitigated before bringing it online to the outside world (build and patch offline to avoid getting re-owned).
So, fresh start but at least get something out of it!
You should read more carefully.
Also, keeping people waiting without an ETA for a down service because you're learning isn't going to result in happy customers.
Furthermore, whomever is running these boxes needs to deploy NIDS and HIDS and properly secure their boxes, because clearly they don't understand what an attack surface is.
Defense in-depth, every little bit helps.
http://insecure.org/stf/secnet_ids/secnet_ids.html
People running SAAS apps probably shouldn't waste much time with NIDS.
Here's a gentleman who put malicious firmware on a hard disk to bypass linux security by serving a neutered /etc/passwd file. http://spritesmods.com/?art=hddhack
Generally, you have to choose the level of rebuild that you can live with given your likely attacker. Usually, flashing firmware is dangerous and likely to alert the operator to the infection, so most attackers interested in spam/phishing wouldn't try that approach. That is probably some three-letter-organization level stuff.
Also, if you happen to know of any Linux utilities that can flash a live OS's HDD firmware without the system going to shit, I'd be curious to learn more.
If this is the case, then EC2 wouldn't work. Anyone with a VM could own the host (since they have root on their own VM?).
edit: at the very least I'd recreate the VM from scratch.
As others have noted, a compromised shell can never be trusted again and you should re-deploy from scratch.
I have to say that I was impressed by the way the hack worked, in this incident and others, I felt that I was up against a far superior adversary.
Edited to include variable replacement. I think there are some catches with things like ${sgasklgna} but it largely works. Just needs prettifying.
Aside from decoding the escaped characters, there's a bunch of simple regex replacements to remove all the random variable usage and then a pass through PHP_Beautifier to fix the formatting.
Most of the cases was because of old CMS versions, but in same others the computer uploading the files was infected and the FTP credentials were stolen (Change your user/password and analyze ftp logs).
I would also check the database and do a clean install of the CMS.
The server could be compromised but I don't think this is the case.
google reveals several posts about this one
Our website and 2 of our client websites have been compromised like this in the last couple of weeks and they are all across different hosting providers (Zen Hosting and Unlimited Web hosting)
http://pastebin.com/PkJFTeGs
Here is a link to the code we found injected into the index page on our FTP and my attempt at decoding it.. interestingly enough it does relay to javaterm.com as the authors comprimsed site does as well..
We are fairly certain it wasn't achieved through our code as one of the sites is literally 6-7 pages of static html content.
From what we can tell it only ever effects the index page in the root of a servers FTP. In my case all of the shells were deleted(Looking from the FTP logs there were 2-3 uploaded all with different names)