Today is the day that tptacek gets to run around screaming "I told you so" :-).
I don't remember when or why I created it, but I have an account on the compromised server. It was a non-unique password, but fortunately one that I've never used on any account of value, and one that I rotated out a while ago on every account that I ever use at all.
We are assuming that the attackers have a copy of the JIRA database, which includes a hash (SHA-512 unsalted) of the password
you set when signing up as '...' to JIRA. If the password you set was not of great quality (eg. based on a dictionary word), it
should be assumed that the attackers can guess your password from the password hash via brute force.
The upshot is that someone malicious may know both your email address and a password of yours.
This is a problem because many people reuse passwords across online services. If you reuse passwords across systems, we urge you to change
your passwords on ALL SYSTEMS that might be using the compromised JIRA password. Prime examples might be gmail or hotmail accounts, online
banking sites, or sites known to be related to your email's domain, gmail.com.
I also got the same letter. But I haven't used it in years. Are you supposed to use your email to log in? I need to find out which password I was using in this account.
* No lockout / notification after hundreds of thousands of failed logins
* XSS vulnerability
* Able to change the configuration to upload executable scripts
* SSH password authentication enabled
And your top concern was that Slicehost didn't immediately shut down a machine? Would it have really slowed the hackers down for more than an hour anyway?
fwiw, 4/5 of your points are specific issues to Atlassian JIRA, a commercial product :)
The 5th point, as explained in the post... We actually tried to disable Password based authentication -- It is the ASF's standard policy for it to be disabled! -- we just failed at testing the configuration on Brutus. On our other machines, we had a block of configuration options we appended to the END of the sshd configuration, and these options successfully turned off password authentication, requiring ssh keys only.
On brutus however, we were missing a "UsePAM no", which meant pam went in and fell back to password authentication. This was a mistake, and we didn't realize until after the vulnerability when we were testing the machine, that its sshd was misconfigured.
I didn't mean to criticize. There were mistakes made - there always are. But amongst the list of mistakes, I would not put Slicehost's slow response anywhere near the top, and I'm surprised that anyone (particularly someone whose opinions I normally respect) could choose that as the lesson to highlight.
Anyway, when is Apache going to get serious about a bug tracking project? :-)
* Able to change the configuration to upload executable scripts
Not familiar with JIRA but... couldn't you prevent this by either changing the ownership of the uploaded files or configure the web server not to execute any files in uploaded files directory.
I dislike the implicit presumption that people on the Internet owe it to you to shut down their server if (you say) they're attacking you. I strongly feel that you should presume that your servers are always being attacked, and it's not helpful to even think that you might be able to get attackers shut down.
Despite the fact that two days later it was used in another hack?
Were I a server provider I would take such notifications very seriously; a trivial investigation would surely have uncovered the illegal activity and shut them down.
(we are seeing more and more use of services such as slicehost and EC2 to perform attacks. I doubt that is a tag those companies want to carry!)
There are hosting providers who openly advertise servers for sending spam illegally. The idea that you can achieve security by shutting down your attackers is ridiculous. Had slicehost acted more rapidly, Atlassian would have been hacked by the same hackers using a different server.
Obviously Atlassian knows this also, because they didn't even bother putting an IP blocking rule into their firewalls. If Atlassian didn't even think that worthwhile, pointing the finger at Slicehost is dubious.
I hope Slicehost was simply going through their own process; notifying the customer and helping them to resolve the issue: the server may have been running a business that is multiple peoples' livelihoods. Those processes take time, and they have to balance the damage you're doing to the 'target' victim vs the damage to the 'compromised host' victim. Given that shutting down the compromised host wouldn't save the target anyway, I would also have prioritized my own customer.
Security 101 we teach clients: if you have a compromised server take it down. It's too risky to do anything else really.
If it was running a clients services they are compromised too and if that is the case, and Slicehost leaves it compromised, that is a serious liability problem.
There is a reasonably high likelihood it was a slice set up specifically to launch attacks (though the blog post does say compromised, so perhaps they have more info on the slice in question than has been made public so far). In which case leaving it running is, again, unnecessary liability.
I think tptacek and I are just surprised this didn't concern Slicehost as much as it, probably, should :)
But what if I phone up your hosting provider and tell them that your webserver is attacking my site? Should they shut it down immediately before notifying you? How long should they wait for a reply from you? How many checks should they do? We're not talking about instantly recognizable traffic here; this was likely HTTPS traffic. Should the hosting company have circumvented your access controls and looked into your VM?
My understanding is that Slicehost is responsible for the VM 'hardware', and you're responsible for what goes on inside it. There are exceptions for illegal behaviour, but I'm sure those involve checks and balances and processes, probably for legal reasons as well as for simple reasons of fairness.
As I suggested I doubt they sent them a quick mail :) I would have hoped they provided detail and logs to show what happened.
We're not talking about instantly recognizable traffic here; this was likely HTTPS traffic.
It's trivial enough to spot if you review the logs and the information available. Anyone with an ounce of security knowledge would spot it very quickly.
Should the hosting company have circumvented your access controls and looked into your VM?
They do this anyway; as root on the dedicated machine they have access to your VM anyway :)
Breaking into someone else's machine is not an illegality on par with facilitating spam. It's crazy to me how inured we've all become to breakins. Breakins are a big deal.
Most of the things you listed are issues with JIRA and Confluence, not the ASF's practices. Everything else on your list has already been fixed.
The ASF infra group did almost everything right but they still got burnt by esoteric details. The sshd conf had PasswordAuthentication set to no, but enabling PAM auth caused regular password auth to work. Did you know about that caveat?
And your top concern was that Slicehost didn't immediately shut down a machine?
tptacek never said that. He said it was a little disturbing that Slicehost took two days to turn off an instance that was exploiting a previously-unknown vulnerability in JIRA.
We notified Atlassian of the previously unreported XSS attack in JIRA and contacted SliceHost. Atlassian was responsive. Unfortunately, SliceHost did nothing and 2 days later, the very same virtual host (slice) attacked Atlassian directly. ( http://blogs.atlassian.com/news/2010/04/oh_man_what_a_day_an... )
Apache's infrastructure group got their shit together quickly. The hacked Slicehost instance wasn't a threat to them anymore. Slicehost's inaction helped get Atlassian hacked, which I think is a bit disturbing.
Would it have really slowed the hackers down for more than an hour anyway?
Oh, you're right. Let's just leave hacked systems running. Shutting them down will only slow down hackers for an hour.
Wow - you're certainly living up to your username :-)
Shutting down a server doesn't stop the hackers using one of the other thousands of servers they have compromised. In this regard, trying to respond to an attack by shutting down the server is essentially a waste of time: the only winning move is not to play.
There were multiple mistakes made (and I'm not trying to cast blame), and I think we should be aware of the repeatable lessons for software development: hashes need to be salted, XSS matters, password lockout matters etc. If there's a lesson from Slicehost, it's that you shouldn't rely on someone else bailing you out.
Yes. Really. It bothers me as a Slicehost customer that Slicehost didn't get on top of a report that one of their VMs was compromised. Those VMs are on the same local networking hardware, and perhaps on the same server hardware, as my own VMs.
I'm unclear why I'm supposed to care about Atlassian. I don't use Atlassian products.
OK, serious question to you as a security guru: Shouldn't we always assume that every host outside of our control is compromised? So why should I care how long it takes someone to shut down a compromised host?
I think there are lessons we should all learn / double check, but I don't understand why you chose to highlight this issue.
No, that doesn't mean hosting providers can ignore breakins on their own hardware.
Seriously, dude: breaks aren't like unfortunate Internet weather incidents. They're crimes. An actual crime took place using Rackspace computers. How do we even know what kind of forensics could have been conducted if Slicehost can't even acknowledge when their machines are compromised? Hosting providers have special responsibilities.
OK, point taken. But you're assuming that Slicehost ignored the issue, rather than being bound by their own process.
I simply expected more from you: as dfranke said, this is your opportunity to hammer home the importance of good security practice, and instead you're talking about something basically beyond our control (Slicehost's incident response policy) rather than the root cause (a large number of fairly basic security mistakes)
OK - so there's another lesson I have to learn from you :-)
Please, publish a blog post series on each of the mistakes made, explaining each one and how not to fall into the trap. Explain why slicehost taking 48 hours to notify a customer before shutting down their machine is a big problem if it belongs in the list. I'll upvote every post once it hits HN.
To me this seems pretty frikin huge. The number of sysadmins and developer's using bugzilla, jira, confluence... I would assume a good number of them are using very crackable passwords (common words etc.), even if it's just a minority, which could then expose countless other systems managed by each of those sysadmins / devs who had their password hacked, because they probably also use a rotation of 1-3 passwords for other services...
I always find it fascinating how fast security falls when you find the one weak spot. They were lucky to spot them so fast (relatively) I'm sure minotaur would have fallen with time too.
This wasn't the case of a single weak spot. There where numerous issues with their security setup on this server. See http://news.ycombinator.com/item?id=1262784 for a short list.
"The attack was crafted to steal the session cookie from the user logged-in to JIRA. When this issue was opened against the Infrastructure team, several of our administators clicked on the link. This compromised their sessions, including their JIRA administrator rights."
I may be wrong here, but isn't using a session cookie to authenticate a login which allows one to upload files to an executable directory a bad idea?
Wouldn't asking for the user's password again when the he needs higher privileges (a la popular bulletin board software) stop the whole attack in its tracks?
37 comments
[ 3.6 ms ] story [ 81.5 ms ] threadI don't remember when or why I created it, but I have an account on the compromised server. It was a non-unique password, but fortunately one that I've never used on any account of value, and one that I rotated out a while ago on every account that I ever use at all.
You are receiving this email because you have a login, '...', on the Apache JIRA installation, https://issues.apache.org/jira/
On April 6 the issues.apache.org server was hacked. The attackers were able to install a trojan JIRA login screen and later get full root access:
https://blogs.apache.org/infra/entry/apache_org_04_09_2010
We are assuming that the attackers have a copy of the JIRA database, which includes a hash (SHA-512 unsalted) of the password you set when signing up as '...' to JIRA. If the password you set was not of great quality (eg. based on a dictionary word), it should be assumed that the attackers can guess your password from the password hash via brute force.
The upshot is that someone malicious may know both your email address and a password of yours.
This is a problem because many people reuse passwords across online services. If you reuse passwords across systems, we urge you to change your passwords on ALL SYSTEMS that might be using the compromised JIRA password. Prime examples might be gmail or hotmail accounts, online banking sites, or sites known to be related to your email's domain, gmail.com.
* Passwords hashed with SHA-512 _unsalted_
* No lockout / notification after hundreds of thousands of failed logins
* XSS vulnerability
* Able to change the configuration to upload executable scripts
* SSH password authentication enabled
And your top concern was that Slicehost didn't immediately shut down a machine? Would it have really slowed the hackers down for more than an hour anyway?
The 5th point, as explained in the post... We actually tried to disable Password based authentication -- It is the ASF's standard policy for it to be disabled! -- we just failed at testing the configuration on Brutus. On our other machines, we had a block of configuration options we appended to the END of the sshd configuration, and these options successfully turned off password authentication, requiring ssh keys only.
On brutus however, we were missing a "UsePAM no", which meant pam went in and fell back to password authentication. This was a mistake, and we didn't realize until after the vulnerability when we were testing the machine, that its sshd was misconfigured.
Anyway, when is Apache going to get serious about a bug tracking project? :-)
Not familiar with JIRA but... couldn't you prevent this by either changing the ownership of the uploaded files or configure the web server not to execute any files in uploaded files directory.
But if someone phones you up and says we were hacked using one of your slices you would hope they would at least suspend access and investigate.
In computer security an hour can be a long time.
Were I a server provider I would take such notifications very seriously; a trivial investigation would surely have uncovered the illegal activity and shut them down.
(we are seeing more and more use of services such as slicehost and EC2 to perform attacks. I doubt that is a tag those companies want to carry!)
who said anything about owing it to them? :-)
Obviously Atlassian knows this also, because they didn't even bother putting an IP blocking rule into their firewalls. If Atlassian didn't even think that worthwhile, pointing the finger at Slicehost is dubious.
I hope Slicehost was simply going through their own process; notifying the customer and helping them to resolve the issue: the server may have been running a business that is multiple peoples' livelihoods. Those processes take time, and they have to balance the damage you're doing to the 'target' victim vs the damage to the 'compromised host' victim. Given that shutting down the compromised host wouldn't save the target anyway, I would also have prioritized my own customer.
If it was running a clients services they are compromised too and if that is the case, and Slicehost leaves it compromised, that is a serious liability problem.
There is a reasonably high likelihood it was a slice set up specifically to launch attacks (though the blog post does say compromised, so perhaps they have more info on the slice in question than has been made public so far). In which case leaving it running is, again, unnecessary liability.
I think tptacek and I are just surprised this didn't concern Slicehost as much as it, probably, should :)
My understanding is that Slicehost is responsible for the VM 'hardware', and you're responsible for what goes on inside it. There are exceptions for illegal behaviour, but I'm sure those involve checks and balances and processes, probably for legal reasons as well as for simple reasons of fairness.
We're not talking about instantly recognizable traffic here; this was likely HTTPS traffic.
It's trivial enough to spot if you review the logs and the information available. Anyone with an ounce of security knowledge would spot it very quickly.
Should the hosting company have circumvented your access controls and looked into your VM?
They do this anyway; as root on the dedicated machine they have access to your VM anyway :)
There are exceptions for illegal behaviour
This is illegal behaviour isn't it :)
The ASF infra group did almost everything right but they still got burnt by esoteric details. The sshd conf had PasswordAuthentication set to no, but enabling PAM auth caused regular password auth to work. Did you know about that caveat?
And your top concern was that Slicehost didn't immediately shut down a machine?
tptacek never said that. He said it was a little disturbing that Slicehost took two days to turn off an instance that was exploiting a previously-unknown vulnerability in JIRA.
From the incident report https://blogs.apache.org/infra/entry/apache_org_04_09_2010 :
We notified Atlassian of the previously unreported XSS attack in JIRA and contacted SliceHost. Atlassian was responsive. Unfortunately, SliceHost did nothing and 2 days later, the very same virtual host (slice) attacked Atlassian directly. ( http://blogs.atlassian.com/news/2010/04/oh_man_what_a_day_an... )
Apache's infrastructure group got their shit together quickly. The hacked Slicehost instance wasn't a threat to them anymore. Slicehost's inaction helped get Atlassian hacked, which I think is a bit disturbing.
Would it have really slowed the hackers down for more than an hour anyway?
Oh, you're right. Let's just leave hacked systems running. Shutting them down will only slow down hackers for an hour.
Shutting down a server doesn't stop the hackers using one of the other thousands of servers they have compromised. In this regard, trying to respond to an attack by shutting down the server is essentially a waste of time: the only winning move is not to play.
There were multiple mistakes made (and I'm not trying to cast blame), and I think we should be aware of the repeatable lessons for software development: hashes need to be salted, XSS matters, password lockout matters etc. If there's a lesson from Slicehost, it's that you shouldn't rely on someone else bailing you out.
I'm unclear why I'm supposed to care about Atlassian. I don't use Atlassian products.
I think there are lessons we should all learn / double check, but I don't understand why you chose to highlight this issue.
No, that doesn't mean hosting providers can ignore breakins on their own hardware.
Seriously, dude: breaks aren't like unfortunate Internet weather incidents. They're crimes. An actual crime took place using Rackspace computers. How do we even know what kind of forensics could have been conducted if Slicehost can't even acknowledge when their machines are compromised? Hosting providers have special responsibilities.
I simply expected more from you: as dfranke said, this is your opportunity to hammer home the importance of good security practice, and instead you're talking about something basically beyond our control (Slicehost's incident response policy) rather than the root cause (a large number of fairly basic security mistakes)
Please, publish a blog post series on each of the mistakes made, explaining each one and how not to fall into the trap. Explain why slicehost taking 48 hours to notify a customer before shutting down their machine is a big problem if it belongs in the list. I'll upvote every post once it hits HN.
- or -
Am I just on crack?
The XSS exploit was like the stopper in the dam.
I may be wrong here, but isn't using a session cookie to authenticate a login which allows one to upload files to an executable directory a bad idea?
Wouldn't asking for the user's password again when the he needs higher privileges (a la popular bulletin board software) stop the whole attack in its tracks?