It took them 21 days to announce this? It's a in-depth post and kudos to them for that, but I'd also like to know _when_ the actual intrusion was (on July 9th when they detected it? Months before?)
Who cares? It was compromised, and customers weren't notified for 21 days.
If this were some highly regulated banking/securities/government system, 21 days MIGHT be ok for public notice, but not for an infrastructure tool with highly technical users.
People get hacked (not good), but responding badly is the true badness.
And now additionally we're in WTF hell because no one knows if linode was compromised meaningfully (again), or if it was just a PagerDuty credential which got compromised.
That's an easy bet, if it was AWS there will be much higher visibility targets than pagerduty, the fact that that is not the case makes it seem as if betting against you at this point is a losing proposition.
According to Julius Kivimaki (whom has hacked Linode countless times, including under the Hack the Planet hacker collective), Linode is pretty much an open door for blackhats.
It seems like if the compromise were due to a failure on the provider's side, rather than the more common stolen/phished credentials, PagerDuty would be more clear about laying fault where it's due. It would help clarify that they didn't directly make an error, and allow them to put the ball in the provider's court to respond.
Instead, the post's comment makes a reference to "an administrative panel provided by one of our infrastructure providers". It's not clear if this refers to the provider's internal tool, or the administrative panel they expose to customers for administering their accounts. If the former: I'd prefer they name the provider. If the latter, it might be worth clarifying the wording so it's more clear.
Here is my guess.. Hacker notified them with proof of the hack on the 9th. Gave them a few weeks for a payout and Pagerduty wouldn't payout so the hacker will then release. So to jump out ahead Pagerduty sent out the email.
Most cases like this the company claims they found the hacker but that isn't the case.
I appreciate the level of detail that PagerDuty went into with this announcement.
I presume they wanted to have all of the facts before they notified customers, but it is totally unacceptable that they waited 3 weeks to notify me about an incident with confirmed external intrusion and confirmed theft of customer data (including my own).
I'd sort of hope they weren't using PHP's built-in crypto or any crypto library that deals in null-terminated strings. Judging from their careers pages, they use no PHP, so this shouldn't apply.
Andrew Miklas wrote in a comment:
The attacker gained unauthorized access to an administrative panel provided by one of our hosting providers.
There have been several breaches in the last months where this was the main cause and it's something almost impossible to defend against – unless you're running your own datacenter hardware, which is very hard to get right.
Few providers properly secure their control panels with 2FA, even though these admin panels are an attractive target and almost always provide full access to the system.
Even 2FA as commonly implemented is kind of a bullshit solution (it's once per 30 days, it's not on every transaction, it's not specific to the transaction, there's no enduring log, no concept of segregating various types of action, no concept of callback auth/validation, often not multiuser/shared credentials, etc.)
Consumer-style 2FA protects somewhat against brute force, against password reuse by users who just don't care, and against theft of a whole password list. It doesn't actually protect the protected resource very much.
PHP. This is a known weakness in PHP's bcrypt implementation. From Wikipedia, "Many implementations of bcrypt truncate the password to the first 72 bytes." I would hope that they're using a competent implementation that either supports longer passwords or throws an error if it's asked to hash a longer password.
Besides, bcrypt takes the salt as a separate parameter anyway. So it doesn't really make sense in context that the salt, pepper, and password were concatenated and passed to bcrypt. Perhaps he was talking about the SHA-1 stretching; perhaps they hash the passwords with SHA-1 before passing it to bcrypt.
I don't think we know enough to conclude that they were definitely doing it wrong, but it would be nice to know more details about the algorithm, though.
> PHP. This is a known weakness in PHP's bcrypt implementation. From Wikipedia, "Many implementations of bcrypt truncate the password to the first 72 bytes." I would hope that they're using a competent implementation that either supports longer passwords or throws an error if it's asked to hash a longer password.
Actually, it's a known weakness in BCRYPT. PHP did not implement bcrypt, it was ported in via crypt(3). Meaning that ALL versions of bcrypt have this issue.
Some implementations error on > 72 bytes, but NONE of them accept longer passwords.
> I don't think we know enough to conclude that they were definitely doing it wrong, but it would be nice to know more details about the algorithm, though.
Given what has been shared so far, there's enough signs pointing that the chances are pretty high they did something wrong. 40 byte salt? Bcrypt only supports a 128 bit salt. So either they did something silly custom (at which point it's no longer bcrypt), they aren't actually using bcrypt, or they did something silly like concatenate the salt + pepper + password and pass it to the password field.
I'm happy with OpsGenie [0] - not so expensive, fully-featured, and now - safer compared to the overpriced PagerDuty. OpsGenie first implemented push notifications while PagerDuty was relying on people not knowing the competitive landscape and had a web-based Android app with SMS for years. OpsGenie innovates now and PagerDuty follows. Research and don't go with the flow!
I love OpsGenie too, but this isn't really the best place for that discussion. Maybe you should write up your positive experiences with OpsGenie as a blog post and submit it to HN; particularly in the context of how to set up a notification system for ops, rotation, etc.
I will, but what's interesting is that I keep getting bugged by PagerDuty sales people and today, just before I read this, I got an email with their "improved" pricing and "improved" service! Had to vent out, sorry!
All of these services seem to take the same approach: AWS US-East, maybe use two availability zones, call it a day.
PagerDuty had to learn this lesson the hard way (circa 2012), as they were entirely in US-East during the catastrophic AWS outage in June. They proceeded to go offline along with a lot of their customers, who subsequently were not alerted.
Multiple region redundancy is not trivial, but it would certainly be a little higher up on my to-do list if it was basically the entire value proposition of my service. I was a little suspicious of PagerDuty's operational chops before, but now I simply don't trust them.
This is why PagerDuty needs a decentralized solution for customers. If products like Chef can have both Hosted Chef and Chef Server and Atlassian can have both both Hosted and Self-hosted options there's no reason they can't have the same so you don't have to put your employees potentially sensitive on-call and server incident alert history in one basket.
31 comments
[ 2.0 ms ] story [ 76.9 ms ] thread> Within hours of the start of the intrusion, we were able to detect and remove the attacker, and shut down the attack.
If this were some highly regulated banking/securities/government system, 21 days MIGHT be ok for public notice, but not for an infrastructure tool with highly technical users.
People get hacked (not good), but responding badly is the true badness.
And now additionally we're in WTF hell because no one knows if linode was compromised meaningfully (again), or if it was just a PagerDuty credential which got compromised.
https://support.pagerduty.com/hc/en-us/articles/202830570-Wh...
Instead, the post's comment makes a reference to "an administrative panel provided by one of our infrastructure providers". It's not clear if this refers to the provider's internal tool, or the administrative panel they expose to customers for administering their accounts. If the former: I'd prefer they name the provider. If the latter, it might be worth clarifying the wording so it's more clear.
Most cases like this the company claims they found the hacker but that isn't the case.
I presume they wanted to have all of the facts before they notified customers, but it is totally unacceptable that they waited 3 weeks to notify me about an incident with confirmed external intrusion and confirmed theft of customer data (including my own).
It's also applicable to crypt(3)...
It has nothing to do with PHP...
There have been several breaches in the last months where this was the main cause and it's something almost impossible to defend against – unless you're running your own datacenter hardware, which is very hard to get right.
Few providers properly secure their control panels with 2FA, even though these admin panels are an attractive target and almost always provide full access to the system.
Consumer-style 2FA protects somewhat against brute force, against password reuse by users who just don't care, and against theft of a whole password list. It doesn't actually protect the protected resource very much.
From their CTO (on Disqus in response to my question):
"The salt, pepper, and password were concatenated together to form the string that was in turn passed to the hashing function."
http://3v4l.org/cE359
40 + 40 + strlen($password) > 72
Uh-oh.
PHP. This is a known weakness in PHP's bcrypt implementation. From Wikipedia, "Many implementations of bcrypt truncate the password to the first 72 bytes." I would hope that they're using a competent implementation that either supports longer passwords or throws an error if it's asked to hash a longer password.
Besides, bcrypt takes the salt as a separate parameter anyway. So it doesn't really make sense in context that the salt, pepper, and password were concatenated and passed to bcrypt. Perhaps he was talking about the SHA-1 stretching; perhaps they hash the passwords with SHA-1 before passing it to bcrypt.
I don't think we know enough to conclude that they were definitely doing it wrong, but it would be nice to know more details about the algorithm, though.
Actually, it's a known weakness in BCRYPT. PHP did not implement bcrypt, it was ported in via crypt(3). Meaning that ALL versions of bcrypt have this issue.
Some implementations error on > 72 bytes, but NONE of them accept longer passwords.
> I don't think we know enough to conclude that they were definitely doing it wrong, but it would be nice to know more details about the algorithm, though.
Given what has been shared so far, there's enough signs pointing that the chances are pretty high they did something wrong. 40 byte salt? Bcrypt only supports a 128 bit salt. So either they did something silly custom (at which point it's no longer bcrypt), they aren't actually using bcrypt, or they did something silly like concatenate the salt + pepper + password and pass it to the password field.
[0] http://opsgenie.com/
PagerDuty had to learn this lesson the hard way (circa 2012), as they were entirely in US-East during the catastrophic AWS outage in June. They proceeded to go offline along with a lot of their customers, who subsequently were not alerted.
Multiple region redundancy is not trivial, but it would certainly be a little higher up on my to-do list if it was basically the entire value proposition of my service. I was a little suspicious of PagerDuty's operational chops before, but now I simply don't trust them.