We are not Equifaxes customers. We are Panera Bread's customers. There is some risk if you directly expose the customers. There is less risk if you lose a third parties data.
Just from speaking to my friends who are not tech people and regular Panera Bread customers. They don't care. Sorry for using this language but as a direct quote one of them said "dude who gives a shit, everyone is leaking shit these days."
Making upfront, definitively costly investments in order to avoid potential negative future consequences, is a hard one to justify.
Especially when the future consequences barely exist. It's very rare for a breach to have serious impact on a company, relative to other areas the company could invest.
Being charitable, I think the more likely explanation is that CTOs/CFOs do not understand how easily these things can happen, and how much data can be exposed by a "small" mistake.
One line of code can expose 30,000,000 records. That's hard to get your head around if you are not a programmer.
We get around this by limiting our databases to only 1000000 records per codebase. Then we duplicate the codebase 30 times (with tweaks) for 30000000 records.. but at least in the retrospective we'll say there was 30 lines of code at fault, not just 1
From the description, this wasn't one line of code, this was a major design oversight. Either nobody is security conscious on the dev team handling their mobile/web services, or it's just one guy with no code review on major components, or people who complained were tuned out, ignored, placed in low priority, or some other totally irresponsible managerial action.
You have to have a lot of bad process in place for something like this to get in.
You would think so.... but I've seen big companies with security type departments, who operate in the tech industry, that have executives that represent them well.... and they spend a lot of time explaining to other executives why they shouldn't sue that security researcher who just did us a favor.... or why engineering really should maybe fix that bug rather than put it off...
Even in companies where good people try to do the right thing security fails not just like this case where they just chose not to act, but also because nobody else at the company cares / is knowledgeable enough to care.
Forget complicated questions about corporate policy, the costs of verifying security, the limited negative impact that companies seemingly face when problems like this happen. Those questions are definitely interesting, but they're hard questions and I'm not in the mood for hard thinking right this second.
What blows my mind here is the actions of a single person. The Security Director got an email about a dead simple vulnerability in his company's website/api. All he had to do was paste a single link into a browser to verify that there was a big big problem. And he did nothing?
I simply can't understand that.
What was this guy doing every day? Did he have any sense of professional self respect at all? Did he think it would just....go away? It's so confusing.
Do you think he....didn't know how to decrypt the PGP encrypted description? And that he was too embarrassed to say so? In a weird way that's my most charitable explanation.
Yeah. He probably googled "How to generate a PGP key", then he generated it, sent it and then he was not able to decrypt it or didn't do it, possibly out of laziness... But also given the current state of the website they probably don't even know how those APIs work either.
While I am sure that there are clueless people that have worked on the security team at Equifax, I'm willing to bet that there have been good people too. It's hard to sort out the good from the bad and what problems are individual and what were systemic.
Putting a person, who majored in music theory, into the Information Security Officer position at a company of Equifax's size, shows how much they actually regard your information as something important.
I actually dislike bringing this up, because I respect the people the have gone through music education; while it doesn't pay in terms of salary, it's certainly something they are passionate about and love dearly. But just as they would not hire me to direct an orchestra, I would not hire them to secure critical Financial systems. This isn't to say there is a fault with them, it's just to say that we all have our strengths in certain areas.
> Do you think he....didn't know how to decrypt the PGP encrypted description? And that he was too embarrassed to say so?
The first part is OK, he is a director, not a grunt. The second is harder to understand. He should've handed the entire thing over, starting with the PGP thing to an actual security knowing engineer.
Or he could google it. I don't know how to PGP decrypt something off the top of my head, but I'm confident that 5 minutes of googling would lead me to any number of web pages to show me how.
That's not an efficient use of his time. The first answer instead of that adversial crap should've been "Let me introduce you to my colleague John Doe who will be handling this matter. John, a mutual acquaintance introduced Dylan to me as a reputable security researcher, please handle his report. -Signature." He is a director, he needs to delegate ferociously. Unless it's organizing, strategizing etc it's below his concern.
I have been on the receiving end of such emails a bazillion times in my carrier. My boss or the boss of my boss is certainly not going to bother with such small affairs. It is my duty to inform him when it turns out to be a big affair then he can begin to coordinate with engineering, marketing and so on to do the release. That's his job.
But he didn't do any of this. If you are CSO or CISO or director, you dang well better know what gpg is, and how to quickly find someone in the organization who knows how to deal with it.
The whole problem is that there has to be a competent engineer who does not want to use this knowledge to get famous. See, a Security Director would love to see a direct report of their inadequacy buried... if someone knows then it is harder to bury.
You are assuming this was merely a technical issue that went unnoticed, but in situations like this it is of course possible that there are other issues with process and governance that are coming into play.
If you want something from a corporation, the best course of action is generally to just write (i.e. snail mail) to the CEO, because they are the only person you can be certain is capable of re-marshaling resources to deal with the root cause of the problem, whatever it may be.
> ...it is of course possible that there are other issues with process and governance that are coming into play.
Likely, but shouldn’t the director of information security be given more power to take action? I’d think that if that was my title for an outfit like Panera, and if I couldn’t enact any kind of change in face of a fairly serious vulnerability in the span of eight months, I would resign from said position. Because what’s the point of my job, if that’s what I’m dealing with?
The @ symbol reminded me... not sure if it's still true, but about a year ago you could create a password that contained an @ symbol but you couldn't login with it in their mobile app.
Also, sometimes when I give them my phone number for my loyalty card it works, and sometimes it doesn't... an alarming number of times it doesn't work, all of which makes me think they have some questionable IT practices going on. I should have seen this coming.
"No, Panera Bread Doesn’t Take Security Seriously"
I didn't expect this to be an actual description of a security event, but just a rhetorical observation: of course Panera Bread doesn't take security seriously. There's no "security" in the name of their company. They are not in the security business. I think they do actually take bread seriously. And store location, and customer service, and stuff like that, because that's the kind of business they are.
Because of PCI you can expect they probably do handle your credit card (except apparently the last 4) reasonably well. Because of other regulations you can expect they take food safety seriously. They take basic business operations seriously because there's a bunch of professional business-runners, and they know they really have to.
I'm no expert, but PCI requires that companies protect stored cardholder data, which includes cardholder names. They have badly failed to do so, so they don't appear to be PCI compliant.
This is actually an interesting case from a PCI perspective. PCI doesn't protect last 4 of a credit card number or names individually, as you wouldn't be able to link name to card number if stored separately. Name is only 'cardholder data' when stored with a full cc number. In this case both name and last 4 are stored and revealed together, but I still don't think that constitutes PCI protected information according to their definition of cardholder data. My initial interpretation is that they wouldn't be in breach of PCI from just the information we have publically available about this issue.
We can't expect that because they didn't. Rules and best practices do not magically get followed because they exist, and in this case they definitely did not take security (of their customer data) seriously at all.
The title is intended to be a play on the most common response companies make when they experience a serious security failure: "We take security very seriously."
We should start a blog called We Take Security Seriously that links to these responses, just like Our Incredible Journey[1] for acquisition announcements.
No they don't. They admit on record using same additives as Subway mostly a rubber-type chemicals invented by BASF that make bread more elastic, won't go dry this quick and has longer shell-life. Basically when you eat their bread some ingredients are the same of the tires your car was put on!
Further good read how horrible quality their food is:
You can't be serious about that link. It is a quack website that claims that wheat and sugar are harmful additives! It also uses the number of ingredients as a good metric of the healthiness of food.
Exactly right. What do they know about it? They have an expensive firewall, and SSL on their website, they think they are covered.
Companies like this would be better off not on the internet at all. It's not like anyone needs to visit the website to know where the nearest Panera Bread is. Google will tell you, if you don't know.
It would appear that they have a very serious systemic security problem in their organization. Their "Director of Information Security" is obviously very very incompetent, there wasn't any real resolution after many months even though they claimed it was fixed. There's no way that a single person or even just a handful of managers could be solely to blame for this.
If this is how they respond to security issues I think it's a fair bet that they're making the same mistakes that e.g. Target and Home Depot did. Who's to say a VPN connection for some lowest bidder third party vendor hasn't already been used to exfiltrate tons of credit card details?
Any lawyers in here who know if I can pass my free credit monitoring down to my kids / grandkids? I'm 35 now, so I won't live long enough to use it all myself.
I don't really like these kinds of articles. It's like "Hey, look, publicly shame this non-technology company that probably never gets bug reports for not jumping to immediately fix my bug report!"
Seems (just a little) like bullying. I'm sure I could do the same to lots of auto shops around my city that have really basic websites and then publicly shame them for not investing enough in security even though security is the biggest money sink ever that never gets fully solved.
Hey look! I told this company about this security breach months ago! They did nothing to resolve it. They literally have a way to find a person's CC info using their phone number.
If people like this don't report it, then bad actors will get their hands on it and put EVERYONE at risk.
Come on - Panera Bread just isn’t some auto shop in your city.
Panera Bread is a company with over 2000 locations, almost 50,000 employees and more than $2 billion in revenue. They let highly sensitive information about millions of customers -- such as dietary requirements, contact details, credit card numbers -- remain publicly accessible for eight months, despite being alerted to it and accepting that it was a legitimate report. Then, once the media found out, they were misleading about the extent of the problem and didn’t even fix it properly.
There are not excuses for a company of Panera Bread's size, with someone actually employed as an 'Information Security Director', to be this incompetent.
Slight difference between the average auto shop and a company of this scale (billions in revenue) with dedicated security/technical staff. And it wasn't an expectation of an "immediate" fix, but somewhere well within the eight months it apparently took.
Without shaming them and publicising what happened, what recourse is there to encourage better corporate behaviour?
I hear what you're saying, but the only way to get this industry to take customer data security seriously is to shame them. Fast food operations place a spotlight on food safety now, as a result of several high-profile outbreaks in the past. The same thing needs to happen for data security.
This would be understandable if it was for a mom and pop shop. Panera has the profit/revenue to hire a competent technical team. They obviously didn't.
the only way security will ever be taken seriously us if there are fines for leaking data. $1000 per user per incident. Nothing will ever happen otherwise. There is zero incentive.
If we're talking $1000 per impacted user as suggested by GP, then I'm wondering just how small a fraction of the 37 billion dollar fine it would take for me to swallow my morals and "accidentally" cause or approve changes which cause a vulnerability like this, so I can then blow the whistle and collect my check. As a security engineer, I doubt my net worth will ever reach even 0.05% of that.
Whistleblowers should be compensated for uncovering intentional negligence and violation of laws, not accidents. There should be a pattern of bad behavior that the whistleblower couldn't create herself.
As someone who receives vague security emails from folks who say we have a 'vulnerability' and ask if we have a 'rewards program for bugs' and try to ask 'how much money I will be paid for the bug', Dylan's first email is very legitimate and specific. It comes from a non-generic domain (ie, not a "john.smith.1234@gmail.com" email), with an identity that could easily be verified.
In fact, as someone who would work on the API facing side of things, even that report would be enough to discover the areas to dig around and find the vulnerabilit(y|ies). There must be an API or HTTP or some other endpoint that takes in a user id, rewards card, zip code, phone number or something similar and returns data for an arbitrary user(s). Let's audit all our endpoints and see where the vulnerability might be.
I've also reported similar vulnerabilities before, and I have received a whole range of responses.
The reply to Dylan's first email is very bizarre. Is it just me or is asking for a PGP key neither suspicious nor uncommon? It sounds like the least scammy request possible. What scammer would want to go through the trouble of PGP encrypting communication?
No shit. Name any company that does. Until there are serious consequences for leaks like this, no company will take security seriously. There is no incentive. Jail some executives and take their earnings and I guarantee this will change. Short of that, or some other serious punishment, this isn't really even news anymore. Self-regulating industry is a joke.
I had posted it on the other link as well (1), but since it is relevant to this discussion, reposting it here:
Commenting only on the speed of response (or the glacial interpretation of it in Panera's case):
For companies operating in European Union, the General Data Protection Regulation (GDPR) (2) mandates that such breaches need to be disclosed under 72 hours. The implementation deadline for GDPR is by end of May 2018 (~7 weeks to go).
Underarmor, a US-based sports apparel manufacturer, who operates in EU as well, recently had a breach that affected 150-million users, and went public within 3 days of discovering the breach (3).
I believe UnderArmor's case is the norm we can expect going forward.
[...] will never respond to a request like the one you sent [...]
Dylan has not asked for a bounty or it wasn't a sales pitch! This Mike guy could not even understand basic underlying tone of the message, not to mention some technical issues the problem relates to. I hope Mr. Mike Gustavison is NOT with the company anymore, or at least is off the public-facing keyboards!
EDIT: okay update from Kerbs twitter -- Mr. Mike used to work at.. Equifax :)
Oh look,the guy my source initially notified at @panerabread EIGHT MONTHS AGO -- their dir. of info security - was senior dir. of security operations at Equifax until 2013. Shocker.
Maybe he should open his own security company... with all of his experience.
It's not hard to guess what's behind Gustavison's knee-jerk accusation of extortion. With his '1337 security skillz, he will have been getting actual extortion demands -- and probably paying them -- for a long time. Probably more than one a day.
For _this_ particular bug, I would have reported in anonymously in private and given them 48 hours before I went public. Vulnerabilities are one thing: you don't want to rush a patch and create more problems when you started with, and it's safe to assume you don't know everything about the systems involved. But direct data leakage is an entirely different matter.
Credit card numbers are sort of like symmetric keys... anyone that knows your credit card number can also authorize transactions. Why can't we have ECDSA instead?
No, they also need CVV2 and expiration date. (That said, a 3 digit number is easy to crack. The date is just 1 bit on top.) In more modern setups, there are things like MasterCard Secure or Visa 3D Secure which require direct authorisation with a token. (One time key, token hardware, phone secure element token.) The tokens are much more secure than PKI without a password.
And then there are decent banks that will put suspicious and/or big transactions on hold for phone authorization.
And you cannot change auth data easily without knowing the account password and potentially again authorizing changes with a token.
(Remember to disallow changing data over the phone. Most banks require extra work to enable phone account management anyway.)
And in any case you can dispute the suspicious transaction and probably get notifications about these.
Even with the other Panera-related threads that made the front page, this is a good read because I was wondering how exactly (and how thoroughly) the researcher attempted to contact Panera -- seems he went above and beyond just trying a few emails to the customer support line. Panera definitely had plenty of warning here.
This article https://news.ycombinator.com/item?id=16739753 (which is linked from the top comment above) was posted 10 hours ago, is currently in position 11 with over 250 votes and 73 comments, and includes much of the same information, including screenshots.
91 comments
[ 2.6 ms ] story [ 165 ms ] threadThere are no fines. People don't stop purchasing stuff from them.
The risks of not following security practices are so low that it makes logical business sense to not care much about them.
Now, say if we add fines on these security breaches. Proper fines, say % of global revenue type fines. Then yeah, they'll start caring.
Until then, wait for more of these security breaches.
Well, I am, yeah.
Especially when the future consequences barely exist. It's very rare for a breach to have serious impact on a company, relative to other areas the company could invest.
One line of code can expose 30,000,000 records. That's hard to get your head around if you are not a programmer.
You don't need to be a Programmer to understand security.
They just don't care because consumers don't seem to care with their information.
I wish technical people made it to CEO/CFO. They're supposedly bad with business, and people?
I've bever understood the process of hiring the head honchos.
My college sweetheart is a big wig in the valley. Besides being WASPy, and having a spotless resume; I wouldn't trust her to feed my dog.
I look back on her cheating her way through college, and the Star Trek ensamble she proudly sports now--and just don't get it.
But---these are the one's in charge?
You have to have a lot of bad process in place for something like this to get in.
Even in companies where good people try to do the right thing security fails not just like this case where they just chose not to act, but also because nobody else at the company cares / is knowledgeable enough to care.
Happy for someone knowledgeable to turn this into a non-rhetorical question.
What blows my mind here is the actions of a single person. The Security Director got an email about a dead simple vulnerability in his company's website/api. All he had to do was paste a single link into a browser to verify that there was a big big problem. And he did nothing?
I simply can't understand that.
What was this guy doing every day? Did he have any sense of professional self respect at all? Did he think it would just....go away? It's so confusing.
Do you think he....didn't know how to decrypt the PGP encrypted description? And that he was too embarrassed to say so? In a weird way that's my most charitable explanation.
I actually dislike bringing this up, because I respect the people the have gone through music education; while it doesn't pay in terms of salary, it's certainly something they are passionate about and love dearly. But just as they would not hire me to direct an orchestra, I would not hire them to secure critical Financial systems. This isn't to say there is a fault with them, it's just to say that we all have our strengths in certain areas.
The first part is OK, he is a director, not a grunt. The second is harder to understand. He should've handed the entire thing over, starting with the PGP thing to an actual security knowing engineer.
I have been on the receiving end of such emails a bazillion times in my carrier. My boss or the boss of my boss is certainly not going to bother with such small affairs. It is my duty to inform him when it turns out to be a big affair then he can begin to coordinate with engineering, marketing and so on to do the release. That's his job.
If you want something from a corporation, the best course of action is generally to just write (i.e. snail mail) to the CEO, because they are the only person you can be certain is capable of re-marshaling resources to deal with the root cause of the problem, whatever it may be.
Likely, but shouldn’t the director of information security be given more power to take action? I’d think that if that was my title for an outfit like Panera, and if I couldn’t enact any kind of change in face of a fairly serious vulnerability in the span of eight months, I would resign from said position. Because what’s the point of my job, if that’s what I’m dealing with?
Also, sometimes when I give them my phone number for my loyalty card it works, and sometimes it doesn't... an alarming number of times it doesn't work, all of which makes me think they have some questionable IT practices going on. I should have seen this coming.
I didn't expect this to be an actual description of a security event, but just a rhetorical observation: of course Panera Bread doesn't take security seriously. There's no "security" in the name of their company. They are not in the security business. I think they do actually take bread seriously. And store location, and customer service, and stuff like that, because that's the kind of business they are.
Because of PCI you can expect they probably do handle your credit card (except apparently the last 4) reasonably well. Because of other regulations you can expect they take food safety seriously. They take basic business operations seriously because there's a bunch of professional business-runners, and they know they really have to.
We can't expect that because they didn't. Rules and best practices do not magically get followed because they exist, and in this case they definitely did not take security (of their customer data) seriously at all.
The title is intended to be a play on the most common response companies make when they experience a serious security failure: "We take security very seriously."
We should start a blog called We Take Security Seriously that links to these responses, just like Our Incredible Journey[1] for acquisition announcements.
__________
1. https://ourincrediblejourney.tumblr.com/
No they don't. They admit on record using same additives as Subway mostly a rubber-type chemicals invented by BASF that make bread more elastic, won't go dry this quick and has longer shell-life. Basically when you eat their bread some ingredients are the same of the tires your car was put on!
Further good read how horrible quality their food is:
https://www.thealternativedaily.com/panera-bread-additives/
Companies like this would be better off not on the internet at all. It's not like anyone needs to visit the website to know where the nearest Panera Bread is. Google will tell you, if you don't know.
If this is how they respond to security issues I think it's a fair bet that they're making the same mistakes that e.g. Target and Home Depot did. Who's to say a VPN connection for some lowest bidder third party vendor hasn't already been used to exfiltrate tons of credit card details?
Seems (just a little) like bullying. I'm sure I could do the same to lots of auto shops around my city that have really basic websites and then publicly shame them for not investing enough in security even though security is the biggest money sink ever that never gets fully solved.
If people like this don't report it, then bad actors will get their hands on it and put EVERYONE at risk.
Panera Bread is a company with over 2000 locations, almost 50,000 employees and more than $2 billion in revenue. They let highly sensitive information about millions of customers -- such as dietary requirements, contact details, credit card numbers -- remain publicly accessible for eight months, despite being alerted to it and accepting that it was a legitimate report. Then, once the media found out, they were misleading about the extent of the problem and didn’t even fix it properly.
There are not excuses for a company of Panera Bread's size, with someone actually employed as an 'Information Security Director', to be this incompetent.
Without shaming them and publicising what happened, what recourse is there to encourage better corporate behaviour?
Source: worked too many years in "Healthcare IT" and left some companies after seeing how they mistreat PII.
In fact, as someone who would work on the API facing side of things, even that report would be enough to discover the areas to dig around and find the vulnerabilit(y|ies). There must be an API or HTTP or some other endpoint that takes in a user id, rewards card, zip code, phone number or something similar and returns data for an arbitrary user(s). Let's audit all our endpoints and see where the vulnerability might be.
I've also reported similar vulnerabilities before, and I have received a whole range of responses.
Paragon Initiative Enterprises.
Commenting only on the speed of response (or the glacial interpretation of it in Panera's case): For companies operating in European Union, the General Data Protection Regulation (GDPR) (2) mandates that such breaches need to be disclosed under 72 hours. The implementation deadline for GDPR is by end of May 2018 (~7 weeks to go).
Underarmor, a US-based sports apparel manufacturer, who operates in EU as well, recently had a breach that affected 150-million users, and went public within 3 days of discovering the breach (3).
I believe UnderArmor's case is the norm we can expect going forward.
(1)https://news.ycombinator.com/item?id=16739753
(2)https://en.wikipedia.org/wiki/General_Data_Protection_Regula...
(3)http://www.bbc.com/news/technology-43592470
[...] will never respond to a request like the one you sent [...]
Dylan has not asked for a bounty or it wasn't a sales pitch! This Mike guy could not even understand basic underlying tone of the message, not to mention some technical issues the problem relates to. I hope Mr. Mike Gustavison is NOT with the company anymore, or at least is off the public-facing keyboards!
[1] https://cdn-images-1.medium.com/max/2000/1*oJEZOkK6qtq2RreBN...
EDIT: okay update from Kerbs twitter -- Mr. Mike used to work at.. Equifax :)
Oh look,the guy my source initially notified at @panerabread EIGHT MONTHS AGO -- their dir. of info security - was senior dir. of security operations at Equifax until 2013. Shocker.
Maybe he should open his own security company... with all of his experience.
And then there are decent banks that will put suspicious and/or big transactions on hold for phone authorization. And you cannot change auth data easily without knowing the account password and potentially again authorizing changes with a token. (Remember to disallow changing data over the phone. Most banks require extra work to enable phone account management anyway.)
And in any case you can dispute the suspicious transaction and probably get notifications about these.
If everyone - wonder if this is "effect of hackers news" :)
EDIT: down for everyone: http://www.isitdownrightnow.com/panerabread.com.html
I guess the DO eventually take security seriously :)))
Archives: https://web.archive.org/web/20180403215610/https://www.akama...
https://archive.org/details/panera-security-video