Whenever someone complains about another company's product, code, features, security....I always wish it was mandatory to include a link to the kind of software the poster is putting into production.
I have. I deploy in ASP.NET and get a hash of the uploaded DLL. I check it twice a day. Never had any incidents to this day but as the saying goes, it's better to be safe than sorry.
In ASP.NET pretty much all output, including html and js is included in the DLL. Only external js files are left out. Sure they could hack them which means my solution isn't bulletproof but I could also produce a hash for them too.
I have a tiny $5 Onion Omega2 on an independent cellular connection that checks file integrity on the production web servers every 15 minutes.
If the content of any of the files change, I get an e-mail.
If the alerts start coming in when I know I've just pushed a new version to production, the mail has a link that I can click that will re-scan all of the files and build new checksums.
If the alerts start coming in in the middle of the night, then I know something is up.
Obviously, this only works in small environments like mine where I'm the only one capable of updating the production servers. But it managed to catch a backdoor left in by the previous developer, who for some reason stored and updated his resume on the production server.
What that won't do is save you from malicious code inserted into 3rd party content (script libraries, etc.) that you load from a CDN. If you're worried about that, you should make a copy of a verified version and serve it yourself.
You can just set up Tripwire to do this sort of thing
I wanted something that was completely independent of the machine. Separate box, separate network, separate architecture, etc...
What that won't do is save you from malicious code inserted into 3rd party content (script libraries, etc.) that you load from a CDN. If you're worried about that, you should make a copy of a verified version and serve it yourself.
I don't CDN on work projects. It's not worth the risk. If something goes wrong, I'd rather it be my fault and something I can understand and fix, whenever possible. Farming stuff out just leads to layers of things that can break, be compromised, or simply go wrong.
Again, it works at my scale (about 15 sites). It won't work for everyone.
It would be interesting to deploy a few of them in different places and check that they all see the same as well maybe.
That's a great idea. And since they're only $5 each (I think I spent $15 with the power shield), it's not a big deal.
Also did you do this as a belts and braces thing or is the system you are auditing particularly high security/risk in some way?
When I got here, most of the web sites were riddled with worms and trojans and spambots other bad stuff. One by one I just nuked them and started over. This was a deliberate isolation to keep an eye on things in case the sites or the dev machines ever got compromised.
What if a developer's machine was somehow compromised and the bad version of the file was put into the actual repo or deploy? I have no idea how likely this is or if it is a possibility in NewEgg's environment but that would be a case where only scanning for changes on the servers wouldn't catch.
What if a developer's machine was somehow compromised and the bad version of the file was put into the actual repo or deploy?
In a complex environment, that's a complex problem. In mine, it's not a big problem. Keeping the security routine on an external device with no other function I think helps. And since the device is on a completely different network, and a cellular connection with changing IP addresses, if someone was targeting the company they'd never find it.
Code is supposed to change on the website all the time, though, when they run a deploy. Surely they do have a team working on checkout, but it isn't obvious how this would be detected. The article leaves out how they got the malicious javascript onto the page in the first place, though, so it's hard to say.
I'm still confused as to how they could insert code here.
Are we talking about a server intrustion where they modified the actual cart code, or something between Newegg and the payment servers? (Sorry this isn't my domain, I'm just curious)
It looks to me like a server intrusion where they modified static files kept on a webserver (like apache or nginx). But it also seems like we don't have enough evidence to know for sure. (Edit: or they might have been static files kept on a CMS.)
For a website with so many visitors and transactions, I’m surprised this API call never threw enough errors for them to see in logging. I will admit that JS logging can be messy because of all the different environments but after some time I would have hoped this would have been caught.
It seems like we just don't have enough information to know how exactly the attackers got the JS onto the page. What it looks like, though, is that some machine serving static files for the website got compromised, and the attackers replaced an innocent file with one that had some custom javascript tacked onto the end, that skimmed the credit card numbers.
For what it's worth, PCI compliance doesn't really help you keep your webservers secure, or to keep people from breaking into your servers at all, it's more about the way you permanently store credit card numbers.
It was appended to "Modernizr.js". That suggests a number of further possibilities by which that may have been replaced on a developer's computer, or at the point of download by a package manager (are packages authenticated?)
It could well have been included in a legitimate re-deploy if that was the case.
Since that file was served from "/cms/global/scripts/lib", perhaps their CMS got compromised. Anecdotally it seems like CMS's are more frequently prone to compromises than systems like nginx or apache.
As an aside, while it's certainly true that if one can compromise a web server (whether serving static HTML or generating dynamic HTML) then it's game over, the modern style of JavaScript-centric development greatly expands the attack surface. An attacker can compromise any of a number of different servers (e.g. analytics providers), via numerous means; moreover, he can serve dynamic malicious code, and quickly respond to changes the targeted site makes (his implant needs to survive for awhile, but his access can be revoked and folks can still be compromised).
Websites which served static pages and received credit-card information via simple forms would be far more secure than dynamic web apps stitched together from JavaScript served from multiple servers in countries across the globe, with data inserted into the DOM retrieved from multiple servers across the globe.
I contend that a more traditional, static architecture would tend to lead to better architectural isolation. As an example, if one needed to install server-side analytics software rather than simply injecting it into a page at runtime, then one would download and install it once: any compromise of the analytics provider after one had download it would be irrelevant. But with dynamic JavaScript, one's users download the analytics software with every single page load: a compromise instantly affects users.
Moreover, with traditional server-side software one tends to have reasonably-well defined protocols for interaction, rather than just saying, 'hey, I'll run all your code in the same process as everything else.' But with client-side JavaScript, any piece of code can see everything.
As an engineering community, we've adopted a style which has great costs as well as great benefits. My opinion is that we moved too quickly, and now we're feeling the pain. While a universal portable app framework is pretty awesome, JavaScript layered over a document system is probably not the best way to achieve that.
Agreed, they should have given us a notice. I don't know if I will ever buy something from NewEgg again if this is how they are going to treat security events.
I also purchased something right in the middle of that time frame, but with a regular credit card. I'm not concerned as I pretty much assume my credit-card is always compromised and I check my charges each month for discrepancies and contest anything bad. Though the credit-card catches the fraudulent transactions most of the time these days, so it usually doesn't come to that.
Replying to myself. I use uMatrix and it just occurred to me that it would have blocked any communication attempts to neweggstats.com from newegg.com.... so yay?
I noticed this too; uMaxtrix should catch this entire class of attack. Privacy Badger probably wouldn't - it watches for this sort of export, but its three-strikes rule targets trackers and misses unique attacks. Killing JS outright would do the trick also, but it looks like it would also have broken Newegg's checkout page.
Looks like yet another point for "treat web browsing as adversarial". The price of allowing pages to load freely isn't just high weight, trackers, and ads - all too frequently it's actual security failures. I understand that ad/tracker blocking is a problem for keeping sites profitable, but I can't really imagine bending on that issue as long as monetization techniques and threat vectors overlap so heavily.
Given the way the code works, I don't see how you could be affected if your transaction went through PayPal (ie you didn't enter any payment information, such as a credit card).
I did eventually pay with Paypal. However, I do know that at some point in the last few months, I attempted to make a transaction on some web site and had issues. It was either:
1. I tried to pay with Paypal and failed. Paid with CC instead.
2. I tried to pay with my CC and failed. Paid with Paypal instead.
I don't remember for which site this happened, but the paranoid part of me is wondering if it may have been Newegg and item 2 above.
Did you happen to receive an unexpected address change notification? I did when I checked out with Paypal. The address didn't change (maybe it's all caps now and wasn't before?) and now I'm wondering too.
It took them a month to find this--could there be other code they haven't discovered?
PCI compliance isn't that hard to get. And, typically the code is changed after the compliance review.
Also, developers and engineers at Newegg are probably changing lines of code every day. Which means, at any point in time someone could have accidentally introduced a vector that allowed the attackers to inject their code.
Auditing and security reviews only work if every line of code is tested, reviewed, and verified. However, that be extremely costly and slow, which isn't "good for business".
Maybe, we as consumers, engineers, managers, and executives should demand that code safety and security is prioritized over profit. I would rather have a secure website then a new redesign every two years.
This isn't fort Knox It's credit card info. There's always a cost benefit. You can spend all your time and money on doingsecurity reviews but without profit there ain't no company.
Do you have any evidence to suggest "that code safety and security" was NOT "prioritized over profit"? How would you falsify this? Are we to assume that every security breach indicates that profit was prioritized over security? Doubtful.
Also, "profit" is the reward for doing things efficiently, it is not some kind of scoreboard for how much a company is ripping off consumers. NewEgg is in such a hyper-competitive & mature space with margins in the low single-digits on average. Incidents like this can absolutely tank them, especially this close to the holiday season.
E-commerce companies do not make money on security breaches, they lose it. Lots of it. I think you should be a little more open-minded to how businesses operate, that's all.
I had a conversation two days ago with the CTO of a very large company you've definitely heard of who said "we don't need to worry about our website security, we have a firewall and SSL"
I think these types of attacks are vastly underreported, if anything.
They are just now (around 11:10 CDT 9/19) sending out notification emails to customers. At the moment they don't even seem to know what accounts were affected.
45 comments
[ 5.2 ms ] story [ 56.7 ms ] threadWith 50,000,000 users a month, surely they have a whole team working on checkout, all the time?
I can dream.
Glass houses and all that.
I have a tiny $5 Onion Omega2 on an independent cellular connection that checks file integrity on the production web servers every 15 minutes.
If the content of any of the files change, I get an e-mail.
If the alerts start coming in when I know I've just pushed a new version to production, the mail has a link that I can click that will re-scan all of the files and build new checksums.
If the alerts start coming in in the middle of the night, then I know something is up.
Obviously, this only works in small environments like mine where I'm the only one capable of updating the production servers. But it managed to catch a backdoor left in by the previous developer, who for some reason stored and updated his resume on the production server.
It would be interesting to deploy a few of them in different places and check that they all see the same as well maybe.
Also did you do this as a belts and braces thing or is the system you are auditing particularly high security/risk in some way?
https://github.com/Tripwire/tripwire-open-source/#open-sourc...
What that won't do is save you from malicious code inserted into 3rd party content (script libraries, etc.) that you load from a CDN. If you're worried about that, you should make a copy of a verified version and serve it yourself.
I wanted something that was completely independent of the machine. Separate box, separate network, separate architecture, etc...
What that won't do is save you from malicious code inserted into 3rd party content (script libraries, etc.) that you load from a CDN. If you're worried about that, you should make a copy of a verified version and serve it yourself.
I don't CDN on work projects. It's not worth the risk. If something goes wrong, I'd rather it be my fault and something I can understand and fix, whenever possible. Farming stuff out just leads to layers of things that can break, be compromised, or simply go wrong.
Again, it works at my scale (about 15 sites). It won't work for everyone.
That's a great idea. And since they're only $5 each (I think I spent $15 with the power shield), it's not a big deal.
Also did you do this as a belts and braces thing or is the system you are auditing particularly high security/risk in some way?
When I got here, most of the web sites were riddled with worms and trojans and spambots other bad stuff. One by one I just nuked them and started over. This was a deliberate isolation to keep an eye on things in case the sites or the dev machines ever got compromised.
In a complex environment, that's a complex problem. In mine, it's not a big problem. Keeping the security routine on an external device with no other function I think helps. And since the device is on a completely different network, and a cellular connection with changing IP addresses, if someone was targeting the company they'd never find it.
That's the theory, anyway. So far, so good!
Are we talking about a server intrustion where they modified the actual cart code, or something between Newegg and the payment servers? (Sorry this isn't my domain, I'm just curious)
https://www.riskiq.com/blog/labs/magecart-british-airways-br...
It seems like we just don't have enough information to know how exactly the attackers got the JS onto the page. What it looks like, though, is that some machine serving static files for the website got compromised, and the attackers replaced an innocent file with one that had some custom javascript tacked onto the end, that skimmed the credit card numbers.
For what it's worth, PCI compliance doesn't really help you keep your webservers secure, or to keep people from breaking into your servers at all, it's more about the way you permanently store credit card numbers.
It could well have been included in a legitimate re-deploy if that was the case.
As an aside, while it's certainly true that if one can compromise a web server (whether serving static HTML or generating dynamic HTML) then it's game over, the modern style of JavaScript-centric development greatly expands the attack surface. An attacker can compromise any of a number of different servers (e.g. analytics providers), via numerous means; moreover, he can serve dynamic malicious code, and quickly respond to changes the targeted site makes (his implant needs to survive for awhile, but his access can be revoked and folks can still be compromised).
Websites which served static pages and received credit-card information via simple forms would be far more secure than dynamic web apps stitched together from JavaScript served from multiple servers in countries across the globe, with data inserted into the DOM retrieved from multiple servers across the globe.
I contend that a more traditional, static architecture would tend to lead to better architectural isolation. As an example, if one needed to install server-side analytics software rather than simply injecting it into a page at runtime, then one would download and install it once: any compromise of the analytics provider after one had download it would be irrelevant. But with dynamic JavaScript, one's users download the analytics software with every single page load: a compromise instantly affects users.
Moreover, with traditional server-side software one tends to have reasonably-well defined protocols for interaction, rather than just saying, 'hey, I'll run all your code in the same process as everything else.' But with client-side JavaScript, any piece of code can see everything.
As an engineering community, we've adopted a style which has great costs as well as great benefits. My opinion is that we moved too quickly, and now we're feeling the pain. While a universal portable app framework is pretty awesome, JavaScript layered over a document system is probably not the best way to achieve that.
As I wrote years ago, JavaScript delenda est.
However, I am disappointed that NewEgg hasn't made any sort of official announcement yet.
Looks like yet another point for "treat web browsing as adversarial". The price of allowing pages to load freely isn't just high weight, trackers, and ads - all too frequently it's actual security failures. I understand that ad/tracker blocking is a problem for keeping sites profitable, but I can't really imagine bending on that issue as long as monetization techniques and threat vectors overlap so heavily.
I paid with Paypal. I assume I'm not affected?
1. I tried to pay with Paypal and failed. Paid with CC instead.
2. I tried to pay with my CC and failed. Paid with Paypal instead.
I don't remember for which site this happened, but the paranoid part of me is wondering if it may have been Newegg and item 2 above.
It took them a month to find this--could there be other code they haven't discovered?
Also, developers and engineers at Newegg are probably changing lines of code every day. Which means, at any point in time someone could have accidentally introduced a vector that allowed the attackers to inject their code.
Auditing and security reviews only work if every line of code is tested, reviewed, and verified. However, that be extremely costly and slow, which isn't "good for business".
Maybe, we as consumers, engineers, managers, and executives should demand that code safety and security is prioritized over profit. I would rather have a secure website then a new redesign every two years.
Also, "profit" is the reward for doing things efficiently, it is not some kind of scoreboard for how much a company is ripping off consumers. NewEgg is in such a hyper-competitive & mature space with margins in the low single-digits on average. Incidents like this can absolutely tank them, especially this close to the holiday season.
E-commerce companies do not make money on security breaches, they lose it. Lots of it. I think you should be a little more open-minded to how businesses operate, that's all.
I think these types of attacks are vastly underreported, if anything.