> In a moment of true dystopian chaos, the official Equifax Twitter account repeatedly tweeted a phishing link, mistaking it for the breach response page.
It wasn't a phishing site, Wired. It was a site designed to point out that their approach is susceptible to phishing. Entering data in the form gave you a scolding.
> The Equifax Breach Exposes America's Identity Crisis
When asked by multiple lawmakers why Equifax set up this separate site, Smith said that the company's main domain was not architected to process the enormous traffic the company knew would come its way after the announcement. In all, Smith said the independent breach response site has had 400 million consumer visits, which would have crumpled the main site.
They could have done a subdomain and pointed it at the site.
> It wasn't a phishing site, Wired. It was a site designed to point out that their approach is susceptible to phishing. Entering data in the form gave you a scolding.
I'd argue it was a phishing site. Just fortunately it was controlled by white hats.
As Brian Krebs reported in the "dumpster fire" article the response site was setup by their PR company. Perhaps they did not want to give away ownership of part of their domain... For security reasons
The developer who set it up tweeted that it was a "phishing site" (albeit one that didn't exfiltrate any actual data), so I'd say that was accurate reporting.
> When asked by multiple lawmakers why Equifax set up this separate site, Smith said the company's main domain was not architected to process the enormous traffic the company knew would come its way after the announcement. In all, Smith said, the independent breach-response site has had 400 million consumer visits, which would have crumpled the main site.
Apparently they've never heard of subdomains, static pages, reverse proxies, and <insert many other solutions anyone on HN could come up with off the top of their heads>?
Right. I think it's code for "I think domain means website, and our technical folks said our existing corp site didn't have the capacity, so we told them to setup a 'completely separate site'".
It strikes me more as a move suggested by a reputation management company, though they may have in-house specialists. While usually, and intentionally, behind the scenes, reputation management companies are often involved in things whee there is massive public outcry.
Most probably their marketing and legal team came up with the idea. Once the outcry blows over turn off the site and all incoming external links become dead. After a while general public will not remember and wouldn’t associate their main site with this incident through search. Legal might have encouraged to have two fully independent sites. I will not be surprised two sites belong to two separate legal entities. Just a speculation on my part considering how US corporate behave.
That's just Horseshit. I have worked on both sides of the table and incompetence is equally widespread. However, vendors are not as bad as they seems to appear and employees are not as good as they seems to appear.
Most of the contracts nowadays have a penalty clause and a stated Service Level Agreements (SLA) around performing the vulnerability scan and closing the GAP within a stipulated time frame. Outsourcing companies might not be innovative but they don't like to lose money on failing SLAs.
Coming back to the issue of incompetence. I have often seen so called innovator of the just taking the credit for the work done by Vendor.
The way typical scenario rolls out is this. Senior management of of the company will throw a challenge to the senior management of Vendor. After a few discussions middle management of will assign the responsibility to their lowest level employees. After a lot of hard work when solution is in sight, all communication channels will be closed.
A few weeks later solution will be presented to their own management while keeping the vendor completely out of loop as if vendor was completely useless and they did all the hard work. Vendor will be presented as mindless robot who can just execute the instruction. Hell I have seen employees not even having the courtesy even the reword our solutions.
Incentives of employees are aligned with making the vendor look less effective or they themselves will be replaced at some point by their own higher management. I think its not their fault. It is just the environment where painting the vendor in positive color will be detrimental to employee's own job.
I'm sorry this happened to you, it sucks to not get credit for your work. I think the explanation is simpler though.
There is no incentive to say either good things or bad things about a Vendor, the incentive is to keep the focus on the internal people that were involved, because recognizing internal employees is critical for morale and general team cohesion. Recognizing a Vendor's effort could potentially backfire and lead to bad blood internally (happens with consultants all the time) and is not really a priority because the engagement is based on a negotiated contract that is primarily driven by financial or expertise reasons, not the human elements.
The fact is, the Vendor is the one that should be worried about the morale of their own employees. The Vendor's management should be praising the good work of the employees, and if there is blatant disrespect coming down from the Company, it is their job to either address it or at least justify to appease their rank and file.
I completely agree with you and I am not sour about it at all. In corporate environment everyone works for their self interest and as vendor we are happy if we are getting paid.
I was just calling out the passive aggressive parent comment.
Without looking it up, I’ll throw my chips on the table and claim that you would like to just blame offshoring for any problem without providing any substantive proof.
If there was a vendor involved - Equifax would have already jumped on that fact and blamed the vendor. They would have already played the oh-we-had-a-bad-vendor-and-fired-them-immediately game by now.
Their executives were incredibly incompetent. Whether your people are offshore or not, a competent leader knows to tell them to do things properly and knows enough to verify the basics. None of that was done.
I doubt there's a well considered plan here. Much more likely the people setting up the site simply didn't know better and/or the people who know technology aren't in decision making roles.
Yup, it's called "gross negligence", which often has the same legal implications of malice.
As an analogy, lending your car to a friend which turns out to have faulty brakes is negligent. But lending it to him when your mechanic told you the brakes don't work is probably grossly negligent, and will often have the the same repercussions as if you cut the brakes yourself.
If the legal team came up with this idea, they need a new legal team. In house lawyers do not have to be technical experts, but they should know about cyber-squatting and DNS related fraud. This should have been in their wheelhouse.
What your undergrad is in really doesn't mean anything if you've got experience, especially if you got it at a time when most schools didn't have cyber-security programs/concentrations within CS and CS was more closely aligned with math.
Some of the best infosec people I know have completely irrelevant degrees.
That said, I'm not trying to make excuses for Equifax. From what we know the head of infosec there was not qualified to do the job.
Seems to me to be dead simple. One dns entry for equifax.com, another pointing off to different hardware for equisecuritybreach.equifax.com. Nothing fancy at all. Make the page have a different look and feel.
Assuming you're serious, get on that my dude! All the best developers I know have a solid mind for ops; operation of your application should inform its design.
That seems to be indication that decision process there is as broken as security process. Either they did not have a professional that could explain the management why the claim above is baloney and what are the solutions to make it work, or that professional wasn't consulted, or his opinion was dismissed by the management. In any case, the symphony of fail continues.
>the company's main domain was not architected to process the enormous traffic
I understand all the sarcasm expressed here, but this actually may be true if they run their own DNS service. Subdomain isn't a solution in this case, but separate domain is.
I fail to see why DNS would be an issue. Set a sufficiently large
TTL and you should see almost no queries hit your DNS servers once intermediate caches have seen the domain.
Look, you could just set the new subdomain as a cname with a large TTL and then handle that a-record via a different provider if you really need to. You could also choose to delegate the whole subdomain. Also, load-balancing via multiple a-records does not at all depend on the TTL. DNS is really the easiest part to handle here. Internal policy might prevent delegation, but that’s not a technical problem.
If true, it begs a question as to why. Why does equifax need to be a DNS expert? There are so many inexpensive DNS solutions that work for huge sites... far cheaper than hiring one developer.
from a security point of view it does kind of make sense to have non-trusted content hosted under a separate domain. so if they were using a third party to host the breach page it may have been risky to have it hosted under the .equifax.com domain. this is why you have .github.io and googleusercontent domains, etc. though, probably the risk from confusing users is worse than the risk from the third party playing silly games with cookies in this case.
Yes the bureaucrats, lobbyists, and special interests will continue fucking everything up.
But the USDS is set up so that at least competent technologists can throw up some objective defense vs. the government officials who really think Equifax is a viable solution aka "Nobody ever got fired buying an IBM."
USDS is not subordinate to Jared Kushner. We work with the Office of American Innovation on projects that align with our values, but ultimately we decide what projects to engage on. Which is pretty unique within the government.
When Mr. Kushner issues a dictat that you are no longer to work on <potentially embarrassing issue for the administration>, you'll get a test of who ultimately decides what projects to engage on.
I think the todo list more than long enough to keep USDS busy for four or eight years even if you strike all the potentially politically sensitive things from it.
That seems like an exercise in futility. These are, at their core, political/management issues. Getting coders involved and misleading them into believing their code will do something to counteract the incentives that cause such behavior is just going to lead to a lot of very frustrated coders.
Right, the solution isn't to take a 1-year term at USDS, it's to go beyond coding to IT management while not losing respect for the people working the technology, building political connections, and getting appointed to positions like Department CIO for a government department. And that's not the solution so much as getting into position to really make an impact on the problem.
And, yeah, it's not easy. Solving big institutional problems isn't.
OTOH, a term at USDS may give you an opportunity to work around the fringes of the problem and grasp it's grand outline better. And the scale of government is such that a small improvement can be a big deal.
"You must be willing to move to Washington, DC". Oh well. Ever heard of videochat? I mean what exactly are they doing there that requires physical presence at DC?
Going where the work is. Actually showing up at an agency and sitting down face to face with people that could use help buys a lot of credibility, especially in the public sector.
Wow, I haven't heard of USDS before! But I recently used a website they apparently built (uscis.gov) and remember being very impressed with it. It surpassed my expectations for a government website by far: sane password policy, material design, easy to use navigation, well designed application flow...
Encouraging to see people with modern best practices in mind working for the government.
I actually went to a talk recently by USDS and 18F. Really enjoyed it and seriously considering going to one of them as my next job. A few clarifications:
* USDS limits employees to two consecutive years.
* 18F, it's sister agency, does support remote work. They're more like consultants and switch between different agencies project by project.
I would be very interested in doing this. Unfortunately I just checked their pay scales and... well, they’re pretty bad. Which to be honest surprises me because I’ve followed some of what USDS/18F do and they have talented folks.
The pay is definitely below Bay Area standards, though it's quite good for government. Ultimately though the main draw is not the pay, it's the chance to use your tech skills to make a truly meaningful real-world impact on a large scale.
> Ultimately though the main draw is not the pay, it's the chance to use your tech skills to make a truly meaningful real-world impact on a large scale.
I, personally, wouldn't value that opportunity at $100k+. This is really Congress' fault for having such an awful pay system.
While I can easily find the GS scales, do any of you know how typical USDS or 18f positions map to the GS? Would a senior developer be a GS10 or GS15? GS15 would be competitive with DC software salaries (all benefits included). GS10, not so much.
Sorry, but signing up for a "tour of duty" at this time would basically mean working for and supporting Trump. My morals won't allow that, no matter how much good you think you can do there.
That is uttlery insane!! I thought the best possible outcome for Equifax was to avoid getting sued into oblivion, but now that get a 7 million dollar contract with the IRS?!
There are clearly two sets of rules - one for large companies where they can brazenly flaunt laws whilst going from strength to strength. And another set of rules for everyone else - where if you take a wrong step it's all over!
People are being advised to perform credit freezes with Equifax, Experian, and TransUnion. For each credit freeze, you have to pay the credit agency $5-$10 depending on the state (only one state has forced them to do it for free; several states have no requirement to implement a credit freeze at all).
The credit reporting agencies are raking in tens of millions of dollars in these fees because one of them was incompetent. That's basically racketeering.
In case you think paying a company $10 not to disclose information it's gathered about you is a raw deal, TransUnion offers a different, free "credit monitoring" service (https://www.transunion.com/product/trueidentity-free-identit...). It's free, there's no catch (their marketing copy), except that the terms of service include an arbitration agreement. So you give up your right to sue TransUnion or be in a class action.
Its even worse as freezing is useless as all of the info you need to lift the freeze is the same that was lost in the hack, provided you don't mind forging a few documents (see 0) and using the mail.
#2 is the most disturbing to me, simply because it's SO SIMPLE. Know your environment, patch it. My guess would be the reason their security scanners didn't alert to the missing patch was that they scanner had not been updated. Again, SO SIMPLE, get updated definitions anytime you scan.
#4 I'm fine with, the CEO meeting on IT security quarterly sounds adequate to me. His involvement in IT security should not extend beyond ensuring policy is in place and being acted on. Quarterly is more than adequate to ensure this is being done.
Agreed on #2. Quarterly meetings with CEO about security don't seem so bad in the general case (although in Equifax's case, security is a way bigger deal), but that shouldn't be all. Only 1 person knew about the patch? Even a medium-size enterprise where security is even a moderate concern should have more people than that focused on security 100% of the time.
Quarterly is possibly fine, but when sparks are flying in the back end, and fluid is pouring out of the transmission on the running engine, CSO needs to be able to access CEO promptly.
First, no one gives a crap about security until something like this happens to them. The incentives are not in place to require it. Time and resources spent on security are time and resources not spent on beating the competitor. The best outcome you can hope for if you _do_ spend on security is that your competitor will have some apocalyptic breach similar to this one, and that doesn't happen most of the time, at least not in a way that results in any accountability or visibility.
It's important to understand that because it's the reason why your bosses will never care unless/until they have an experience like this, or until they expect someone important to be looking for an opening (audits, due diligence, etc).
Second, there are indeed some clowns and cronies hired at these sort of places, but a lot of the people are decently skilled. I'm speaking generally about the workforce at these larger enterprisey companies, as I don't know anyone at Equifax afaik.
Technical competence is less of an issue than intra-managerial political games like "shift the blame" and "silo defense", at least some of the time. Part of "intra-managerial games" is having the least-skilled people in the most authoritative positions, leaving decision authority with those least equipped to evaluate a situation and determine a responsible response.
I am sure that there is someone at Equifax who saw the news on this vulnerability and registered it as being something they needed to patch. I wouldn't be surprised at all to learn that this person didn't say anything because they learned long ago that there's nothing to be gained by actually trying to get the patch done in any non-routine manner.
I also wouldn't be surprised to learn they did try to raise the alarm and were mocked and/or shut down, both by other technical groups trying to avoid the appearance of an issue with "their" section of the system, and/or by management, who are likely to interpret such attempts as alarmism, if not plain political malfeasance.
Hate to say it but we're on the brink of a formalized licensing program for security, servers, etc. I am honestly surprised that the mainstream outlets and politicians have not been touting this yet, with all the high-profile and extremely costly data security breaches that have been occurring over the last few years, but I really expect it to come soon. I wouldn't be surprised at all to see Equifax become the Enron-like scapegoat for such legislation.
> First, no one gives a crap about security until something like this happens to them. The incentives are not in place to require it. Time and resources spent on security are time and resources not spent on beating the competitor.
Exactly. I'm reminded about the tale about the two hikers and the bear. Two hikers are out hiking one day, when off in the distance they see a bear running their way to attack them. One starts to run, but the other stops to change his hiking boots to running shoes. The other says: "What are you doing? You can't outrun a bear!". To which the running shoe changer replies: "I don't have to outrun the bear, I just have to outrun you".
Same goes here. Don't have to really spend on security, just enough to not actually be the one of the two (or three or four) who suffer a breach.
Open source vulnerability management is a technology process that has levels of maturity. That being said, a freeware tool could have spotted this right away.
Although open source likely makes up a large part of software inventory, particularly if developing or customizing software, most usually open source isn’t singled out from “software vulnerability management”
"The company announced Monday that the total number of people impacted by its breach is not 143 million—the amount it first disclosed—but in fact 145.5 million. Its ability to casually misplace 2.5 million lives upended by the breach is alarming"
I can't really take the author too seriously after reading that.
It's kind of a ridiculous statement. A counting error is nowhere near is shocking as the initial breach, especially considering it happened after the breach.
It is bad because not only they let someone in but they don't know what he did despite having recorded everything (or so they said).
It means that maybe more is to come.
Forensics is hard for even companies that are well versed and adept with tech. Evidence suggests that Equifax is not among that group of companies.
The number being revised doesn't surprise me. I'd have been surprised if it hadn't been revised. I'd expect additional revision, as the investigation continues and the breach more fully understood.
For the record, that doesn't show I support them, nor am I making excuses for them. It's just the opposite. I believe them to be very, very inept - to the point of being incompetent, illsuited, and inexcusable. I expect additional revision, not because that is justified but because they simply suck that bad.
I can't think of a breach handled quite this poorly, and that includes my biases from the OPM breach. I'd say the OPM breach was far worse but handled marginally better. Equifax had to work hard to take the crown, but they have managed to edge them out by a narrow margin.
It's purposefully alarmist. "[Equifax] casually misplace[d] 2.5 million lives" sounds much more dramatic than "Equifax revised their estimate of the number of affected customers by +1.7%."
Or maybe the latter is purposefully anodyne? 2.5 million people is rightfully alarming. 1.7% minimizes the sheer amount of people affected that weren't revealed earlier.
To me if they make mistake like that is that they most likely try to estimate and don't really know who was affected. If they knew they could simply just exact number of affected people.
This explain why their site seem to return randomly who was affected.
Yeah that's overly dramatic. What is probably "we used row count from the old monthly backup instead of current row count of the database" turns into "OMG millions of lives!" And, of course, these lives are not "upended" in any way - some of them potentially might be if they suffer from identity theft, but surely not by a minuscule counting mistake. Bad journalism.
> Second, Smith blamed a scanning system used to spot this sort of oversight that did not identify the customer-dispute portal as vulnerable. Smith said forensic investigators are still looking into why the scanner failed.
Do these scanners ever work? Without naming names, the only reason we used this at a previous company was to kind of handwave around hey we have this tool doing regular security checks.
> The first excuse Smith gave was "human error." He says there was a particular (unnamed) individual who knew that the portal needed to be patched but failed to notify the appropriate IT team.
How is this one individual responsible for tracking all IT security vulnerabilities in all the technologies that equifax uses across its stack? How do they not have an admin team whose job it is to do these things?
We do not know anything about the number of people who could have known about this issue and escalated it, but it does not surprise me at all that there is one primary person who is responsible for it. In fact, if there were not exactly one person who is responsible for the security of the application that would be a different sort of problem as diffuse responsibility can be neglected quite easily in large organizations. Still, at my own company - A LOT of people were focused on the timely remediation of this issue and there was no chance of it being left to one person, even for development environments of internal facing applications.
The scanners do work, we used them at my company and they found this specific vulnerability. It surprised some development teams because there are libraries that use struts without really advertising it, so even applications that don't "use struts" had a transitive dependency on it, it was running in their container and showed up on the lists that were circulating.
This reminds of a fairly well known online/offline service company that does not patch their production Windows servers at all. Their mode of defense is firewall. Apparently they has some servers/apps breaking when Windows servers were patched, so they do not patch their servers at all.
From the Ninja Threat Model at https://www.nccgroup.trust/us/about-us/newsroom-and-events/b...: The attacker is going to sit on the same network segment as the application. There’s no firewall or filters. There’s a special place in hell reserved for products that require firewalls or filtering to protect themselves against attack.
I just ran into a company who has updates disabled on all Windows PCs, leaving them as the stock install. Their IT got defensive about it very quickly when I asked them to allow us to update the PC they were running our software on. They don't handle consumer data, but they have their own set of problems which could come from a data breach.
> How is this one individual responsible for tracking all IT security vulnerabilities in all the technologies that equifax uses across its stack? How do they not have an admin team whose job it is to do these things?
They do have a vulnerability management team; the responsibility does not fall on one person.
For all we know this unnamed individual is just a low-level developer, analyst or pentester who had some knowledge of the exposure and failed to act.
Having personally seen the sausage factory on one of the "scanner" companies, I would be very careful and do due diligence on any such thing I contemplated buying.
Compliance & Security "in the enterprise" is a smoke-and-mirrors game full of weird terminology divorced from the nuts-and-bolts issues with people who don't know what they're selling engaging with people who don't know what to buy, playing the industry-form and trade-show and junket and relationship game (probably no different from the DB market, the network market, the ERP market).
The DKs that fail at technology and don't really know it move on to these segments of the security market, to everyone's detriment.
Yes they do. But you have to have a process to triage what the scanners produce, and have a team whose job it is to keep the ops/dev side accountable.
They are quite useful when scanning all the internal parts of a datacenter. There's a fair amount of nitpicking but it helps weed out the obvious (like installing some open source package which defaults to some bad cipher for SSL, or leaving internal links unencryped under pretext that it's "safe behind the firewall").
Often, though, the issues flagged by the scanner trigger deeper conversations about security. That's where the real value is, but that requires an organizational culture that actually cares about security. Instead, many companies just throw money at the problem of "security" and consider the scanner will fix all their issues with zero effort.
For this particular vulnerability? I doubt it. (Source: I worked on building such a scanner recently).
Let's begin with a vulnerability that is obviously detectable with an off-the-shelf scanner: "Buffer overrun in Linux CIFS Server". We can detect a host with this vulnerability by simply scanning the local subnet for live IP addresses then fingerprinting the host to determine it is Linux, checking if it responds on the SMB port and finally sending it a test exploit payload and see if it responds in the expected way (or crashes). This all takes a few ms.
But now consider what if the vulnerability is only exploitable by an authenticated session? Well we could have our scanner ask the operator for a set of credentials for each CIFS server it finds. But what if the vulnerability requires a mounted share? Well the scanner can ask the operator for the name of a share, or if it is lucky it could try to guess one. Perhaps we could be happy with the scanner identifying the version of Samba running on the server and concluding from patch history knowledge that it is vulnerable. But boxes get locally patched and often there isn't enough information in the externally visible version info to tell one way or the other.
Now think about trying to do this in the context of the vulnerability "If your application uses Struts and allows file upload then it might have this RCE vulnerability". We don't even know if we're using Struts in any of our applications. We may have hundreds of applications. Is it possible to tell from the outside that Struts is being used? (I'm not sure but probably not). You could note that the web server is Tomcat and therefore the application is written in Java and therefore it might use Struts. There will be hundreds if not thousands of potential CVEs to check for given only that information.
You don't know if a given web application even supports file upload. How to you tell? Look at pages for the string "Click to upload"?? (Yes I have seen this done).
Given that it is often a hard task for a human to figure out how to use one of these applications, and that they would need to possess all kinds of valid data to even get the application into the state where it permits file upload, I think you can see this is not going to be easy.
Add in the fact that each state change may take a sizable fraction of a second and there could be thousands of plausible vulnerabilities to check for. The driving of the application has to be done inside a headless browser process which will often sail off into space paying you no heed..
Even if the 10,000 monkeys happen to type Shakespeare, there will be a mass of false positive results in the report which a human has to trawl through.
And the scan will not complete in finite time.
Not so easy after all.
What is relatively easy is to have humans keep an eye on the applications for which they are responsible, reading the security mailing lists for the dependencies and taking appropriate action.
That protects the data between the browser and the web server (or SSL offloader) from eavesdropping and tampering. It does not protect against the case where the attacker gains authorization, over HTTPS, to administrative access at the web server.
If the vulnerability got to admin level, then since the database can read everything, all is lost.
Encryption at rest essentially protects the disks from being compromised if they are physically stolen. Or if the attacker manages root on the system and reads at the sector level. But even then, if you are root, you can find the key, and you are in anyway.
Sure. But it is a good first step, a must really when dealing with sensitive data. Proper encryption at rest, like let's say a 256 bit AES encryption with a symmetric key itself encrypted with a PKI key pair with private key physically stored on a separate physical machine and frequent key rotation procedures in place would go a long way towards protecting the data.
It's not 100% clear exactly what happened at Equifax so it's hard to tell if at-rest encryption would have helped, from what I understand the working theory is that apache struts CVE-2017-5638 was exploited but it's not 100% clear exactly what went on so yes encryption might have not helped in this particular case.
Can you explain how, given that an administrator who has access through the web site can access all the information in the database, and given that an exploit on the front end gets administrator access, how in the world encryption does anything to prevent this? If at any point the web server has access to the data, the game is over. Encryption does nothing.
Well, yea there are scenarios where encryption alone doesn't help, but again it's one of the cornerstones of data safety.
Other security measures like restricting data access to a limited set of source ips, masking of the data returned to the browser etc are typically put in place when dealing with sensitive info in addition to encryption of data at rest.
Also, that's not what happened at Equifax, at least based on the "struts vulnerability" narrative that Equifax has been pushing.
With cloud computing so cheap and easy, I believe no encryption is safe. Once you have the data in your reach, you can run any dirty algorithm, if the data is worthy enough. Which in this case is millions of credit card details.
#2 isn't surprising. A ton of shops only patch on a quarterly basis, and even then they only consistently patch the OS. Only when there is a vendor (like my employer, Red Hat) sending out notices that there is a critical vulnerability do people move any faster.
Which is twice as bad for application dependencies. These don't even have quarterly patch cycles. Dependencies may be updated when a new release is deployed, which may be a couple times a year or never. One of my favorite questions for clients is "who is responsible for patching applications with no development team anymore?"
This is an underappreciated benefit of CICD. An automated process lets you get a central group to take ownership of third party libraries and their security, and then let them trigger all applications to rebuild and release. Especially with containers, this is essential.
> "who is responsible for patching applications with no development team anymore?"
Struts 2.2.3 (oldest vulnerable version) was released on May 7, 2011. Personally I find it easy to imagine an app depending on that, going into prod, and just chugging away, forgotten for 6 years on some obscure corner of an enterprise's public (or internal) web assets.
Getting business lines to sign off and invest the resources to update applications can be a huge challenge. Shiny new applications get all the attention, but maintaining and updating legacy applications is not prioritized.
I personally have no idea how large companies manage this. Every time I perform a full Docker update on all of the libraries and applications my app is running on, something breaks. Sure, tests catch it, but it takes a developer and time to figure out how to fix it. No idea how huge non-developer organizations handle this.
Not really. A specialist law firm can give you a risk assessment, give you an overview of all relevant regulations that you need to follow, recommend security firms, connect you to any federal agencies that can help or need to know and all discussed in layman's terms (or at least exec-speak). If your CSO is good, then it's a second opinion. If he or she is not, then it's the only way to get the help you need.
Why isn’t Equifax offering free credit monitoring and identity protection for all their customers? I had to go and put a fraud alert on mine but I feel this should have happened the moment they knew about the hack. Even now there is no talk of that. Of course I wouldn’t have confidence in what they do but am hoping the other two are not totally incompetent.
No one important will go to jail. If someone important is prosecuted and convicted they will be let off very very lightly.
No companies will learn any lessons from this, as companies are already doing everything they can within skillset limits of their employees and budgets, or they are not and don't give a shit.
No one really cares about the consumer in all of this, they will do the minimum not to look bad or face further legal issues but in the end people will invest lots of hours dealing with any problems that might arise. Who will pay for those hours? lol im just kidding, once again they dont give a shit really, good luck to us.
Nothing will change in the industry for companies like Equifax, this is a very small nitch market and we cant seriously hope for any government mandated reforms in today's political climate.
This will blow over soon and no one will remember it after the next breach or mass shooting or a war that Trump will start.
Sorry to say all this but sometimes people need to say the truth.
I don't disagree with any particular assertion you make.
Let me ask the obvious question. What do we do then?
(If this doesn't spark discussion I'll gladly contribute my own ideas, which involve finding methods of partially-civil disobedience to even the scales of inconvenience a little bit.
I honestly believe a large issue in the current system is that there's no real pressure being held against any of the people with real power. Why would someone who can bring in tens of millions _FROM FAILING AS BADLY AS ONE CAN FAIL_ in their job, give a damn about anything but optimizing that number and minimizing time/effort spent to achieve that? With senate and house incumbency rates hovering around 90%, why would a politician ever deviate from the status quo? The wellbeing of the people is so far separated from any of their driving incentives, we must find ways to realign that.)
What caused things to turn around in the 30’s was people getting so fed up they voted in a different crew who actually fixed things. Voting (and voter outreach) is still the best path to reform.
> Voting (and voter outreach) is still the best path to reform.
so we get the likes of Trump, who takes advantage of voter's desire for "a different guy".
You can do more than just voting - participate in civic conversations, run for office! Go to public hearings and senate committees, and make your opinions heard. If you're adventurous, organize protests or attent them. If you are able, use the media to promote your causes.
> (In a moment of true dystopian chaos, the official Equifax Twitter account repeatedly tweeted a phishing link, mistaking it for the breach-response page.)
These institutions are cancerous devolutions in their current form. They are no better than predatory lenders. What has motivated their current structure seems cynical at the core.
Most of the time person giving the orders is wholly inadequate to understand what they are doing.
At one of my previous jobs, the system I was working on was hacked years ago. You would think, this will give people some semblance of what not to do. No one seemed to have heard of the concept known as PII. Salaries of the entire company was sent out daily attached as a text file in an email to a very large group of people. The claim to do nothing was? It contains employee ids so it's kind of masked. And "many" people don't understand the data anyways.
Trying to explain the "powers that be" why this was against industry standard fell on deaf years. I am sure if things do go wrong they will blame that "one guy" who was unable to "convince" them of the right thing to do.
#3 bugs me. It gets asked every time there is a breach, but its irrelevant.
Encryption at rest is only useful if the only way to access the data is to type in the key. But for Equifax there are going to be hundreds or thousands of accesses per second. If you encrypt the data then you have added no protection at all because you still have a huge pipeline out through an always-on decryption mechanism. Any attacker is going to access the data through that mechanism and ignore the encryption completely.
It depends on how they get access, and it can prevent some types of access. For example, they can’t steal backups, and they may need access through a web app server that has the database key rather than being able to go directly to the database. It’s not a catch-all solution but it is correct security practice and they should have been doing it.
Depends on vulnerability. If you have RCE (like I understand Equifax had) then encryption at rest is useless, the code can access the data, obviously, so the RCE code can too (maybe with a little more work, but ultimately it is not a barrier anymore).
150 comments
[ 2.2 ms ] story [ 207 ms ] threadIt wasn't a phishing site, Wired. It was a site designed to point out that their approach is susceptible to phishing. Entering data in the form gave you a scolding.
> The Equifax Breach Exposes America's Identity Crisis When asked by multiple lawmakers why Equifax set up this separate site, Smith said that the company's main domain was not architected to process the enormous traffic the company knew would come its way after the announcement. In all, Smith said the independent breach response site has had 400 million consumer visits, which would have crumpled the main site.
They could have done a subdomain and pointed it at the site.
I'd argue it was a phishing site. Just fortunately it was controlled by white hats.
https://news.ycombinator.com/item?id=15295146
If folks missed it, it's worth skimming it to get a recap of the details.
Apparently they've never heard of subdomains, static pages, reverse proxies, and <insert many other solutions anyone on HN could come up with off the top of their heads>?
Especially given that random numbers input to the site returned that data had been compromised.
Most of the contracts nowadays have a penalty clause and a stated Service Level Agreements (SLA) around performing the vulnerability scan and closing the GAP within a stipulated time frame. Outsourcing companies might not be innovative but they don't like to lose money on failing SLAs.
Coming back to the issue of incompetence. I have often seen so called innovator of the just taking the credit for the work done by Vendor.
The way typical scenario rolls out is this. Senior management of of the company will throw a challenge to the senior management of Vendor. After a few discussions middle management of will assign the responsibility to their lowest level employees. After a lot of hard work when solution is in sight, all communication channels will be closed.
A few weeks later solution will be presented to their own management while keeping the vendor completely out of loop as if vendor was completely useless and they did all the hard work. Vendor will be presented as mindless robot who can just execute the instruction. Hell I have seen employees not even having the courtesy even the reword our solutions.
Incentives of employees are aligned with making the vendor look less effective or they themselves will be replaced at some point by their own higher management. I think its not their fault. It is just the environment where painting the vendor in positive color will be detrimental to employee's own job.
There is no incentive to say either good things or bad things about a Vendor, the incentive is to keep the focus on the internal people that were involved, because recognizing internal employees is critical for morale and general team cohesion. Recognizing a Vendor's effort could potentially backfire and lead to bad blood internally (happens with consultants all the time) and is not really a priority because the engagement is based on a negotiated contract that is primarily driven by financial or expertise reasons, not the human elements.
The fact is, the Vendor is the one that should be worried about the morale of their own employees. The Vendor's management should be praising the good work of the employees, and if there is blatant disrespect coming down from the Company, it is their job to either address it or at least justify to appease their rank and file.
I was just calling out the passive aggressive parent comment.
If there was a vendor involved - Equifax would have already jumped on that fact and blamed the vendor. They would have already played the oh-we-had-a-bad-vendor-and-fired-them-immediately game by now.
As an analogy, lending your car to a friend which turns out to have faulty brakes is negligent. But lending it to him when your mechanic told you the brakes don't work is probably grossly negligent, and will often have the the same repercussions as if you cut the brakes yourself.
Some of the best infosec people I know have completely irrelevant degrees.
That said, I'm not trying to make excuses for Equifax. From what we know the head of infosec there was not qualified to do the job.
I'm just a .NET developer, I have no idea what my code runs on.
I understand all the sarcasm expressed here, but this actually may be true if they run their own DNS service. Subdomain isn't a solution in this case, but separate domain is.
The no-bid contract was issued last week, as the company continued facing fallout from its massive security breach.
http://www.politico.com/story/2017/10/03/equifax-irs-fraud-p...
https://www.usds.gov/join#tours-of-duty
Yes the bureaucrats, lobbyists, and special interests will continue fucking everything up.
But the USDS is set up so that at least competent technologists can throw up some objective defense vs. the government officials who really think Equifax is a viable solution aka "Nobody ever got fired buying an IBM."
Not to mention that the idea of taking a massive pay cut and having to move to DC holds no appeal to me.
I think the todo list more than long enough to keep USDS busy for four or eight years even if you strike all the potentially politically sensitive things from it.
And, yeah, it's not easy. Solving big institutional problems isn't.
OTOH, a term at USDS may give you an opportunity to work around the fringes of the problem and grasp it's grand outline better. And the scale of government is such that a small improvement can be a big deal.
And yes, I've heard of video chat.
Encouraging to see people with modern best practices in mind working for the government.
* USDS limits employees to two consecutive years.
* 18F, it's sister agency, does support remote work. They're more like consultants and switch between different agencies project by project.
I, personally, wouldn't value that opportunity at $100k+. This is really Congress' fault for having such an awful pay system.
18F, which is part of the GSA, is something I could consider. But USDS is too close to Trump, Kushner, and the rest of Trump's swamp.
On one hand, you don't want to do support Trump. On the other hand, you don't want incompetent or malicious people supporting him instead.
If Trump's goal is to shrink and destroy the credibility of the federal government, which it seems to be, then there's a good argument for the latter.
There are clearly two sets of rules - one for large companies where they can brazenly flaunt laws whilst going from strength to strength. And another set of rules for everyone else - where if you take a wrong step it's all over!
People are being advised to perform credit freezes with Equifax, Experian, and TransUnion. For each credit freeze, you have to pay the credit agency $5-$10 depending on the state (only one state has forced them to do it for free; several states have no requirement to implement a credit freeze at all).
The credit reporting agencies are raking in tens of millions of dollars in these fees because one of them was incompetent. That's basically racketeering.
In case you think paying a company $10 not to disclose information it's gathered about you is a raw deal, TransUnion offers a different, free "credit monitoring" service (https://www.transunion.com/product/trueidentity-free-identit...). It's free, there's no catch (their marketing copy), except that the terms of service include an arbitration agreement. So you give up your right to sue TransUnion or be in a class action.
[0]: https://help.equifax.com/s/article/ka137000000DRjGAAW/How-do...
#4 I'm fine with, the CEO meeting on IT security quarterly sounds adequate to me. His involvement in IT security should not extend beyond ensuring policy is in place and being acted on. Quarterly is more than adequate to ensure this is being done.
It's important to understand that because it's the reason why your bosses will never care unless/until they have an experience like this, or until they expect someone important to be looking for an opening (audits, due diligence, etc).
Second, there are indeed some clowns and cronies hired at these sort of places, but a lot of the people are decently skilled. I'm speaking generally about the workforce at these larger enterprisey companies, as I don't know anyone at Equifax afaik.
Technical competence is less of an issue than intra-managerial political games like "shift the blame" and "silo defense", at least some of the time. Part of "intra-managerial games" is having the least-skilled people in the most authoritative positions, leaving decision authority with those least equipped to evaluate a situation and determine a responsible response.
I am sure that there is someone at Equifax who saw the news on this vulnerability and registered it as being something they needed to patch. I wouldn't be surprised at all to learn that this person didn't say anything because they learned long ago that there's nothing to be gained by actually trying to get the patch done in any non-routine manner.
I also wouldn't be surprised to learn they did try to raise the alarm and were mocked and/or shut down, both by other technical groups trying to avoid the appearance of an issue with "their" section of the system, and/or by management, who are likely to interpret such attempts as alarmism, if not plain political malfeasance.
Hate to say it but we're on the brink of a formalized licensing program for security, servers, etc. I am honestly surprised that the mainstream outlets and politicians have not been touting this yet, with all the high-profile and extremely costly data security breaches that have been occurring over the last few years, but I really expect it to come soon. I wouldn't be surprised at all to see Equifax become the Enron-like scapegoat for such legislation.
Exactly. I'm reminded about the tale about the two hikers and the bear. Two hikers are out hiking one day, when off in the distance they see a bear running their way to attack them. One starts to run, but the other stops to change his hiking boots to running shoes. The other says: "What are you doing? You can't outrun a bear!". To which the running shoe changer replies: "I don't have to outrun the bear, I just have to outrun you".
Same goes here. Don't have to really spend on security, just enough to not actually be the one of the two (or three or four) who suffer a breach.
Open source vulnerability management is a technology process that has levels of maturity. That being said, a freeware tool could have spotted this right away.
I can't really take the author too seriously after reading that.
The number being revised doesn't surprise me. I'd have been surprised if it hadn't been revised. I'd expect additional revision, as the investigation continues and the breach more fully understood.
For the record, that doesn't show I support them, nor am I making excuses for them. It's just the opposite. I believe them to be very, very inept - to the point of being incompetent, illsuited, and inexcusable. I expect additional revision, not because that is justified but because they simply suck that bad.
I can't think of a breach handled quite this poorly, and that includes my biases from the OPM breach. I'd say the OPM breach was far worse but handled marginally better. Equifax had to work hard to take the crown, but they have managed to edge them out by a narrow margin.
"Ah, s--- we got hacked."
"How many users?"
"145,534,902"
"Ouch, that's a lot. Let's say it was 143,000,000."
"He he he, nice."
---
I've misplaced more bytes in a megabyte.
E.g. I say one million but it turns out to be 1.05 million.
This explain why their site seem to return randomly who was affected.
Do these scanners ever work? Without naming names, the only reason we used this at a previous company was to kind of handwave around hey we have this tool doing regular security checks.
> The first excuse Smith gave was "human error." He says there was a particular (unnamed) individual who knew that the portal needed to be patched but failed to notify the appropriate IT team.
How is this one individual responsible for tracking all IT security vulnerabilities in all the technologies that equifax uses across its stack? How do they not have an admin team whose job it is to do these things?
The scanners do work, we used them at my company and they found this specific vulnerability. It surprised some development teams because there are libraries that use struts without really advertising it, so even applications that don't "use struts" had a transitive dependency on it, it was running in their container and showed up on the lists that were circulating.
See https://www.tenable.com/blog/apache-struts-jakarta-remote-co...
Yes, they do. Detecting the flaw is not a problem, but often operationally patching can be a big deal.
They do have a vulnerability management team; the responsibility does not fall on one person.
For all we know this unnamed individual is just a low-level developer, analyst or pentester who had some knowledge of the exposure and failed to act.
Compliance & Security "in the enterprise" is a smoke-and-mirrors game full of weird terminology divorced from the nuts-and-bolts issues with people who don't know what they're selling engaging with people who don't know what to buy, playing the industry-form and trade-show and junket and relationship game (probably no different from the DB market, the network market, the ERP market).
The DKs that fail at technology and don't really know it move on to these segments of the security market, to everyone's detriment.
Yes they do. But you have to have a process to triage what the scanners produce, and have a team whose job it is to keep the ops/dev side accountable.
They are quite useful when scanning all the internal parts of a datacenter. There's a fair amount of nitpicking but it helps weed out the obvious (like installing some open source package which defaults to some bad cipher for SSL, or leaving internal links unencryped under pretext that it's "safe behind the firewall").
Often, though, the issues flagged by the scanner trigger deeper conversations about security. That's where the real value is, but that requires an organizational culture that actually cares about security. Instead, many companies just throw money at the problem of "security" and consider the scanner will fix all their issues with zero effort.
That's the only reason for most of their clients. "We get periodic security scans from a respected vendor, yes sir!".
I can't rule out that the scanners might work, but if the vendor isn't naive, they'd be optimizing for their real target audience.
For this particular vulnerability? I doubt it. (Source: I worked on building such a scanner recently).
Let's begin with a vulnerability that is obviously detectable with an off-the-shelf scanner: "Buffer overrun in Linux CIFS Server". We can detect a host with this vulnerability by simply scanning the local subnet for live IP addresses then fingerprinting the host to determine it is Linux, checking if it responds on the SMB port and finally sending it a test exploit payload and see if it responds in the expected way (or crashes). This all takes a few ms.
But now consider what if the vulnerability is only exploitable by an authenticated session? Well we could have our scanner ask the operator for a set of credentials for each CIFS server it finds. But what if the vulnerability requires a mounted share? Well the scanner can ask the operator for the name of a share, or if it is lucky it could try to guess one. Perhaps we could be happy with the scanner identifying the version of Samba running on the server and concluding from patch history knowledge that it is vulnerable. But boxes get locally patched and often there isn't enough information in the externally visible version info to tell one way or the other.
Now think about trying to do this in the context of the vulnerability "If your application uses Struts and allows file upload then it might have this RCE vulnerability". We don't even know if we're using Struts in any of our applications. We may have hundreds of applications. Is it possible to tell from the outside that Struts is being used? (I'm not sure but probably not). You could note that the web server is Tomcat and therefore the application is written in Java and therefore it might use Struts. There will be hundreds if not thousands of potential CVEs to check for given only that information.
You don't know if a given web application even supports file upload. How to you tell? Look at pages for the string "Click to upload"?? (Yes I have seen this done).
Given that it is often a hard task for a human to figure out how to use one of these applications, and that they would need to possess all kinds of valid data to even get the application into the state where it permits file upload, I think you can see this is not going to be easy.
Add in the fact that each state change may take a sizable fraction of a second and there could be thousands of plausible vulnerabilities to check for. The driving of the application has to be done inside a headless browser process which will often sail off into space paying you no heed..
Even if the 10,000 monkeys happen to type Shakespeare, there will be a mass of false positive results in the report which a human has to trawl through.
And the scan will not complete in finite time.
Not so easy after all.
What is relatively easy is to have humans keep an eye on the applications for which they are responsible, reading the security mailing lists for the dependencies and taking appropriate action.
(HTTPS/TLS)
They didn't steal the disk. They gained access to the web server, which would have access to the unencrypted data.
If the vulnerability got to admin level, then since the database can read everything, all is lost.
Encryption at rest essentially protects the disks from being compromised if they are physically stolen. Or if the attacker manages root on the system and reads at the sector level. But even then, if you are root, you can find the key, and you are in anyway.
Sure. But it is a good first step, a must really when dealing with sensitive data. Proper encryption at rest, like let's say a 256 bit AES encryption with a symmetric key itself encrypted with a PKI key pair with private key physically stored on a separate physical machine and frequent key rotation procedures in place would go a long way towards protecting the data.
It's not 100% clear exactly what happened at Equifax so it's hard to tell if at-rest encryption would have helped, from what I understand the working theory is that apache struts CVE-2017-5638 was exploited but it's not 100% clear exactly what went on so yes encryption might have not helped in this particular case.
Also, that's not what happened at Equifax, at least based on the "struts vulnerability" narrative that Equifax has been pushing.
Which is twice as bad for application dependencies. These don't even have quarterly patch cycles. Dependencies may be updated when a new release is deployed, which may be a couple times a year or never. One of my favorite questions for clients is "who is responsible for patching applications with no development team anymore?"
This is an underappreciated benefit of CICD. An automated process lets you get a central group to take ownership of third party libraries and their security, and then let them trigger all applications to rebuild and release. Especially with containers, this is essential.
> "who is responsible for patching applications with no development team anymore?"
Struts 2.2.3 (oldest vulnerable version) was released on May 7, 2011. Personally I find it easy to imagine an app depending on that, going into prod, and just chugging away, forgotten for 6 years on some obscure corner of an enterprise's public (or internal) web assets.
I'm sure the record count will be revised upwards several more times.
Did it strike anyone as disheartening that legal was the first call?
No companies will learn any lessons from this, as companies are already doing everything they can within skillset limits of their employees and budgets, or they are not and don't give a shit.
No one really cares about the consumer in all of this, they will do the minimum not to look bad or face further legal issues but in the end people will invest lots of hours dealing with any problems that might arise. Who will pay for those hours? lol im just kidding, once again they dont give a shit really, good luck to us.
Nothing will change in the industry for companies like Equifax, this is a very small nitch market and we cant seriously hope for any government mandated reforms in today's political climate.
This will blow over soon and no one will remember it after the next breach or mass shooting or a war that Trump will start.
Sorry to say all this but sometimes people need to say the truth.
Let me ask the obvious question. What do we do then?
(If this doesn't spark discussion I'll gladly contribute my own ideas, which involve finding methods of partially-civil disobedience to even the scales of inconvenience a little bit. I honestly believe a large issue in the current system is that there's no real pressure being held against any of the people with real power. Why would someone who can bring in tens of millions _FROM FAILING AS BADLY AS ONE CAN FAIL_ in their job, give a damn about anything but optimizing that number and minimizing time/effort spent to achieve that? With senate and house incumbency rates hovering around 90%, why would a politician ever deviate from the status quo? The wellbeing of the people is so far separated from any of their driving incentives, we must find ways to realign that.)
so we get the likes of Trump, who takes advantage of voter's desire for "a different guy".
You can do more than just voting - participate in civic conversations, run for office! Go to public hearings and senate committees, and make your opinions heard. If you're adventurous, organize protests or attent them. If you are able, use the media to promote your causes.
These institutions are cancerous devolutions in their current form. They are no better than predatory lenders. What has motivated their current structure seems cynical at the core.
At one of my previous jobs, the system I was working on was hacked years ago. You would think, this will give people some semblance of what not to do. No one seemed to have heard of the concept known as PII. Salaries of the entire company was sent out daily attached as a text file in an email to a very large group of people. The claim to do nothing was? It contains employee ids so it's kind of masked. And "many" people don't understand the data anyways.
Trying to explain the "powers that be" why this was against industry standard fell on deaf years. I am sure if things do go wrong they will blame that "one guy" who was unable to "convince" them of the right thing to do.
Encryption at rest is only useful if the only way to access the data is to type in the key. But for Equifax there are going to be hundreds or thousands of accesses per second. If you encrypt the data then you have added no protection at all because you still have a huge pipeline out through an always-on decryption mechanism. Any attacker is going to access the data through that mechanism and ignore the encryption completely.