Our general advice to businesses and individuals utilizing Apache Struts as well as any other open or closed source supporting library in their software products and services is as follows:
1. Understand which supporting frameworks and libraries are used in your software products and in which versions. Keep track of security announcements affecting this products and versions.
2. Establish a process to quickly roll out a security fix release of your software product once supporting frameworks or libraries needs to be updated for security reasons. Best is to think in terms of hours or a few days, not weeks or months. Most breaches we become aware of are caused by failure to update software components that are known to be vulnerable for months or even years.
3. Any complex software contains flaws. Don't build your security policy on the assumption that supporting software products are flawless, especially in terms of security vulnerabilities.
4. Establish security layers. It is good software engineering practice to have individually secured layers behind a public-facing presentation layer such as the Apache Struts framework. A breach into the presentation layer should never empower access to significant or even all back-end information resources.
5. Establish monitoring for unusual access patterns to your public Web resources. Nowadays there are a lot of open source and commercial products available to detect such patterns and give alerts. We recommend such monitoring as good operations practice for business critical Web-based services.
#1 here can be automated. My current employer sells one such solution, though there are of course others. Here is a blog post about the vulnerability from back when it was discovered earlier this year.
> 1. Understand which supporting frameworks and libraries are used in your software products and in which versions. Keep track of security announcements affecting this products and versions.
This issue is solved by having your server OS download and install security updates automatically, which amounts to more or less uncommenting 1 line of config.
I am not the downvoter (and I didn't donwvote you) but I explain why it won't help. Struts is packaged as a jar file which is distributed inside of the war file which is basically application. The struct's jar is an essential part of the application and it can't be updated separately by the OS.
With respect to This issue is solved I suggest rewording it as This issue is partially solved, since in almost any real-world large web application, there is a significant number of third-party libraries that won't be automatically updated.
You don't come across this too often these days in OSS or startups. It's a pretty big explosion in complexity. What hot N-tier enabling frameworks have you read about recently?
The problem is that Equifax put within webserver's reach information that had no business being there. Apache Strut's vulnerability is unfortunate, but it shouldn't have been the keys to the kingdom, where the kingdom is the personal information of nearly the entire US adult population with a credit history. If I knew a service relied only on the security of a web server to protect deeply personal information such as my name and SSN I'd never sign up. We didn't have that choice with Equifax.
Having said that, definitely keep up on the vulnerabilities of software you use. It's hard though, especially when you're relying on a great deal of dependencies. A company the size of Equifax should have had a team dedicated to this. A team. It doesn't seem like they had anyone who knew anything about basic security at all.
This is a problem, yes, but I think it's worth keeping the perspective that it's a practically universal problem.
That doesn't mean we should be letting Equifax off the hook. But nobody should pretend that RCE on an on-prem webserver wouldn't be game-over, for the entire internal network, across the majority of the Fortune 100.
We have as a society chosen to trade the security of our personal information for greater and cheaper access to products and services. Nobody on HN will like that fact (at least, as stated bluntly like that; plenty of them do like the increase to their earnings capability that results from that tradeoff), but it's generally true.
It very much is a trade-off, because the expense that would go in to making a large enterprise network reliably secure would be extreme, and would drive prices up, retard economic growth, and most likely result in software engineers making substantially less money.
I'm not saying you should like the tradeoff, or that the existence of the tradeoff doesn't have implications for how we should regulate banks and credit ratings agencies. I'm just saying that if you think there's some other CRA that has reliably protected its IT assets from threats like this, you're wrong.
this seems like goalpost moving. first you said 'we as a society', yet there isn't much 'we' in this. it doesn't seem to me that people (the general public) are properly aware at just how vulnerable they are. they haven't therefore 'decided' that this is okay.
> the expense that would go in to making a large enterprise network reliably secure would be extreme, and would drive prices up, retard economic growth
Part of living in a society is recognizing that you still belong to it even when it makes decisions you don't agree with.
To more directly answer your question, though, consider a regime of general commercial liability for security incidents caused by failures of engineering (which Equifax is an example of). What would it be like to bring new products to market under such a regime? How much more difficult would it be to enter the field? How much more difficult to hire qualified people? What would that do to the prices of things made possible largely through software engineering work? When the prices of things increase, they're put out of reach of more consumers; markets constrict. What does that do for employment? What does reduced employment in a sector do to wages?
I'm not saying this to justify insecurity. I'm ambivalent about the tradeoffs our society has made on these issues. I just don't think we tend to grapple with them honestly on HN.
If that new product exposes scads of private data, then it's rather irrelevant. These credit scores aren't something most people signed up for, they are a product for financial lenders and not for the direct consumer.
Sorry, I have to disagree. Performing regular pen testing is costly, so skiping it is a tradeoff. Employing a team of security engineers is also costly and there aren't enough of them to make that viable for all companies. Tradeoff. But putting such data in direct access of a web server, a single RCE away from exposure? That's not a tradeoff, that's negligence.
You've lost me. I think you may have misunderstood my point about pentests. I mean "success" from the perspective of the testers: you asked me to own your network up, and I succeeded.
My point is: the most savvy companies contract annual sitewide pentests, in addition to all the other stuff they do to secure their networks, and virtually all of those pentests "succeed" (from the perspective of the testers).
Further: not only have I never personally been on an internal network pentest (the kind where you start with an IP "behind the firewall" and nothing else) that failed, but I'm not sure I've even ever heard of one that failed. I'm certain there are some networks that are so resilient that they've stymied a competent internal pentest --- but again, I wouldn't know which one, because I've never heard of that happening.
Apple’s payments systems haven’t been breached, nor do I recall a breach of Stripe and I would bet my teeth that their security is far better and more proactive than Equifax — a company that’s entire business is based on data brokering. If they can’t handle the “cost” of correct security or, at the very least keeping dependencies updated, then they ought not be in business. If security makes it expensive, then perhaps there should be more competition? The credit bureaus have a near monopoly so they don’t have to be aggressive about security as a Stripe might need to be. That complacency and just general malfeasance means they ought to be sued out of existence. I bet any number of Silicon Valley startups have better security than Equifax — at far lower budgets. It isn’t a cost issue — it’s a competency one.
More of my former colleagues work on the Stripe security team than at any other firm (besides NCC, our acquirer). They do a great job. But Stripe is a tiny company with very few products and a unified development culture with a management team organized around bringing technology products to market.
And they aren't perfect. They will eventually screw something up.
Comparing Stripe to Equifax is unreasonable. Equifax has about 10,000 employees. Stripe has less than 1,000.
And, once again, please don't try to win an argument by putting me in the position of defending Equifax. That's not what I'm doing. My point is that practically every company of Equifax's size has similar problems.
If Equifax is too big to have their shit together, they ought not be allowed to store sensitive data. If this were a HIPAA situation, they’d be facing a statutory penalty of about $10,000 per record.
Apple, Facebook and Google have far more data under their control than Equifax and they seem to take security far more seriously than Equifax. The Apple InfoSec team is extremely aggressive — InfoSec is ingrained in Apple culture. I am skeptical that Equifax takes security as seriously.
Being “big” or “other large companies have the same problem” isn’t an excuse. They have no problem fucking people’s credit by reporting inaccurate information — they should be held accountable since that information has such severe consequences.
My hatred of Equifax runs deep because they reported 7 state tax liens on my credit report that weren’t mine and were from a state in which I never lived. Despite providing all kinds of documentation, I ultimately had to sue them. I won the case and a bit of money, but the point is, they were willing to have a team of lawyers go to court against me and tie it up in court for several years — to defend something they knew was wrong. I even had letters from the state tax agency attesting to the error and they still had to be sued to fix the information. If their IT security is as arrogant as the rest of the company, then this breach doesn’t surprise me.
How is that relevant? It’s highly relevant because they have demonstrated a willful disregard for facts. They demonstrate an arrogance based on their near-monopoly power over people’s lives. They don’t care about accuracy — they care about the appearance of accuracy. It follows that they aren’t good stewards of the information with which they have been trusted.
The fact that they are too big and IT security is “hard” — that’s an excuse. General Motors might have bad IT security (I don’t know,) but they also don’t control the financial lives of almost every American.
If they are so big that they can’t handle it, then they ought to be shut down.
To be clear, I don’t have a problem with credit reporting. I have a problem with the fact that Equifax is going to survive this. Some big judgement happens, they’ll file for bankruptcy, they’ll reorganize and be right back at it.
No, they would not. HIPAA providers have suffered mass breaches repeatedly in the past. HIPAA is so toothless large firms just set aside cash to pay the fines rather than invest millions to speculatively comply.
Apple is great. I like Apple's product security team and I like Apple IS&T. But so what? I'm not denying that there are 10-20 firms in the world that have world-class security teams. I'm denying that those teams are in any sense an industry norm, even in the technology industry.
Your car kills people? Your biscuits are mouldy, or full of lead? We have strict rules, and companies that don't follow then end up in serious trouble.
If they can, why don't they? How many annual site-wide pentests have you heard of that failed hard? And those are the people savvy enough to conduct annual site-wide pentests!
Parent post is saying that that should constitute a failed audit. And it should be repeated as long as a 2 week effort by a pentesting team yields results.
The target for most network pen tests is to obtain domain by noon of the first day. Generally, they keep going until they succeed and/or someone in the chain cries uncle.
Insufficient incentives, either caused by inadequate regulation or a belief the regulation won't be enforced.
Think of the standards of network security and software validation used by, say, a Certificate Authority to keep their root certificates safe. Or by Google to secure GMail. Equifax just hasn't made reaching that standard their highest priority.
Certificate authorities to keep their root certificate keys safe... hmm... do go on.
:)
Google would be most people's exception to the rule I'm talking about here, having as they do one of the world's greatest information security teams. And Google has had major incidents!
> But nobody should pretend that RCE on an on-prem webserver wouldn't be game-over, for the entire internal network, across the majority of the Fortune 100.
You certainly have more experience with this than the vast majority of the rest of us, but with that kind of data on millions of people, shouldn't it be, and why isn't it, standard practice to put servers storing that data behind an extremely limited API, with a firewall in place to monitor queries and responses and severely rate-limit or stop additional traffic if something suspicious happens, until the monitoring people can take a look?
Because that not only adds operational expense but also requires engineering teams to have a level of systems engineering expertise and diligence that virtually no line-of-business software project has.
*adds operational expense without a counterparty willing to pay for that expense
Those with data on Equifax's servers aren't Equifax's customers, so other than "not having a PR cluster&#@_" economic incentives are not aligned to prevent this.
I've spent about half my professional career finding vulnerabilities in the software of developers who are convinced they're "qualified to design with least privilege".
And, even if you know what you are doing, this is intrinsically a hard problem, you need to know exactly what information should be accessible to each server
You can add all the security layers you want, but if the web server needs to display a piece of information, the web server must have access to that information. The keys must be in the car somewhere.
Also... even with multiple layers, what do you think the backend stack would be? Unless it's built by a separate team deliberately using a different technology platform, you're stuck with the same Struts (or whatever) vulnerability.
> We have as a society chosen to trade the security of our personal information for greater and cheaper access to products and services.
Agree, and it's one of the reasons I generally try to avoid using products that are low-cost or free (the other reason being questionable sustainability, regardless of funding). At some point, products need to break even or they will go out of business. As a customer, you are always paying for the product in some form or another. For something that's free, you might be paying with your personal information now... or later if running on a low budget has caused the product developer to cut some corners when it comes to infosec.
Shouldn't we hold credit reporting agencies to a higher standard than the rest of the Fortune 100 though? After all, they store sensitive personal information about the vast majority of US adults.
This was a reaction to Quartz Media's (QM) article that made Apache Strut appear to be incompetent. [0]. QM updated the article. [1]. Equifax was vulnerable for at least two and half months, and database access wasn't detected. [2].
> The problem is that Equifax put within webserver's reach information that had no business being there.
Can you please link to your preferred tutorials (or other sources) which explain how to set up the most secure server architecture, with regards to this story/subject, for those of us who would like to do a better job?
It doesn't really exist. You "can't" because some computer still needs access to the data.
You can do layered security, but how? Think about a customer service rep. They need access to ~everybody's data at a moment's notice because they're a CS rep. That's what they do. That means every CS rep machine has access to all the data.
I'm not sure this is solvable. SSN's are the problem, not the way that we store SSNs. They still need you to read off your SSN when you call in, so you can't just encrypt it.
Think about Equifax in particular. They're a credit bureau. Of course they need all the data available all the time -- that's what a credit bureau does. The question is, how would you partition the data so that a failure in one place wouldn't lead to exfiltrating all the data everywhere?
The trouble is, it wasn't just "they popped Struts and then ran an scp." Even if you partition the data, they popped Struts! They have RCE inside your network. It's absurdly easy to pivot from one machine to another in a given network. You keep a helpful ~/.bash_history file that says exactly what's important and where everything is installed. Stuff like this is why I argued for deleting it, since it was the #1 most helpful tool I had that helped me pivot inside a target network. But then you get people yelling about taking away their bash history.
You can imagine how terrible this problem becomes in practice. Nobody here gives a crap about Equifax; they just want their heads to roll. Unfortunately you'll be next on the chopping block because no one has solved these issues. As soon as the world goes crazy over the lack of security, it's game over for us all because it leads to a highly regulated world that won't be much safer.
At the very least, you could separate identity and data access and treat each person's data as a separate tenant.
The main data store that contains everyone's credit reports (high side), should have physical and network isolation from the public site (low side). Only copy from high side to low side when someone becomes a customer.
(disclaimer: IANA security expert, but the perfect is the enemy of the good)
> You can do layered security, but how? Think about a customer service rep. They need access to ~everybody's data at a moment's notice because they're a CS rep. That's what they do. That means every CS rep machine has access to all the data.
That is true. But you can observe that a typical CS rep accesses maybe 10 customer's data sets per hour. If they work 8 hours a day, 200 days a year, that's 16k customers they could leak per anno without arousing suspicion Maybe 60k if you allow a fudge factor of 4 to prevent false-positives from highly motivated CS reps.
But only if you monitor for this kind of thing. (And better yet, you should rate limit, and monitor).
Most security problems don't have perfect solutions, but they do have some solutions that get you 80% there. And if you invest in sufficiently many such 80% solutions, you get to a point where it's not economical for an attacker to continue hammering on your application.
Monitoring for anomalous access patterns is a good idea.
It wouldn't have helped in the face of RCE in struts. RCE in struts would let the adversary scp off all the files that make up the database, for instance. This would completely bypass any application-level checks or monitoring.
There is no one-size-fits-all solution, and probably no "most secure", but lots of people have thought about stuff like that. If you google for "software architecture for security" or such terms, you'll find some helpful material.
The principles below it aren't too hard though: encrypt data in transit and in rest, think hard about key management. Keep attack surface minimal. Validate at all layers where it makes sense (for example in a multi-tenant application, you should authorize access to a row by tenant both in the application layer, and use row-level security in the DB as second layer of defense). Make your assumptions explicit; write them down. If possible, write automated tests that check if they still hold up.
While the principles are not hard to understand, designing an application with them in mind can be quite a different beast, and keeping stuff secure as features and integrations creep in requires quite some dedication.
I'd guess that it didn't rely on the security of a web server, but the entry point was the "thin end of the wedge" for probing and deploying more sophisticated attacks.
At some point, the front end of any web application has to go to the back end database to get information. So there's just no way to cleanly and completely isolate the front facing parts of an application from possible breach.
Most transactional websites need access to the core customer database, otherwise they could not function. Just DMZing everything is not always possible.
Maybe segmentation is the only possible defence - customers A-C served out of this DB with a different stack to limit the damage.
But once they get in, they get in, right. Its just a matter of finding the next vulnerability.
As long as you are providing a rich customer experience, with the ability to generate reports based on historical information, you have risk.
I completely agree about having a dedicated team, and I'd expect a company of their nature to be at the forefront of security best practices.
I just checked some Equifax domains against SSL Labs, and while their Canadian site (https://www.econsumer.equifax.ca) scores an A-, it has no forward secrecy. I'm surprised to see a modern web server not supporting FS today. Worse, the main entry point to their Canadian site (http://www.consumer.equifax.ca) as indexed by Google does not redirect to a TLS enabled page, although they do seem to have a TLS endpoint for that domain -- but not sure how people are expected to get to it.
Edited to add: The first link is only accessible through a redirect by clicking on the "Get Started" button on their main Canadian site. Furthermore, even selecting Canada from the drop-down on https://www.equifax.com/personal/ redirects to the insecure non-TLS site.
Wouldn't an extension like HTTPS Everywhere do it for you? (Provided you're savvy enough to have it installed; in other words, they should still put in that redirection...)
We won't really know until Equifax releases the full RCA (root cause analysis).
The attack could be way more sophisticated and involve several hacks through the stack.
Injecting a JSON blob to the Struts REST deserializer that effectively results in executing payload code would indicate high investment in the attack.
The hacker(s) possibly had insider information. They possibly used social hacking techniques to acquire data decryption keys.
Given Equifax stores more than 143M records, the initial report would imply some sort of data segmentation limited the scope of damage. If the doors were wide open, why didn't the hackers take 100% of the records?
None of this excuses Equifax or removes liability (I've personally initiated a credit freeze and will likely need a LifeLock Plus account for the rest of my life thanks to this breach).
>Injecting a JSON blob to the Struts REST deserializer that effectively results in executing payload code would indicate high investment in the attack.
Why does it indicate high investment? IMO it's similar to SQL injection attack. People, just take a look at points where complicated data can be deserialized and check whether adequate checks can be overcome.
As I see from their report here: https://cwiki.apache.org/confluence/display/WW/S2-052 they use XStream to deserialize REST data. XStream is basically Java serialization/deserialization to XML instead of binary. As far as I remember from my experience of working with XStream, XStream can deserialize almost any object, even the ones which aren't serializable by default. It's quite trivial to go from here to execution of arbitrary code.
> It's quite trivial to go from here to execution of arbitrary code.
Code that does what? Dumps all environment vars? OK, maybe that produces a DB2 connection string. Refactor code, re-inject, does the connection work? OK. Refactor code, re-inject and dump all tables. etc...
This attack requires an iterative trial-and-error probing process, typically done behind a proxy for many weeks/months.
From Equifax's PR: "the unauthorized access occurred from mid-May through July 2017.".
That's a reasonable amount of time to probe all systems involved and develop sophisticated code.
For most attackers, it is vastly more efficient to run automated sql/wordpress-vuln/ssh-user scanners dropping bitcoin miners and botnet nodes. Those require zero human effort and self-replicate. That gives you less publicity than a hundred million personal finance records, but it has a much higher success chance and a more reliable income.
Equifax is in the business of selling this information to other companies. They are offering services, which allow banks and other institutions to check this information through web service interfaces. It is kind of a core functionality that this information is available through a web server. More old fashioned way would have been to expose this information through some obscure protocol, but I don't think that would make it any more secure.
Now a different thing is that should this information be available through a server that is accessible directly through the Internet (and I don't know if it was) or if they should require for example a dedicated VPN connection.
> Having said that, definitely keep up on the vulnerabilities of software you use. It's hard though, especially when you're relying on a great deal of dependencies. A company the size of Equifax should have had a team dedicated to this. A team. It doesn't seem like they had anyone who knew anything about basic security at all.
If you don't have a full team for this, another strategy is to rely on a vendor, and try to obtain as many dependencies as possible from this vendor. Of course, you still need a process for updating the vendor-supplied components as quickly as possible.
This is not Struts' "fault"... An entity like Equifax cannot hold such private info and power over our finances, then not have multiple layers of protection to prevent this. You expect bugs in user faceing software, so you protect against breaches so that you don't expose 140 MILLION damn records..
You could stop at "An entity like Equifax cannot hold such private info and power over our finances"
Credit rating is an instrument of control used by banks and credit card organizations to force us into debt we don't want.
This system is CRAZY and I don't understand why normally anti-establishment and "leave me alone" Americans so sheepishly agree to be involved in a system that hurts our individual liberties every day.
When I came to this country the "credit score" system was the single most incomprehensible thing about american society.
Still is.
I suspect the statement is meant this way: in order to build credit (to buy a house later for example), you need to get in debt: you need to get credit cards, loans, etc... and "climb the ladder". Only then will you have a credit worthy status to get a loan for a house.
When I came to the US from Europe, I had no credit history and that made it difficult to get any type of credit card, or loan, or anything. I thought it was a catch 22 since I couldn't build credit because I didn't have credit, but there is this thing called "Secured Credit Card", where the bank lends you your own money (that you deposit) to see if you can repay yourself. Had to do that for about a year, and I also managed to get a Macy's card with a $100 credit limit (yes, one hundred) that forced me to shop there to "build credit".
After that the climbing of the ladder started with credit limits increasing slowly, etc... up to the point that I was able to get a loan for a house.
There's a bit of strategic planning to building credit worthiness, and that requires you getting into some "managed debt". I think that was the spirit of the comment you are asking about.
I've always thought it was a little ironic that increasing your credit score is done by getting more credit line than you need then not actually using it (reducing the percentage of your credit line that you do use assuming expenses are fixed).
This is because credit score is a measure of two things. The first one is obvious - that you are capable of paying your debt if you find yourself in it. The second one however is how valuable you are to the banks. What good are you to them if you never borrow more money than you need? Their best customer is always in debt but regularly making payments (with interests of course).
A history of regular payments on a card with a $100 limit doesn't require you to go into debt in any way except for an accident of nomenclature. Pay your $5 every month; you'll have normal credit and no carried debt.
No. That is how you build a credit score. If your goal is to build a credit score, it is not worthless, and it is also not a risk to your personal finances or a means by which it is even possible to get into nontrivial debt.
A large part of the credit score concept is to flag people for whom paying a trivial bill on time is difficult.
American Express has helped me out twice with this because they are a global entity. Originally became a member in Germany, moved to the UK and all I had to do was change my address with Amex and I had a UK credit card with no credit history in that country. I did the same thing when I came to the US, called them up and said I’m in the US now. Got a new card and my credit score was at 700 or so within a couple of months.
> why normally anti-establishment and "leave me alone" Americans so sheepishly agree to be involved
Because it's a private system, and it's at least theoretically voluntary. Nobody forces you to get credit cards or borrow money.
Credit cards are immensely convenient. I hardly ever carry cash anymore. I buy everything from coffee to lunch to groceries to gasoline using credit cards, and pay the balance every month. If I had to make arrangements to always carry cash to cover those sorts of expenses, it would be frustrating.
In the current system paying cash out with a debit card means you are losing money. Most businesses charge the same no matter what method you pay with, but credit cards are usually the only way to get part of that profit.
I've seen a slight increase in debit card rewards, like with Yelp, airline company dining rewards, and BoA rewards, but those are for the most part random and need to be planned spends. My credit cards will always reward me for using them.
I think this highlights the general problem of Java EE style architecture. There are many moving pieces and many permutations of how they interact together. It's practically impossible to understand it in its entirety, and thus impossible to guarantee that it's secure. You're basically plugging holes as you find them, but you're never sure that there aren't more holes you don't know about.
The real problem is that the Java EE style architecture introduces so many bullshit layers that you lose sight of real layers that you need to worry about.
The EE eco system while not perfect is basically a best in class for framework for everything we've learned about building business apps over the last 20 years.
This doesn't refute the point you're replying to, you're essentially saying "Nuh uh it's the best"
Too many knobs and dials can be a big problem, and verifying you have everything set up optimally for production is tough with all that configuration present.
> Too many knobs and dials can be a big problem, and verifying you have everything set up optimally for production is tough with all that configuration present.
this seems like a tautology. too many knobs is too many.
What style is that? The style of composing persistence technology, Authentication and Authorization, dependency injection, messaging, web communication etc to make a business application? If you know of a magical tech that doesn't aim to solve these problems in a standardized modular way so that the developer isn't socket programming from the ground up then please chip in.
"There are many moving pieces and many permutations of how they interact together."
Agreed. The problem space isn't todo apps or "Hotdog/No hotdog".
"It's practically impossible to understand it in its entirety, and thus impossible to guarantee that it's secure."
I don't understand the finer points of the myriad of TLS cypher suites etc, probably means I can't guarantee that they're secure. But what other option do I have? roll my own? Nope I'll take sensible defaults and twiddle the knobs where the client's requirements demand something different.
"You're basically plugging holes as you find them, but you're never sure that there aren't more holes you don't know about." - this is FUD.
The JEE stack is a monolith with many deeply intertwined components. The alternative is having a stack composed of small focused libraries that can be composed to fit the needs of the particular application.
I'm not talking about understanding the finer points of the internals such as the implementation of the TLS cypher. I'm talking about the complexity of how different moving pieces fit together in the stack.
This isn't FUD, it's the reality and it causes security breaches all the time in the real world.
CVE-2017-9805 is yet another deserialization vulnerability from a language and set of frameworks that seem designed to produce an endless stream of deserialization vulnerabilities. This should very much reflect on Java EE.
Normalized by usage I would be surprised if it's actually as high as you suggest. The more popular a platform is, the more attention from bad actors there will be. Every platform has had similar vulnerabilities along the way.
Is Struts Java EE? There are quite a few web frameworks in the Java space and anyone who reviews Apache CVE history will see immediately that Struts has by far the most serious problem with exploits of any product they do, let alone any Java product.
Hm, I feel like this should be true for most web stacks which automagically do everything for you. At least PHP and Ruby-frameworks have had trouble with convenient just-working auto-serializers and auto-deserializers.
I agree absolutely, any framework that does things magically is difficult to reason about. Things that are difficult to reason about are inherently insecure in my opinion.
I don't think you're being very fair or particularly accurate.
There is a massive difference between an over-engineered WebSphere-based deployment using EntityBean EJBs (1.0 no less!) and a tight deployment of Jetty + Struts 2.0 (or more recent frameworks). Anyone who has done real Java development knows this difference. The embedded Jetty + web framework movement was the start of today's modern frameworks, such as Play and various microservices toolkits.
The Servlet spec was probably the only good thing to come from JavaEE and Struts falls into that camp, even if it's fairly dated these days. Painting it with a broad "Java EE" brush, with all the bloat that went with it, isn't right.
I'm not jumping to any conclusions. The exploit appears to be a deserialization bug in a the REST plugin for Struts that allowed the attackers to perform arbitrary code execution:
>The bug specifically affects a popular plugin called REST, which developers use to handle web requests, like data sent to a server from a form a user has filled out. The vulnerability relates to how Struts parses that kind of data and converts it into information that can be interpreted by the Java programming language. When the vulnerability is successfully exploited, malicious code can be hidden inside of such data, and executed when Struts attempts to convert it.
>That means intruders could easily inject malware into web servers, possibly without being detected, and use it to steal or delete sensitive data, or infect computers with ransomware, among other things.
None of that would've happened if Struts didn't have remote code execution bugs in it either. My original point was precisely that there are many attack vectors, and a complex stack makes it nearly impossible to account for all of them.
This kind of statement is the basis of a lot of false equivalence. You're right: there have been exploits found in many different types of systems. Is it easier for bugs to hide in complex systems? Are some systems more complex than others? I think it's pretty unconstriversial to say the answer to both of these are yes. I don't see your parent saying that this could only occur in this stack.
Ok, let's try from this angle? Are you familiar with Capability Maturity Model https://www.sei.cmu.edu/reports/93tr024.pdf? Now we know how to build reliable software to fly airplanes and land men on the moon. The CMM level 5 is a pretty effective component of this. Now if you take the bottom 400 companies of the fortune 500, where do you think their public-facing web sites fare on the CMM scale? As another part of the exercise, what software that does anything remotely interesting is not complex?
Thanks for the link. I haven't read it, but from the name and the little I know about NASA software development practices, it's not clear to me at all that these are inconsistent. They're orthogonal, if anything. Are they not? Is there something in CMM that says that complex systems are not more difficult to understand (and therefore more likely to hide/contain bugs)?
Where a particular set of companies falls on such a scale is also independent of this, isn't it? I feel like I'm missing something (which admittedly may be because I haven't read the linked PDF), so if you'd be willing to take the time to expound, I'd appreciate it.
The point of the CMM is that it is a measure of an organization's capability to develop error free software. The trick is that it is a very time consuming, deliberate process. Not exactly agile. Read: vastly increases cost of developing software, certainly increases time to market.
Thanks for the synopsis. Given what you've written here I don't see how that would be in opposition to what I've written above. A team could have a goal of writing simpler, composable code and use CMM to measure their capability against CMM criteria. Or am I misunderstanding?
Without question. But CMM covers more than that--how requirements are gathered, reviewed, approved; who in the chain has the proper training and at what level. It much more than just coding. And how all of this is measured. So if the goal is writing simpler code, it is about how you would measure that.
Pick any complex web application, whether it be written in python, .net, C, C++. The complexity of anything interesting is beyond the scope of one mind to comprehend.
Interesting to also note that the "workaround" listed in the first announcement by the Apache Struts team was wrong. I followed the directions and my web application was still vulnerable. They have since updated it, but without an announcement.
What does her competence in music theory say about her knowledge of IT security? Intelligent humans beings usually aren’t limited to a single area of interest.
I honestly couldn't care less what she majored in a quarter century ago. Do you honestly believe a CS bachelors degree from a couple of decades ago would have any relevance to information security today?
Certainly more relevance than a Music degree. Having been a security researcher in the past would have helped too, but that's not on the menu either.
I would like to understand the thought process that resulted in hiring a Music major to run an organization in charge of protecting one of the largest troves of PII in the history of mankind.
Around 5 years into career, it does not matter what you studied or whether you finished. It matters only what you do now and how old are you. The process starts with music major getting job out of music industry and then working her way up the corporate ladder just as anybody else ever worked his way up - except with having some disadvantage in the beginning.
What's a "batchelor" degree? If you meant "bachelor", I was quite pleased to see that the surgeon who was about to operate on me also had a pre-med engineering degree from MIT.
I described how it works in this industry. Lawyers and doctors work in different industry and are bound by different laws.
There was no line of reasoning, only observation. You asked the question, I answered. I assumed you work in different place/industry and your question is honest quest to find how things work.
It is pretty common to have people in tech who studied something else. Or dropped out. And security is one of those places where career changers end up more often. Also, general computer science majors actually learn only very little security in school. So, that is how and why it probably does not even matters all that much.
Yet moreover, corporate security is quite uncool to people who do academic anything (including security) or worked as security researchers. Corporate security is more about negotiation, enforcement, standards compliance and other significantly less interesting things to someone who like research or low level details.
As a Java developer, this gives me the lesson not to use smart meta programming facilities, like reflection, where possible. You reduce amount of code, but at the cost of making your protocols injectable to arbitrary code often in unobvious ways.
That's a good conclusion to take away. But reflection and dynamic bytecode manipulation is used in Java all over the place. Class loaders are a core Java feature, and reflection is used in almost all modern annotation-based packages for DB access, object serialization, remoting, dependency injection, etc.
Heck, any JITing language requires process images allowed to execute code in dynamic memory segments, such that basic NoExecute hardware features can't be used. Combine this with Java server-side apps being run in a single process/address space, and I hope you can see that, if nothing else, from a security PoV Java is a dead end.
153 comments
[ 3.9 ms ] story [ 106 ms ] thread1. Understand which supporting frameworks and libraries are used in your software products and in which versions. Keep track of security announcements affecting this products and versions.
2. Establish a process to quickly roll out a security fix release of your software product once supporting frameworks or libraries needs to be updated for security reasons. Best is to think in terms of hours or a few days, not weeks or months. Most breaches we become aware of are caused by failure to update software components that are known to be vulnerable for months or even years.
3. Any complex software contains flaws. Don't build your security policy on the assumption that supporting software products are flawless, especially in terms of security vulnerabilities.
4. Establish security layers. It is good software engineering practice to have individually secured layers behind a public-facing presentation layer such as the Apache Struts framework. A breach into the presentation layer should never empower access to significant or even all back-end information resources.
5. Establish monitoring for unusual access patterns to your public Web resources. Nowadays there are a lot of open source and commercial products available to detect such patterns and give alerts. We recommend such monitoring as good operations practice for business critical Web-based services.
http://blog.sonatype.com/2017-struts2-vulnerability-exploit-...
This issue is solved by having your server OS download and install security updates automatically, which amounts to more or less uncommenting 1 line of config.
Edit: Downvoter(s), please comment.
What are your best sources/tutorials explaining the perfect server architecture setup which address this point?
You don't come across this too often these days in OSS or startups. It's a pretty big explosion in complexity. What hot N-tier enabling frameworks have you read about recently?
Having said that, definitely keep up on the vulnerabilities of software you use. It's hard though, especially when you're relying on a great deal of dependencies. A company the size of Equifax should have had a team dedicated to this. A team. It doesn't seem like they had anyone who knew anything about basic security at all.
That doesn't mean we should be letting Equifax off the hook. But nobody should pretend that RCE on an on-prem webserver wouldn't be game-over, for the entire internal network, across the majority of the Fortune 100.
We have as a society chosen to trade the security of our personal information for greater and cheaper access to products and services. Nobody on HN will like that fact (at least, as stated bluntly like that; plenty of them do like the increase to their earnings capability that results from that tradeoff), but it's generally true.
I'm not saying you should like the tradeoff, or that the existence of the tradeoff doesn't have implications for how we should regulate banks and credit ratings agencies. I'm just saying that if you think there's some other CRA that has reliably protected its IT assets from threats like this, you're wrong.
this seems like goalpost moving. first you said 'we as a society', yet there isn't much 'we' in this. it doesn't seem to me that people (the general public) are properly aware at just how vulnerable they are. they haven't therefore 'decided' that this is okay.
> the expense that would go in to making a large enterprise network reliably secure would be extreme, and would drive prices up, retard economic growth
what is this based on?
To more directly answer your question, though, consider a regime of general commercial liability for security incidents caused by failures of engineering (which Equifax is an example of). What would it be like to bring new products to market under such a regime? How much more difficult would it be to enter the field? How much more difficult to hire qualified people? What would that do to the prices of things made possible largely through software engineering work? When the prices of things increase, they're put out of reach of more consumers; markets constrict. What does that do for employment? What does reduced employment in a sector do to wages?
I'm not saying this to justify insecurity. I'm ambivalent about the tradeoffs our society has made on these issues. I just don't think we tend to grapple with them honestly on HN.
My point is: the most savvy companies contract annual sitewide pentests, in addition to all the other stuff they do to secure their networks, and virtually all of those pentests "succeed" (from the perspective of the testers).
Further: not only have I never personally been on an internal network pentest (the kind where you start with an IP "behind the firewall" and nothing else) that failed, but I'm not sure I've even ever heard of one that failed. I'm certain there are some networks that are so resilient that they've stymied a competent internal pentest --- but again, I wouldn't know which one, because I've never heard of that happening.
And they aren't perfect. They will eventually screw something up.
Comparing Stripe to Equifax is unreasonable. Equifax has about 10,000 employees. Stripe has less than 1,000.
And, once again, please don't try to win an argument by putting me in the position of defending Equifax. That's not what I'm doing. My point is that practically every company of Equifax's size has similar problems.
Apple, Facebook and Google have far more data under their control than Equifax and they seem to take security far more seriously than Equifax. The Apple InfoSec team is extremely aggressive — InfoSec is ingrained in Apple culture. I am skeptical that Equifax takes security as seriously.
Being “big” or “other large companies have the same problem” isn’t an excuse. They have no problem fucking people’s credit by reporting inaccurate information — they should be held accountable since that information has such severe consequences.
My hatred of Equifax runs deep because they reported 7 state tax liens on my credit report that weren’t mine and were from a state in which I never lived. Despite providing all kinds of documentation, I ultimately had to sue them. I won the case and a bit of money, but the point is, they were willing to have a team of lawyers go to court against me and tie it up in court for several years — to defend something they knew was wrong. I even had letters from the state tax agency attesting to the error and they still had to be sued to fix the information. If their IT security is as arrogant as the rest of the company, then this breach doesn’t surprise me.
How is that relevant? It’s highly relevant because they have demonstrated a willful disregard for facts. They demonstrate an arrogance based on their near-monopoly power over people’s lives. They don’t care about accuracy — they care about the appearance of accuracy. It follows that they aren’t good stewards of the information with which they have been trusted.
The fact that they are too big and IT security is “hard” — that’s an excuse. General Motors might have bad IT security (I don’t know,) but they also don’t control the financial lives of almost every American.
If they are so big that they can’t handle it, then they ought to be shut down.
To be clear, I don’t have a problem with credit reporting. I have a problem with the fact that Equifax is going to survive this. Some big judgement happens, they’ll file for bankruptcy, they’ll reorganize and be right back at it.
Apple is great. I like Apple's product security team and I like Apple IS&T. But so what? I'm not denying that there are 10-20 firms in the world that have world-class security teams. I'm denying that those teams are in any sense an industry norm, even in the technology industry.
Your car kills people? Your biscuits are mouldy, or full of lead? We have strict rules, and companies that don't follow then end up in serious trouble.
The system/network I inherited at new job it absolutely would be, it won't be in a year though.
We aren't a fortune 500 but there are simple, practical things you can do to limit the damage an RCE on a webserver can actually do.
But I disagree, security teams can and should manage large networks with large numbers of stakeholders.
What's needed is an architectural security review, which is even rarer than people conducting an annual pentest.
Think of the standards of network security and software validation used by, say, a Certificate Authority to keep their root certificates safe. Or by Google to secure GMail. Equifax just hasn't made reaching that standard their highest priority.
:)
Google would be most people's exception to the rule I'm talking about here, having as they do one of the world's greatest information security teams. And Google has had major incidents!
You're too optimistic :(
You certainly have more experience with this than the vast majority of the rest of us, but with that kind of data on millions of people, shouldn't it be, and why isn't it, standard practice to put servers storing that data behind an extremely limited API, with a firewall in place to monitor queries and responses and severely rate-limit or stop additional traffic if something suspicious happens, until the monitoring people can take a look?
Those with data on Equifax's servers aren't Equifax's customers, so other than "not having a PR cluster&#@_" economic incentives are not aligned to prevent this.
Also... even with multiple layers, what do you think the backend stack would be? Unless it's built by a separate team deliberately using a different technology platform, you're stuck with the same Struts (or whatever) vulnerability.
Agree, and it's one of the reasons I generally try to avoid using products that are low-cost or free (the other reason being questionable sustainability, regardless of funding). At some point, products need to break even or they will go out of business. As a customer, you are always paying for the product in some form or another. For something that's free, you might be paying with your personal information now... or later if running on a low budget has caused the product developer to cut some corners when it comes to infosec.
[0]: https://web.archive.org/web/20170908214344/https://qz.com/10...
[1]: https://qz.com/1073221/the-hackers-who-broke-into-equifax-ex...
[2]: https://baird.bluematrix.com/docs/pdf/dbf801ef-f20e-4d6f-91c...
Can you please link to your preferred tutorials (or other sources) which explain how to set up the most secure server architecture, with regards to this story/subject, for those of us who would like to do a better job?
Thanks a lot!
You can do layered security, but how? Think about a customer service rep. They need access to ~everybody's data at a moment's notice because they're a CS rep. That's what they do. That means every CS rep machine has access to all the data.
I'm not sure this is solvable. SSN's are the problem, not the way that we store SSNs. They still need you to read off your SSN when you call in, so you can't just encrypt it.
Think about Equifax in particular. They're a credit bureau. Of course they need all the data available all the time -- that's what a credit bureau does. The question is, how would you partition the data so that a failure in one place wouldn't lead to exfiltrating all the data everywhere?
The trouble is, it wasn't just "they popped Struts and then ran an scp." Even if you partition the data, they popped Struts! They have RCE inside your network. It's absurdly easy to pivot from one machine to another in a given network. You keep a helpful ~/.bash_history file that says exactly what's important and where everything is installed. Stuff like this is why I argued for deleting it, since it was the #1 most helpful tool I had that helped me pivot inside a target network. But then you get people yelling about taking away their bash history.
You can imagine how terrible this problem becomes in practice. Nobody here gives a crap about Equifax; they just want their heads to roll. Unfortunately you'll be next on the chopping block because no one has solved these issues. As soon as the world goes crazy over the lack of security, it's game over for us all because it leads to a highly regulated world that won't be much safer.
See how resistant we all were to the mere mention of increasing security procedures: https://news.ycombinator.com/item?id=14104156
I was snarky in that thread which certainly didn't help, but you have an uphill battle if you want to implement anything on a large scale.
The main data store that contains everyone's credit reports (high side), should have physical and network isolation from the public site (low side). Only copy from high side to low side when someone becomes a customer.
(disclaimer: IANA security expert, but the perfect is the enemy of the good)
That is true. But you can observe that a typical CS rep accesses maybe 10 customer's data sets per hour. If they work 8 hours a day, 200 days a year, that's 16k customers they could leak per anno without arousing suspicion Maybe 60k if you allow a fudge factor of 4 to prevent false-positives from highly motivated CS reps.
But only if you monitor for this kind of thing. (And better yet, you should rate limit, and monitor).
Most security problems don't have perfect solutions, but they do have some solutions that get you 80% there. And if you invest in sufficiently many such 80% solutions, you get to a point where it's not economical for an attacker to continue hammering on your application.
It wouldn't have helped in the face of RCE in struts. RCE in struts would let the adversary scp off all the files that make up the database, for instance. This would completely bypass any application-level checks or monitoring.
https://www.computer.org/cms/CYBSI/docs/Top-10-Flaws.pdf (PDF warning) might be an interesting start.
The principles below it aren't too hard though: encrypt data in transit and in rest, think hard about key management. Keep attack surface minimal. Validate at all layers where it makes sense (for example in a multi-tenant application, you should authorize access to a row by tenant both in the application layer, and use row-level security in the DB as second layer of defense). Make your assumptions explicit; write them down. If possible, write automated tests that check if they still hold up.
While the principles are not hard to understand, designing an application with them in mind can be quite a different beast, and keeping stuff secure as features and integrations creep in requires quite some dedication.
At some point, the front end of any web application has to go to the back end database to get information. So there's just no way to cleanly and completely isolate the front facing parts of an application from possible breach.
Maybe segmentation is the only possible defence - customers A-C served out of this DB with a different stack to limit the damage.
But once they get in, they get in, right. Its just a matter of finding the next vulnerability.
As long as you are providing a rich customer experience, with the ability to generate reports based on historical information, you have risk.
I just checked some Equifax domains against SSL Labs, and while their Canadian site (https://www.econsumer.equifax.ca) scores an A-, it has no forward secrecy. I'm surprised to see a modern web server not supporting FS today. Worse, the main entry point to their Canadian site (http://www.consumer.equifax.ca) as indexed by Google does not redirect to a TLS enabled page, although they do seem to have a TLS endpoint for that domain -- but not sure how people are expected to get to it.
Edited to add: The first link is only accessible through a redirect by clicking on the "Get Started" button on their main Canadian site. Furthermore, even selecting Canada from the drop-down on https://www.equifax.com/personal/ redirects to the insecure non-TLS site.
https://www.eff.org/https-everywhere
https://observatory.mozilla.org/analyze.html?host=equifaxsec...
The attack could be way more sophisticated and involve several hacks through the stack.
Injecting a JSON blob to the Struts REST deserializer that effectively results in executing payload code would indicate high investment in the attack.
The hacker(s) possibly had insider information. They possibly used social hacking techniques to acquire data decryption keys.
Given Equifax stores more than 143M records, the initial report would imply some sort of data segmentation limited the scope of damage. If the doors were wide open, why didn't the hackers take 100% of the records?
None of this excuses Equifax or removes liability (I've personally initiated a credit freeze and will likely need a LifeLock Plus account for the rest of my life thanks to this breach).
Why does it indicate high investment? IMO it's similar to SQL injection attack. People, just take a look at points where complicated data can be deserialized and check whether adequate checks can be overcome.
As I see from their report here: https://cwiki.apache.org/confluence/display/WW/S2-052 they use XStream to deserialize REST data. XStream is basically Java serialization/deserialization to XML instead of binary. As far as I remember from my experience of working with XStream, XStream can deserialize almost any object, even the ones which aren't serializable by default. It's quite trivial to go from here to execution of arbitrary code.
Code that does what? Dumps all environment vars? OK, maybe that produces a DB2 connection string. Refactor code, re-inject, does the connection work? OK. Refactor code, re-inject and dump all tables. etc...
This attack requires an iterative trial-and-error probing process, typically done behind a proxy for many weeks/months.
From Equifax's PR: "the unauthorized access occurred from mid-May through July 2017.".
That's a reasonable amount of time to probe all systems involved and develop sophisticated code.
Of course this attack is quite laborious, but it doesn't require anything special or unique.
For most attackers, it is vastly more efficient to run automated sql/wordpress-vuln/ssh-user scanners dropping bitcoin miners and botnet nodes. Those require zero human effort and self-replicate. That gives you less publicity than a hundred million personal finance records, but it has a much higher success chance and a more reliable income.
Now a different thing is that should this information be available through a server that is accessible directly through the Internet (and I don't know if it was) or if they should require for example a dedicated VPN connection.
If you don't have a full team for this, another strategy is to rely on a vendor, and try to obtain as many dependencies as possible from this vendor. Of course, you still need a process for updating the vendor-supplied components as quickly as possible.
Credit rating is an instrument of control used by banks and credit card organizations to force us into debt we don't want.
This system is CRAZY and I don't understand why normally anti-establishment and "leave me alone" Americans so sheepishly agree to be involved in a system that hurts our individual liberties every day.
When I came to this country the "credit score" system was the single most incomprehensible thing about american society. Still is.
When I came to the US from Europe, I had no credit history and that made it difficult to get any type of credit card, or loan, or anything. I thought it was a catch 22 since I couldn't build credit because I didn't have credit, but there is this thing called "Secured Credit Card", where the bank lends you your own money (that you deposit) to see if you can repay yourself. Had to do that for about a year, and I also managed to get a Macy's card with a $100 credit limit (yes, one hundred) that forced me to shop there to "build credit".
After that the climbing of the ladder started with credit limits increasing slowly, etc... up to the point that I was able to get a loan for a house.
There's a bit of strategic planning to building credit worthiness, and that requires you getting into some "managed debt". I think that was the spirit of the comment you are asking about.
A large part of the credit score concept is to flag people for whom paying a trivial bill on time is difficult.
Here you get full credit worthiness based on your income, savings and loans.
The income and (by tax proxy) savings are public, so no loans and some income and moderate savings will give you full credit.
Wait, how is that?
https://www.youtube.com/watch?v=spG-Yj0zEyc
Because it's a private system, and it's at least theoretically voluntary. Nobody forces you to get credit cards or borrow money.
Credit cards are immensely convenient. I hardly ever carry cash anymore. I buy everything from coffee to lunch to groceries to gasoline using credit cards, and pay the balance every month. If I had to make arrangements to always carry cash to cover those sorts of expenses, it would be frustrating.
I've seen a slight increase in debit card rewards, like with Yelp, airline company dining rewards, and BoA rewards, but those are for the most part random and need to be planned spends. My credit cards will always reward me for using them.
The EE eco system while not perfect is basically a best in class for framework for everything we've learned about building business apps over the last 20 years.
Too many knobs and dials can be a big problem, and verifying you have everything set up optimally for production is tough with all that configuration present.
this seems like a tautology. too many knobs is too many.
(Whether it's true or not is another matter, and not a point I'm qualified to argue)
What style is that? The style of composing persistence technology, Authentication and Authorization, dependency injection, messaging, web communication etc to make a business application? If you know of a magical tech that doesn't aim to solve these problems in a standardized modular way so that the developer isn't socket programming from the ground up then please chip in.
"There are many moving pieces and many permutations of how they interact together."
Agreed. The problem space isn't todo apps or "Hotdog/No hotdog".
"It's practically impossible to understand it in its entirety, and thus impossible to guarantee that it's secure."
I don't understand the finer points of the myriad of TLS cypher suites etc, probably means I can't guarantee that they're secure. But what other option do I have? roll my own? Nope I'll take sensible defaults and twiddle the knobs where the client's requirements demand something different.
"You're basically plugging holes as you find them, but you're never sure that there aren't more holes you don't know about." - this is FUD.
I'm not talking about understanding the finer points of the internals such as the implementation of the TLS cypher. I'm talking about the complexity of how different moving pieces fit together in the stack.
This isn't FUD, it's the reality and it causes security breaches all the time in the real world.
For example:
https://www.google.com/search?q=rails+rce&ie=UTF-8&oe=UTF-8&...
There is a massive difference between an over-engineered WebSphere-based deployment using EntityBean EJBs (1.0 no less!) and a tight deployment of Jetty + Struts 2.0 (or more recent frameworks). Anyone who has done real Java development knows this difference. The embedded Jetty + web framework movement was the start of today's modern frameworks, such as Play and various microservices toolkits.
The Servlet spec was probably the only good thing to come from JavaEE and Struts falls into that camp, even if it's fairly dated these days. Painting it with a broad "Java EE" brush, with all the bloat that went with it, isn't right.
Sounds like you didn't even bother reading the article and jumped at the opportunity to criticize something you don't like.
The breach has nothing to do with Java EE and everything to do with Equifax' absurd network configuration.
>The bug specifically affects a popular plugin called REST, which developers use to handle web requests, like data sent to a server from a form a user has filled out. The vulnerability relates to how Struts parses that kind of data and converts it into information that can be interpreted by the Java programming language. When the vulnerability is successfully exploited, malicious code can be hidden inside of such data, and executed when Struts attempts to convert it.
>That means intruders could easily inject malware into web servers, possibly without being detected, and use it to steal or delete sensitive data, or infect computers with ransomware, among other things.
https://qz.com/1073221/the-hackers-who-broke-into-equifax-ex...
This has nothing to do with Java EE.
Where a particular set of companies falls on such a scale is also independent of this, isn't it? I feel like I'm missing something (which admittedly may be because I haven't read the linked PDF), so if you'd be willing to take the time to expound, I'd appreciate it.
I would like to understand the thought process that resulted in hiring a Music major to run an organization in charge of protecting one of the largest troves of PII in the history of mankind.
Most practicing programmers that are called engineers are not trained in Engineering either. Computer Science is not Engineering.
There was no line of reasoning, only observation. You asked the question, I answered. I assumed you work in different place/industry and your question is honest quest to find how things work.
Yet moreover, corporate security is quite uncool to people who do academic anything (including security) or worked as security researchers. Corporate security is more about negotiation, enforcement, standards compliance and other significantly less interesting things to someone who like research or low level details.
Heck, any JITing language requires process images allowed to execute code in dynamic memory segments, such that basic NoExecute hardware features can't be used. Combine this with Java server-side apps being run in a single process/address space, and I hope you can see that, if nothing else, from a security PoV Java is a dead end.