This attitude should stop. They're volunteers. It's open source. Users are required to use such piece of software at their own will and risk.
Log4j didn't ever get a single penny from most (all?) of their users, and now they should take all the blame. Why?
The disclosure process was terrible, they clearly had no time to perform a thorough validation of the hotfixes, and this is the result. Why didn't anybody else step up to help them? Did any of the large companies using log4j pour some thousand dollars into the Apache foundation to let a couple of security-oriented engineers take a complete review of the patches?
But then, it's log4j developers and Apache that should be blamed. Bah. That's SO pretentious.
>But then, it's log4j developers and Apache that should be blamed. Bah. That's SO pretentious.
Um, yeah. They wrote the idiotic code that allowed this to happen. Who should get the blame then, the users? Duh, no. Should the hackers taking advantage of the exploit? Maybe, some, but they wouldn't be able to do it if the code wasn't so badly wrritten.
>Why didn't anybody else step up to help them?
To me, this is the pretentious part. I didn't write the code, I don't use the code, so why in the world should I be expected to fix the code? Do you use the code? Why didn't YOU fix it?
Just because someone releases code with AS IS/NO WARRANTY/BLAH BLAH BLAH doesn't mean they should not get called out for introducing such a vuln. Yes, you chose the code, failed to evaluate it's security, etc, but at the end of they day, the Log4* devs made a very very bad decision. Sadly, because of it's deep use in so many projects, it has kind of become too big to fail. I doubt %1 of users will swap out the logging package for something else. They will all wait for the Log4 devs to keep releasing updates.
People keep singing the praises of FOSS, but yet whenever this kind of shit happens, it's always the user's fault for choosing poorly which packages are being used. I'm saying that the original devs are criminals, but they did the original sin here. The fault ultimately lies at their feet. Why is this such a unrespected point of view?
Would it be any better if this library was a proprietary product, the vulnerability would have been discovered by NSA and exploited until the end of time?
Come to think of it, they wouldn't even have to search for any exploits. Just ask devs nicely to put the backdoor in and be done with it.
What original sin? What fault? You’re basically saying that log4j developers lured you into temptation, otherwise you would have coded your own logging library, which, for sure, would have been better.
I haven’t seen 1/100 of this hatred for Apple iMessage vuln that led to NSO zero-click exploits, and I don’t get why.
> I haven’t seen 1/100 of this hatred for Apple iMessage vuln that led to NSO zero-click exploits, and I don’t get why.
I think there are a few reasons for this, and most of them aren’t based on logic, but emotion.
First, as bad as the NSO zero-click exploits are, they aren’t things that developers were potentially injecting into their own projects. So there is an easier force to “blame” — Apple (for making the mistakes in the first place — and to be clear, all software can have bugs) and NSO Group (for being pieces of shit) — whereas if you either chose Log4j for your own project, or more likely, chose a project/product that chose to use Log4j, the “who to blame” question becomes a lot less comfortable to answer.
At least with the NSO thing, there is a face of who the bad guy is. So it’s easy to sort of not hate Apple, provided Apple patched exploits as soon as they found them. That doesn’t mean that Apple, a company that has used its “better” security relative to its competitors one of its marketing messages, gets off easy. It’s a black mark for them for sure and will make it harder for the company to argue that it is so much more secure than x, y, z service or platform.
With Log4j, similar to OpenSSL, people have to grapple with the reality that they don’t really know a lot about a lot of the code they run — and even worse, acknowledge that they aren’t capable of understanding or auditing that code themselves. I certainly don’t understand a lot of the code and libraries I rely on for things I build. And if I really stop to think about that, that’s scary. Fortunately, my own projects are personal and aren’t taking data from others. But facing that reality can be a difficult pill to swallow. And Log4j is even worse than OpenSSL in a way, because one could be forgiven for having trust in a well-respected cryptographic library, and for not having a deep understanding of how that library works. But having an insecure logging library? That almost feels avoidable — even if it isn’t.
So people want to blame someone. And when it looks like the code — made by volunteers or not — wasn’t the best written, that’s an easy thing to go to. Rather than self-reflecting about all the libraries we use every day that we don’t think about the security or code quality of.
It doesn’t help that the disclosure, for a host of reasons, wasn’t good. That isn’t me blaming the Log4j team, it just is what it is. Disclosure wasn’t great and the fury to patch made some mistakes and now that there is outsized attention on the library, even more vulnerabilities have been found. Which is all understandable and indicative of what often happens in high stress situations.
Ultimately, we should all accept that mistakes happen and that this wasn’t done out of malice. We all have some level of culpability for what we choose to use in our projects. But mistakes happen.
And as you say, discussion is OK but disrespect isn’t. But I think we see lashing out because people want to find someone to blame.
It's actually fairly similar to the Apple bug, in that it was also a library with unnecessarily complex functionality that wasn't needed for the use case but just came along for the ride and ended up exploitable.
Disrespect for calling a spade a spade? Where have we found ourselves?
No, I would never use a logging library, so there was no luring. If you want to call those that did use it lured, then you're already coming at it that the original software devs were coniving.
Sorry, I absolutely meant to include NOT in that sentence. No, I do not believe they are criminals. I am guilty of not double checking my post to make sure it says what I meant to say.
Not approving of what OP said, I’d point out that everyone has an interest in it being fixed, because it’s a wide spread and severe CVE that impacts lots of software that we all probably use daily without knowing it.
> Um, yeah. They wrote the idiotic code that allowed this to happen.
And released it with NO WARRANTY. And no mega corp thought it’d be a good idea to try and buy one.
> To me, this is the pretentious part. I didn't write the code, I don't use the code, so why in the world should I be expected to fix the code? Do you use the code? Why didn't YOU fix it?
If you don’t use it, then noone’s expecting you to fix anything. But those out there using it for free, complaining that it’s broken, and not doing anything to help? They should hush their hush holes. Or open pull requests. But mostly hush their hush holes.
This comment isn't even consistent with itself. First you say:
> Who should get the blame then, the users? Duh, no.
And then you say:
> I don't use the code, so why in the world should I be expected to fix the code? Do you use the code? Why didn't YOU fix it?
Just take a step back, stop trying to throw shade at anyone and everyone, and look at the situation analytically.
I don't really see the point of these comments. The people who make a contribution to the world are the people who see a problem and take practical steps to fix it and improve the situation, rather than the people who see a problem and immediately descend into finger-pointing and shouting.
Because the only unsolved problem in logging libraries is: "When do I stop adding features?"
Decent logging is surprisingly complicated, but it's not "We need an arrrchitecturreee! for infinitely configurable logging or we are utterly doomed" complicated.
> The disclosure process was terrible, they clearly had no time to perform a thorough validation of the hotfixes, and this is the result.
Even with proper disclosure, patching can be harder than it seems at face value. Take shellshock (the bash RCE from 2014) for example, the first patch was released a week after discovery and publication happened after the patch was released. Yet there were still a flurry of further CVEs found and further emergency patches required as a result.
The problem is once a major vulnerability is discovered, you then have a the worldwide attention focused on that specific vulnerability looking for other ways to break it. So it’s not really a surprise that you find multiple CVEs for most major vulnerabilities.
Having worked with security-minded institutions in the past, I've endured and participated in year(s)-long audits of three different open source software projects. The thing that amazes me is that nobody using this software ever did their due diligence. You literally get what you pay for. Kindly step on your own legos and be happy this didn't happen in early April.
> The thing that amazes me is that nobody using this software ever did their due diligence.
How could they be expected to? Reviewing all of the source code for log4j is not a walk in the park. Even if you thoroughly reviewed all the source code, would this specific exploit have crossed your mind?
The game theory and assumptions around a "logging" library probably led a lot of organizations to not even consider it a potential threat and focus their efforts elsewhere.
There is also the transitive dependency angle which turns this into more of a plague than a blame assignment session.
I don't expect every user to audit code. But one of the audits I was privvy to was a part of the federal government and had really stringent requirements for the code that they run -- which, of course, includes tracking down all the dependencies, and, yes, is expensive. The fact that this extremely widely-used project never hit such an audit is a curiosity. And a bit of a tragedy of the commons -- many millions of dollars of commerce has depended on this project, and now folks are up in arms because every individual decided that it wasn't their problem to investigate.
Even with an audit and 100% unit test coverage, and static code analysis, vulnerabilities slip through. Software is not perfect and you do not have a silver bullet.
I'd agree if this was the first CVE. But that got some attention to log4j, which quickly exposed two more CVEs. There's no such thing as perfect, but these problems almost certainly would have been caught by a competent security audit.
Has any of your software been through a security audit? Mine has. It’s 99.9% OWASP coverage with some automated fuzzing with almost no imagination applied by the auditors. You’re assigning far too much ability to auditors.
> How could they be expected to? Reviewing all of the source code for log4j is not a walk in the park. Even if you thoroughly reviewed all the source code, would this specific exploit have crossed your mind?
Back a long time ago, companies were extremely wary of including any open source library (let alone thousands!) into their product. It took a very long time with audits, lawyers, etc to get a single library approved.
Yes, it was kind of a pain.
So the world has swung into the other extreme these days. Just pull in a couple thousand libraries from the internet, nobody knows what they even are or what's in them. All in the name of speed. So it has its benefits.
But the fundamental concerns of long ago are still there. Bringing in unreviewed third party code is a vector for vulnerabilities, supply chain attacks and all that joy.
> Reviewing all of the source code for log4j is not a walk in the park.
Traditionally if something is too large to audit then you shouldn’t use it, which few exceptions (like Linux). Maybe this should be a red flag in the future, especially for a logging library that can almost be replaced with a call to stdout and a log collector.
Thanks to all Log4j developers. As users of your free software since ~20 years, we're sure that, if this vuln disclosure had been better and there were no sign of widespread in-the-wild exploitation, you would have done a better job at patching it, and a single release would have been enough.
We won't hold a grudge against you; open source means collaboration, and you don't blame hard-working people that give away their software for free for honest mistakes. Your software still did far more good than bad to the software world.
Thanks to Log4J developers for turning me off to Java 20yrs ago due to their horrible API design and developer experience. Does not shock me that this security vulnerability happened, but it did shock me that so many people kept using this awful library.
I thus enjoyed a development career (mostly) free from this kind of needless pain and suffering.
log4j2 has little to do with the log4j you claim turned you off 20 years ago and almost no one works directly to the logging library API rather than an slf4j facade.
Java ecosystem is rich and much more developed than anything else. There are plenty of better solutions on the market, both open source and commercial. There exist alternatives to log4j for a while, e.g. logback, which is the default option for Spring Boot applications - there was absolutely no need to use log4j2. It was always a free choice.
It is worth mentioning, that any even very basic solution always has a possibility to shoot in the leg, and only professionalism of the developers of the client code can avoid it, so the real reason why this is happening now on a massive scale is that Java world is big and has probable hundreds of thousands, if not millions of developers, who do not have enough knowledge or experience to write better code, understanding all the possible side effects and other consequences of their decisions.
This was going to happen eventually. It will happen again, but maybe next time it will be a Python or TypeScript library. Would it be right to call their ecosystems toxic? I honestly do not know. But let's at least have some respect for people who have built it, almost for free.
Why can't we point out that the "default" choice for most java shops is one that is massively overcomplicated, a nightmare to use, and subject to security disasters like we are seeing? This wasn't "unknowable". Using overly complicated, poorly designed software is a direct cause.
Why should I show respect for software I hate using and that causes problems for the entire tech community? I appreciate that they built it for free. I write open source code too, and glady accept criticism if I release something that sucks (which I do often)
This saga is a lesson in choosing the right tech, and switching off libraries built for use cases 100x more complex than what 99.9% users need.
I'm not a Java developer, but I don't quite understand your complaint. You haven't used the library in question, you don't have any experience beyond the 'smell of the ecosystem' two decades ago, but you're opining at length on its 'poor design' and proneness to security disasters.
What knowledge do you have, about this library which you've never used, which qualifies you to say that it's especially prone to security vulnerabilities - other than hindsight bias?
Most of the Java software is in fact not affected because it is using different logging solutions. It is not „default“ as you say and in many cases it was still a reasonable choice made by people with a lot of experience, because the library isn’t bad. It has plenty of useful features, and the fact that once in a decade someone found a vulnerability in an obscure part of it does not mean they have to switch now or reinvent the bicycle with own logging.
As for respect… did you do the code review and security audit of every dependency that you use? Do you do it regularly with all updates and patches? What makes you confident, that this is not going to happen with your favorite tech of choice?
> Why should I show respect for software I hate using and that causes problems for the entire tech community?
You aren't expected to. You're expected to show respect for the medium with which you are expressing those thoughts (HN), and the other people you are communicating with on that medium, by keeping the discussion civil and doing your part to keep it from devolving I to a flame-fest.
Almost nobody cares that you have and express dislike for this software. They care that you're doing it in a way which encourages additional discourse which isn't useful and wastes everyone's time.
At extremes it's the difference between staying you think a person is immoral or untruthful with reasons and calling someone an asshole. One of those is easier to have an actual discussion over than the other.
Of course the ecosystem is toxic. Java is literally the only ecosystem in the world where people regularly run into classloader issues because of conflicting logging libraries. And when it happens, that is a smash the fucking computer moment for the momentous stupidity that lead to it.
Yes, it's free. No, the vast majority of users never actively opted into it. No, it's not disrespectful to suggest that a log library that interpolates parameters was never fit for purpose.
I’ve been writing on Java since v1.0, and had the mentioned classloader issue probably once - in mid-2000s with something on JavaEE.
This class of problems was back then called DLL hell, and guess where the „DLL“ comes from? Hint: not Java.
In any case, it is quite rare by now and calling ecosystem „toxic“ because of that?..
Pretty sure you haven’t seen the others long enough.
Slf4j is one of the two examples I point to for really great java design. It's broken down into one jar for the no-op interfaces, and other jars for the various implementations. This lets library owners instrument their libraries for logging without having to make a final decision on the actual logging library and its settings. It's great.
The other example for those that are curious is micrometer. The basic API and the implementations (prometheus, statsd, etc.) are separate jars.
I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an operating system or a runtime. Windows, Linux, Java, and .NET are all commonly updated, but "modules" aren't.
This wasn't just a predictable scenario, it was predicted. Or more accurately, it has occurred already repeatedly in the NPM ecosystem, but for some mysterious reason those incidents were simply ignored by security teams world wide.
Instead of chilling them to the bone, they simply shrugged their shoulders and said "Well, we don't use NPM... I think. Probably?" and went on with their paper-pushing or whatever it is CISOs do these days.
If you think Log4j is bad, wait until Rust gets popular and has something similar happens with a commonly used crate. How would you scan for a vulnerability in compiled code that doesn't even have separate module files?
I had to help an organisation with log4j that had on the order of 5K distinct executables/applications across 3K servers on two clouds and three on-prem networks. At least with log4j it's a simple matter of finding JAR files and scanning their contents. They're just zip files! With languages that output a single binary by default such as Rust and Go, we would have been screwed. No way to scan, no way to self-help update.
Introspection and post-shipping updates for security are mandatory. Rust and Go like to pretend they aren't, because they originate from organisations that are in total, end-to-end control of the software on their own network. Google famously uses a monorepo and can build everything they run from scratch in short order. The shortcuts they can take with their post-release management will never work for ordinary organisations. Never.
So let this be a lesson: Log4j was made for a language that at least allowed us all to find the issue and fix it ourselves. We have Sun's forward-thinking and the enterprise-friendliness of the Java ecosystem to thank for that.
If you still have the source code for your rust code, it contains the lock file. Look in there and you'll see all dependencies and their version. Why do you want to look at the binary instead?
> I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an operating system or a runtime.
If there is the possibility to plan ahead from a sufficiently strong bargaining position, you will not end up in the situation you described. The seller should either continue to support it with a good SLA, or you get the source code with a turn-key build system if the seller discontinues its support. Or you switch out the software if the SLA ends. Should you not be able to agree on such terms, I don't think you would have enough influence to tell the seller to write their software in such a way that dependencies are easy to hotfix/upgrade.
Ok, after putting the unicorns herd to bed, an event happens and congratulations, you now own a license to PeopleSoft v.whatever source code. Oh yeah, you also got the management platform for your network provider too.
It's called code escrow and it's a real thing even if it's news to you. Although yes, it's usually an insurance plan you want to never ever need because using it will suck.
There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation.
If it came from an external organisation, things are exponentially harder.
Modern deployment systems are largely "one-way", with no way to trigger a full recompile from the deployment end of things. If you have a VM with "SomeRandomBinary.exe" running on it that has a vulnerability in it, you don't have many options.
IMHO, typical IT as done in the wild suffers from this write-once attitude, where the chain of provenance is too easily broken.
Companies like Google have a very non-typical setup that can be copied only by very small orgs like startups. Typical enterprises look nothing like a startup or a FAANG.
It shouldn't be hard for a dev to update a dependency which should trigger an auto build and roll your pipeline. Shouldn't be hard for an upstream vendor to patch and have that pulled to kick off a new build.
IDK where this idea that no one can know what's in a dependency tree is coming from but is not ops best practice for more than a decade.
> IDK where this idea that no one can know what's in a dependency tree is coming from
I’d guess people running commercial non source available binaries in production. And outside startups and unicorns, that’s almost everybody.
How do you find the dependancy tree for your on prem Oracle db, or your self hosted Atlassion stuff, or your non cloud ServiceNow or PeopleSoft stuff, or your Huawei network management stuff, or or or…
And what can you do for any of your cloud hosted saas stuff, beyond telling legal “I dunno, their website hasn’t said whether they’re vulnerable to exploit de jour yet”?
My point is that patching third party binaries is generally not something companies take responsibility for. They wait for vendors to patch the binaries and ship them.
Having been on the receiving end of waiting a long time to get a patch for a actively being exploited zero day - the point is, even if you want to fix it yourself (which I desperately did), you can’t.
If it’s some important piece of software you also can’t stop or the business stops - well, you and everyone else is just going to take it in the pants.
Many enterprises are in exactly this bind. And support contracts only get you so far.
People don’t get Oracle because they want to get stuck with a super expensive (and still only ok), or super insanely expensive (and maybe sweet) support contract. They get it because they usually don’t know about these things, are trying to do something and BLAM they get hooked.
Many people don’t even realize how long it can take to get a real issue fixed even with a super sweet support contract. For instance, I once was a wet behind the ears Oracle DBA, and when creating a schema i dumbly used a feature in the documentation without checking to see how long it had been released. Unfortunately, it had been released about 6 months before, as I found out later.
It took 6 months for them to figure out that it was the cause of our sporadic ORA600 errors (aka core dumps), by which point we’d already migrated to MySQL for many of the workloads because the database was having at best 95% reliability because of it.
Once we migrated the table off with the feature, somehow our problems ceased, but that was still well before they told us what was going on.
It is! But a lot of ops departments have problems! OPs point is that a lot of deployments out there make it really hard to remediate this kind of vuln.
Moreover he is arguing that java, as opposed to fat binaries with potentially stripped symbols, made things solvable. He fears that many companies would be much worse off with a big vulnerability in e.g. a rust crate.
Yes, some places suck. They're going to suck either way. If their system is so backwards that they need to manually patch services because everything is EOL, and compiled code will break their workflow, they have plenty of other shit to deal with.
I don't think we should optimize for garbage companies with garbage practices.
I'd say places lie on a spectrum of sucking. With a lot of places being very important and far from perfect. Those places really should get good security.
That means designing for security in the common situation rather than the situation of 'best practices'.
I think we should optimize security for say the 80th percentile company as far as not following best practices (i.e. 80% of companies have better practices than what I suggest targeting).
In no way is manually writing a patch and then live patching the common situation.
The vast majority of cases, and this has been the case for decades, will be that you take responsibility for your own dependencies, and you let your vendors take responsibility for theirs.
> The go command now embeds version control information in binaries including the currently checked-out revision and a flag indicating whether edited or untracked files are present.... Additionally, the go command embeds information about the build including build and tool tags (set with -tags), compiler, assembler, and linker flags (like -gcflags), whether cgo was enabled, and if it was, the values of the cgo environment variables (like CGO_CFLAGS).
At the time that Microsoft started doing it, paranoiacs thought it was some kind of anti-piracy software fingerprinting mechanism, embedding the compiling computer's MAC address or something. But really, it serves exactly the same purpose that Golang's effort does here: to let you map compiled-artifact back to inputs.
So if you make a commit to the README.md fixing a typo, now your build of the source code has a different checksum? I wonder why they don't instead hash all the source files and then embed that hash.
You store the source packages alongside the binary packages, and then index hashes of all the source/binary package contents. Then you know which source files were used from which source/binary packages to produce a particular executable.
> So if you make a commit to the README.md fixing a typo, now your build of the source code has a different checksum?
Yes...? What's the problem?
> why they don't instead hash all the source files and then embed that hash.
This is almost exactly what the git sha is. If you're arguing that README.md files shouldn't be included in the checksum, that's subjective. Many people would argue otherwise. And it doesn't hurt to be more accurate than less.
> The source code is not what is deployed.
There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation.
Do you not have this? Our docker images are tagged with the git hash they were built from, so at any point, for any of our envs, I can pull up the lock file of that build.
Our deployment config also describes everything that is running the relevant env, and nobody has machine access- going through the pipeline is the only way to get access, so there’s nothing deployed that’s a mystery.
Yeah our org does the same thing and I'd say this is the gold standard (or at least mine) but sadly I'm assuming most companies don't do this, not everyone runs on docker nor has a build pipeline like this.
Sure, but everyone should be able to have their built process embed that info into build artifacts, whether it's by passing GCC a DEFINE flag that ends up in --version output, passing nasm a data structure embedded at X bytes into the binary, a version.txt in the output zip, an autogenerated comment inlined in each HTML/JS/CSS file, whatever... somehow, you can embed build info.
This happened last week, for real with ${dayjob} customer. Put yourself in their shoes for a second.
Imagine you are Mr SecOps guy, and you've just ran some sort of Log4j tool across literally three thousand servers. Of those, several hundred came back positive.
Those included about a dozen flavours of Linux, a smattering of manually built(!) containers, and every version of Windows from 2008 R1 to 2022. Most of the code was built by third parties, some under support contract, some not. Most was built and installed manually, with developers using RDP or SSH to edit config files an whatnot directly on servers.[1]
So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent.
Now what?
No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago.
A random hash string is totally useless to you. Even a repo URL and the commit hash will more than likely just end with an "Access Denied" URL, assuming you even have a network route to the Super Secure SCM Server.
There is no way you can figure out who needs to do what to make this go away. Not at this kind of scale at any rate. Not without first-class automation for literally everything. Which you can't have, because third-party software just doesn't play nice with any one tooling you'd like to use.
Containerisation? Bahaha... haha... snort. You're dealing with vendors that literally advertise "now with 64-bit support" and are unable to comprehend the concept of unattended command line installers. Vendors that insist on USB dongles for licensing. License that expire. Annually. And are tied to CPUID values. And on, and on.
[1] Oh, you think you can dictate release methodologies to these people? They're bureaucrats and they play politics better than you. Any word of changing their workflow in any way will immediately bring their boss, their bosses' boss, and maybe a few more levels up down upon your lowly head. You will have people literally screaming at you that your fanciful notions of build pipelines is "too much" and would "impact the work". That's the end of the conversion. I said THE END, and good day sir.
[2] Just get them to update it under their support contract? Ha-ha. Ha. Haaaa... We had vendors straight up lie about the vulnerability of their software. Then another vendor said that updating the JVM already mitigates the issues and hence they're not going to release an update. (Narrator: JVM updates aren't sufficient.)
So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent.
Now what?
The remediation (until you can get an update from the vendor) is to remove the JndiLookup.class file from that jar. It's been fairly well publicised, as has the way to do it.
That provides incomplete protection even for this Log4j vulnerability, and similarly trivial "self help patching" may not be available for future vulnerabilities.
This generalizes to "apply a vulnerability-specific tricky auto-fix to a binary it was never tested on". That might be a reasonable solution if there's nothing better (and since speed is of the essence, there might not be). But it's not exactly a reliable solution, and not every vulnerability will have hacks like that available, nor is it really clear to what extent they fix the problem; it's pretty easy to miss something using this kind of approach, or to have some future deploy bot restore the vulnerability in normal operation. And hacks like this can break things, both directly (the patch breaks actually used functionality unexpectedly), and indirectly (the fact that the software was patched at all breaks sanity checks or other cleverness like binary diffing or whatever).
I can't see how this is going to work as a strategy. Ad hoc cleverness? Sure, and that's always good to have when you're smart and lucky enough to pull it off. But it'd be nice to have a strategy that works without relying on being smart, willing and able to engage with tricky ad hoc solutions, and somewhat lucky.
> No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago.
Your company has an inventory of hosts to teams which you can look up and contact. In addition the package manager tracks which package the file belongs to. Therefore you raise a ticket with the team from the inventory tracker that package XYZ on host ABC contains vulnerable file /x/y/z/log4j-xyz.jar (The reverse is also true, for each package we can find the corresponding repo and set of hosts its deployed on). Fixing it is now that team's problem, and secs ops guy's problem is now just to verify when they claim to have fixed it.
If the software is JIRA? Doesn't matter, still needs a package built before it gets deployed on our machine. "No really, the vendor's only supported installation method is sudo curl | bash" - into a container it goes, the owning team of the container can be identified much the same way as the owning team of the host.
Now the problem you find is sometimes the result is that the host belongs to team XYZ and you find the team was reorged by some exec's great idea in 2019 and only one guy who used to be on the team is still in the company, but he left the team in 2018 and has no idea what that team did since he left, but the application has been still runnning and power $millions of real revenue. That one is much harder to fix by automation. The business reality is the security team can't enforce to the execs to not lay anyone off or disband any teams without a concrete transition plan for their systems.
With languages that output a single binary by default such as Rust and Go, we would have been screwed. No way to scan, no way to self-help update.
ELF binaries are just collections of another sort. You can scan for symbol names or assembled instructions common to a dependency, and you might be able to update with an LD_PRELOAD library that patches the symbol table.
Just look at what game modders have accomplished without access to source code.
This isn't how the real world works. Symbol patching is only effective if there's a symbol for it, which is often not the case (and probably wouldn't be the case here) if the binary has been stripped[0].
> Just look at what game modders have accomplished without access to source code.
Game modding is usually done on Windows given the target market (up until recently of course) and thus usually means Windows PE's, which do not have debug symbols attached - ever. Windows debug symbol databases are emitted as PDB files and are almost always omitted from game releases.
This means modding has to do a sigscan[1] in most cases in order to find something interesting - especially in the case of ASLR[2]. Then whatever modding framework (or hack) can set itself up and hook into the game.
These techniques are NOT what we should be encouraging, and are certainly not common, even for vulnerability mitigation. Re-compilation and re-deployment should be the defacto mode of operation in production, especially for systems that have sensitive information. Relying on bin-patching is not something many security specialists would regard as a "good" mitigation.
Library .so files are not harder to scan than jar files. They just need different tools, which everyone has. And Java systems are not easier to patch than others.
> Library .so files are not harder to scan than jar files.
Of course they are. You can trivially reverse engineer Java bytecode back to Java, with almost everything except generic annotations due to erasure, and a few other small things. Tools have been around for ages that do this[0], and while they're not perfect, they can re-generate nearly perfectly compilable source code.
.so files are ELF files, which means while they have sections, those sections don't typically reveal much about the internals of the code. This means you need to convert machine code (assembly) back to a higher level language - which is not a trivial transformation. There are tools that do do this (e.g. hexrays[1]) but they are far from perfect, especially for optimized machine code.
Java systems are very, very easy to crack, RE and patch. There are virtually no protections. JAR injection is still a valid means of patching a Java application. No, manifest files and integrity checks are not a security measure but an error detection measure, and are trivially bypassed.
If you asked me to choose between patching this vulnerability in a .JAR vs a .SO, I would choose the .JAR any day of the week.
Please re-read my original comment. If the binary is stripped, no internal symbols would have been visible or even present in the file to be scanned. The jumps between subroutines happen directly, without a symbol jump table (because a jump table adds runtime overhead and is largely unnecessary, especially in statically linked executables).
I addressed the point about symbol tables, and that's assuming the vulnerable code was pulled in via an external dependency or exported directly (flat function/subroutine) from an .SO.
That also assumes that the binary wasn't linked statically - in which case, there are no external symbols to even look for.
Lastly, please stop calling someone educating you "BS".
We're all on the same team here, there's no need for this hostility back and forth.
Binary patching, JAR patching, disassembly, library interception, syscall interception, whatever, these are all tools that are on the table. You do what it takes to get the job done, knowing and weighing the pros and cons.
> This means you need to convert machine code (assembly) back to a higher level language
...why? You don't need code to be in an HLL representation in order to patch it. Where by "patching" I mean "automatic insertion of a JMP into the subroutine prologue, that branches to your replacement code."
You don't even need to write any assembly in order to do that, either; you can write your replacement subroutine in C or whatever. The replacement subroutine just needs to codegen as a C-ABI subroutine with the same register preconditions/postconditions as the subroutine it's replacing.
You don't need to, you're right. But people aren't going to be writing very complex mitigations by reverse engineering the bytecode generated by a compiler, especially for OSS. There are many compilers, many optimizations, and some of them can even produce non-deterministic outputs (not common but it can happen). The amount of patching for an internal subroutine, which is probably what would have resulted here, would have been completely practically infeasible.
Besides this, who actually scans the deployed binaries? Just check which version of the binary was deployed, backtrack to the source, and get the dependency tree from there. Problem solved. (Except for shaded dependencies…, but scanning for that should happen at build time anyway.)
Good luck doing that on the custom software some vendor provided you 5 years ago and since has gone out of business.. and that’s a GOOD scenario for a lot of enterprise software. Sometimes it’s 20 years.
We need some sort of escrow for source code of closed-source product. Whenever there is practical need to have access to source code, it should be available.
Okay. Just like any other software that should be firewalled in to only connect to what it should connect to. Moreover, I think meta sploit already has this vulnerability built in so it should be trivial to test/fuzz the application.
I think you could take this further. There's another "log4j" vulnerability out there right now, I guarantee it. It might be exploited, it might not be. But there's no patch.
What are you doing about it?
With log4j you have a brutal combination of:
1. RCE
2. Exposure
RCE happens frequently but often attackers don't have an easy time getting to the exploitable code. With log4j it's trivial - every app can be owned.
But here are some questions:
1. Why do those apps have the ability to make network requests?
2. For the apps that need to make those requests internally, why aren't those over mTLS?
3. For the apps that need to make them externally, why isn't that going through egress proxying?
4. Why are there credentials spewed all over your environment variables across your services? Are they short lived?
It's actually not super hard to have a worst-case scenario vulnerability like log4j be not that bad for your organization. A bit of hardening and even if something like this happens you're in a good position to wait, monitor, and patch.
it's definitely time for a language with a capabilities based security model that allows scope to the function/method level, at minimum
not just for security either. i can't count the number of times i've traced a performance problem or bug to a piece of code that was doing some io when it didn't need to
Regarding your questions, they make sense in the context of modern software built in the last 5 years. However, most large enterprises have a long tail of legacy applications built years ago. These may be inherited from a combination of acquisitions, from vendors that have since gone under, some from. The common factor is that the original developers are long gone, and they may be supported using a skeleton crew or not at all. Documentation if it ever existed is likely lost. What these applications are supposed to connect to are likely unknown. This is a major problem at enterprises
Exactly. Managers and execs love pointing out the cost savings they made this year or this quarter. By the time the chicks start to hatch "in the long run", they most likely would have already moved to another enterprise so someone else has to clean up their mess. And by clean up, I mean repeat exactly the same mistakes since nobody promotes you for implementing stuff that might show its benefits in the long run.
Also one of the root problems with politics. 2035 emissions targets are the next government's problem. Rehabilitative policing pays off in 20 years, retributive buys voters now.
Egress is a very fair question IMO. I've worked at more than one place where Infosec would start integration talks at 'What are the static IPs and how do we rotate certs' Every layer you go lower (longer certs, domain whitelisting vs static IPs, etc.) is a tradeoff of maintenance and security.
> What these applications are supposed to connect to are likely unknown. This is a major problem at enterprises
Only if they fail to consolidate vendor integrations and/or have already insufficient staff. (Which is often indeed a problem.)
Simple, if brutal solution; Infosec audits logs and works with application teams to confirm existing traffic and pre-document new traffic out. With properly staffed teams you should run out of surprises within a year.
I suspect that a manager that suggested / insisted on such properly staffed teams, would get fired or no longer promoted, at most big companies. Since it'd cost a lot, and not show any results the nearest time / years. And also, they might hire the wrong people, and just burn money for nothing
> Simple, if brutal solution ... properly staffed teams
The recruitment part is not simple, maybe bordering to impossible? for some larger companies. I say, based on my past experiences in how clueless big companies can be concerning what technical people they hire.
> and not show any results the nearest time / years.
Depends on how you game OKRs. Implement auditing Q1. Contact all teams Q2. Have 75% of teams Q3. 90% of teams Q4. Just an example but its all in how you sell it. Big IT understands icebergs, you just have to give a good/accurate roadmap and execute if they agree on it.
You might be correct that a lack of insight on this sort of thing is an organizational problem. My experience indicates that if you can't get a list of dependencies based on a series of e-mails down an org chart, something is wrong. Either someone is managing too many services for the level of tooling you have, or there's a complete breakdown in org comms (which happens more in larger enterprises.)
> We have Sun's forward-thinking and the enterprise-friendliness of the Java ecosystem to thank for that.
I find it quite ironic to praise the forward-thinking of the company that has been instrumental to bring us into this mess via their vision of loading dependencies at runtime from an online repository. Sun imagined that the code and its configuration does not have to worry about how to fulfill its dependencies, but instead JNDI [1] could magically fetch the appropriate objects from wherever, don't worry.
I'd say I would even find it ironic to praise the Java ecosystem, which, due to its excessive over-engineering, has produced best practice frameworks where fully documented behavior has the latent potential to just be catastrophically exploitable by accident, as nobody is able to reasonably understand how all of it plays together. I have a hard time to imagine how you could enable loading of external code in Rust or Go by accident using run-of-the-mill logging frameworks.
I'm writing this from memory, with checking any sources. IIRC:
Designs like JNDI were widespread in the IT software industry at the time.
The academic prototype for all this was CORBA, the Common Object Request Broker Architecture. An object-oriented was to transmit data between systems running on separate processes, CPUs, remote servers.
CORBA was a next-gen RPC (Remote Procedure Call) framework, the data format on the wire closely follows the format for C structs using Sun-RPC, but was way better because these CORBA things were discoverable and described at run-time, rather than compile-time libraries.
That's right: RPC was initially a Sun UNIX thing.
Microsoft needed a way to do this, so they had COM, the Component Object Model, for run-time discovery and linking (late binding).. Microsoft got COM to reach across network connections, "DCOM", for Distributed systems.
Meanwhile, corporate IT was still pulling text out of databases and shoving it into C structs. It sucked.
Microsoft's object model was better, and Java was better than better, because you didn't have to license it to use it. Well, you did; it was like an MIT License.
Reason why IBM writes huge Java systems. And why Oracle owns Java. For some value of ownership.
Sure. But you have a slight problem with ABI stability, at least in Rust, if you ever want to compile an ABI compatible replacement library. Until that problem is resolved, if you want to make a useful replaceable object you need to make a C style foreign function interface and use that, something that isn't exactly convenient for Rust to Rust use, even if rather helpful for making Rust libraries usable from other languages. C++ has a very similar issue, and has had it for years, albeit one mostly resolved by the advent of stable C++ ABI standards.
It’s funny because when I worked in .gov IT, we were all characterized as stupid donkeys by because we weren’t able to just use NPM, etc to do cool kid stuff.
Good luck getting security people with the power and brass balls to do that in any company.
When HeartBleed came out, there was a single dynamically linked library to be replaced. If you had a vulnerable version of libopenssl, then you replace the .so or .dll with a patched version, restart any applications that were using the old version, and you're set.
This works because the binary is compiled against the dynamic library, using the system-provided library. It isn't statically compiled, because even though that frees the developers from worrying about library versions, it makes the sysadmin's job tracking library versions be much harder. It isn't vendored, for the same reason.
Depend on your dependencies instead of embedding them. Specify version ranges instead of pinning dependencies. Report issues in your dependencies and contribute back fixes for them.
... hear back from upstream that your fix, developed against version X (that you pinned and use), does not apply to HEAD. Pull HEAD and realize there have been major changes which conflict with your platform, use case, or any number of your other dependencies.
Sometimes you can get away with just depending, and that’s nice, but other times there are strong reasons to embed. At that point, you own the code, and can actually do the things you need to do. (Of course, it follows that you own the code - and need to then actually do the things you need to do.)
Both approaches have their advantages, and both have potential for technical debt. Which approach offers the most advantage for the least debt will depend on the individual circumstance; I don’t feel that you can really advocate for one over the other in a general sense.
> If you think Log4j is bad, wait until Rust gets popular and has something similar happens with a commonly used crate. How would you scan for a vulnerability in compiled code that doesn't even have separate module files?
Our software scans take 3 forms:
1. Searching for file names
2. Running test exploits that have a payload that just logs the vulnerability with a central monitoring system
3. Dependency resolution and checks against a DB (our solution _already_ supports Cargo.lock files.
Rust (and Go, C++, etc.) only prevent option 1. Thanks to bundlers it also can miss it in JS, or shaded fatjars it can also fail kn Java too. So it's not a new challenge
> I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an operating system or a runtime. This wasn't just a predictable scenario, it was predicted. Or more accurately, it has occurred already repeatedly in the NPM ecosystem, but for some mysterious reason those incidents were simply ignored by security teams world wide. Instead of chilling them to the bone, they simply shrugged their shoulders and said "Well, we don't use NPM... I think. Probably?" and went on with their paper-pushing or whatever it is CISOs do these days.
I think this attitude is one of the main issues for this kind of incidents in larger organisations. I'm not sure why you would be willing to lay blame at your colleagues for this.
Most of the time, development teams don't tend to like some governance over the code and the dependencies they are pulling in. Not just because they know of course what they are doing, but also because they are under pressure to deliver features. That is what matters for business.
Also, convincing management that budget is needed for correct tooling to track all stuff deployed; is also not as straight forward as you seem to suggest.
In this case, the problem goes even beyond just the code of your own dev teams. This is embedded in countless software packages deployed all over your organisation. Same here, people want to buy and use whatever they want. And all processes to keep some form of control over it, are mostly seen as overhead.
And I'm sure they are lots of "security" people who are just producing documents and policies which are complete detached from reality. But developers who consider security completely as somebody else his responsibility, are a problem as well.
The only good think I see coming from this mess, is that security teams probably will get the means to try to get more control and insight over this. For the coming weeks/months at least. After that, everyone in management of dev will be forgotten about it. But something tells me the security people who are working on this right now, won't.
They found a way to make several catastrophic mistakes while writing a logging solution (one of the most trivial kinds of software) in a memory-safe language.
It's surprising they manage to breathe properly and thus survive.
If information security people would spend the same amount of time they are spending complaining about this vulnerability trying to educate their organisations on the importance of supporting open source solutions they depend on we wouldn’t be in this situation.
I am sick of reading of very senior people complaining about the impact this vulnerability has had on their week when their companies don’t even contribute a penny to open source projects.
> educate their organisations on the importance of supporting open source solutions
Githubs of the world could just gate downloads, pull requests etc. behind a payment to see what is the real valuation of open source software; I imagine it'd mostly settle around $0 excluding couple of big projects.
Up until the semicolon, this was actually a decent point. I can see a future for high-quality open-source software hosted on an 'app store' which requires a small payment for use - or for commercial use, perhaps.
That's the question that such an effort would have to solve. I think you're assuming a model where software is directly used by an individual consumer, though, which is not always the case. For commercial applications where your software is integrated into the end product, for instance, companies would never risk stealing unlicensed code considering the risk. How one would address the question you posed is entirely dependent on the domain one's operating in.
I'm baffled by these claims that we should compensate the authors and maintainers of what has been conclusively shown to be software too dangerous to exist. It has the same problem as calls to pay the maintainers of openssl. No amount of money is going to impart good taste and best practices onto these projects. The uncomfortable truth is these libraries need scratch rewrites with better authors.
Saying things like "I don't like this software because <concrete reason>" is reasonable; it can be considered constructive criticism, especially if you include a way it could be done better. Heck, even just "I think it's too complicated to use, and the underlying code base needed to support that complexity lends itself to bugs that can be exploited" would be better.
Saying things like "No amount of money is going to impart good taste and best practices onto these projects" is insulting the people who wrote it with no real basis. It's not helpful and it makes you (and some other people in the thread) sound like a jerk.
One of the maintainers in question holds two jobs to live. Having money might allow him more time to test and retest and update the software more expediently.
Sorry, I was posing that question semi-sarcastically. I agree it's blindingly obvious to any sensible person that money would help immensely. However, sadly, there are apparently plenty of people who are just about sensible enough to be able to get out of bed and navigate to this site, but not sensible enough to understand much else.
That wouldn’t be funding log4j2 though (from the ‘better authors’ part). That would most likely be an internal effort (devs you trust) published as a new open source logging framework in a sea of existing ones.
Honestly, absolutely not, and I think money would be harmful. Lots of software doesn't need a lot of work. It can be simple. When people get paid to solve problems there's an incentive to add more code, to add new domains, etc.
Use the standard java logger and move on, it's that easy.
Where’s my incentive to work the weekend supporting a project after a long week of supporting my production code? If we want solid and secure OSS, we gotta pay someone!
Presumably it's the same incentive that pushed you to write it the first time.
If you don't want to work on it anymore that's OK too. Open source libraries are generally not held to particularly stringent requirements by the vast majority of people - perhaps accepting an obvious patch in a critical situation, at most.
You obviously can pay people. They're called employees. The thing about employees is that your money translates to business value, which is much broader than a single library. This means that when a single library is done the developer doesn't have to think "ok but what's next for this library" and start adding more code. They can stop and move on to other tasks.
Paying for a single library is nice. I support multiple open source developers because I appreciate their work. I recommend everyone does so.
I don't at all believe it's going to help security though.
The unfortunate thing about scratch rewrites is that they are not a good business. Open Source has conditioned people to think that libraries should be free. It's not clear to me that the alternative where libraries are obscenely expensive (like in the hardware world, where a hash table is a $100k investment) is any better. Nobody has ever built a successful company charging between $0 and $10,000 for a library. That is just how people see things, and unfortunately it is extremely hard to change.
Some platforms have reasonable markets for libraries; Advanced Messaging Systems[0], DevExpress[1] and Aspose[2] are three examples of companies that successfully sell software libraries for .NET, Java, and Office.
Maybe we should tell them to engage with open source, not "support it". Companies should look for a way to spend work hours on open source, not finding someone to send money.
Don't worry, legal will answer a completely different question that's unrelated, but rests on a fundamental misunderstanding of the entire premise due to a combination of misinterpreting technical jargon as plain English and then misapplying a legal generalization, resulting in absolutely essential advice that needs to be followed immediately, which is couched in legalese that the techs will proceed to misunderstand by misinterpreting legal jargon as plain English in combination with a misapplied technical generalization, resulting in an absolute quagmire when it turns out that these essential compliance steps turn out to have a minor impact on the next quarter's bottom line. Thus some even more obtuse interpretation will be dreamed up by some manager to claim CYA without hurting the bottom line (or addressing the reinterpretation of the reinterpretation of the reinterpretation of the original issue). The issue will be resolved, pending small details like the actual legal risk and technical issues, and all will be fine.
If it's an enterprise (https://zwischenzugs.com/2018/10/02/why-are-enterprises-so-s...) the latter will be a much easier sell to management. Paying for support is a given but contributing developer time will require many approvals from different departments if it can be done at all.
I read a post recently that said part of the open source funding issue comes down to a financial "structure" mismatch. Open source folks are often individuals or loose collections of individuals. Sometimes they have things like Patreons, ask to receive crypto currencies, etc. Companies don't/can't provide funding via these types of structures, so the writer of the post ended up incorporating a company, setting up a mercantile bank account, commercial payment processing, etc., so companies would be comfortable funding his efforts. Fortunately, this path is lot easier lately with services like Stripe, etc.
That feels backwards. Little shops are flexible and can probably do what needs to be done. Lumbering behemoths have (too many) rules and restrict themselves (often) unnecessarily.
Just because a dev lives in the Midwest doesn’t mean they’re idiots compared to coastal elites. The hate/mockery/minimizing of anyone not from a major coastal city is really gross. Screw you.
I am a developer in the midwest, and maybe if you hadn't completely missed my poin you wouldn't be such a jerk.
Nebraska is a reference to an XKCD Comic (https://xkcd.com/2347/). The idea is that there are nameless people all over the world contributing to critical open source infrastructure.
Back to the point.
Running Google Maps on one of my low traffic sites, would generate at most, several dollars a month in charges. It would be a rounding error in our budget, and we can't pay it. Our purchasing rules are not flexible enough to work with google.
Paying Google is hard enough, an independent oss developer with no company would be impossible. I would get laughed at if I suggested throwing a few bucks at a developers Patreon.
They must adapt. Period. And don't worry, when they have to, they do find solutions. The thing is, with open source, you don't have to, worse, the vast majority of enterprises rely because you don't need to make any effort.
> If information security people would spend the same amount of time they are spending complaining about this vulnerability trying to educate their organisations on the importance of supporting open source solutions they depend on we wouldn’t be in this situation.
Security teams are stuck with securing the tire fire; they didn't choose the library or platform. If anyone should be advocating for supporting open source, it's the developers who benefit by using open source libraries.
Just leads to the xkcd “there are now 14 competing standards.”
Honestly the biggest problem of package managers is finding a reputable package that one can trust.
The best one can do to aid this situation is reviewing, vetting and warning of which packages that can be trusted and not. Automatic scanners to find code smells and vulnerabilities.
“Security rating: 2/10, This package seems to use JNDI loading, are you sure you want to continue (y/n).”
Humor me: if log4j had 13 (independently developed) competitors, wouldn't the impact of this be much lower, because many consumers would use different libraries? Like genetic diversity as a hedge against disease.
I bet they'd all compete on features and probably implement the same exploit trying to maintain market share.
Logging as it's understood today should ship with most STD libraries. Things like logback an log4j really should just eventually be rolled into the std.
Not in and of itself.. However if part of that funding was used to cover formal (external) audits, investment in fuzzing/automated security testing etc.. then it might well have done
"Might have" is the operative word. In a world where Project Zero's _This shouldn't have happened_ [1] was also published this month (buffer overflow on NSS), I'm less certain. Quoted from the article:
----
This wasn’t a process failure, the vendor did everything right. Mozilla has a mature, world-class security team. They pioneered bug bounties, invest in memory safety, fuzzing and test coverage.
NSS was one of the very first projects included with oss-fuzz, it was officially supported since at least October 2014. Mozilla also fuzz NSS themselves with libFuzzer, and have contributed their own mutator collection and distilled coverage corpus. There is an extensive testsuite, and nightly ASAN builds.
I'm generally skeptical of static analysis, but this seems like a simple missing bounds check that should be easy to find. Coverity has been monitoring NSS since at least December 2008, and also appears to have failed to discover this.
Until 2015, Google Chrome used NSS, and maintained their own testsuite and fuzzing infrastructure independent of Mozilla. Today, Chrome platforms use BoringSSL, but the NSS port is still maintained.
Did Mozilla have good test coverage for the vulnerable areas? YES.
Did Mozilla/chrome/oss-fuzz have relevant inputs in their fuzz corpus? YES.
Is there a mutator capable of extending ASN1_ITEMs? YES.
Is this an intra-object overflow, or other form of corruption that ASAN would have difficulty detecting? NO, it's a textbook buffer overflow that ASAN can easily detect.
----
There's no way to argue that less than zero funding would have prevented the log4j maintainers from merging a patch that was sent to them by a third party.
Never understood this.
Companies that make billions base their operations and products on free open source code. Free as in free beer. Then they complain about them not being...perfect.
OTO, people write software for free and licence it in a way that allows above companies to use it without giving them a single penny. And then they complain about not getting a penny whilst somebody else makes billions.
This has absolutely nothing to do with the GDPR. In fact, the GDPR and various data retention laws are at odds with each other which is why the GDPR has a very explicit exemption for this.
Not sure what you are answering. If you for instance expose an endpoint by accident and then can't show logs for the access to it, you are going to have a problem in one of those GDPR hearings. The irony of GDPR is that you may end up with more logging of user data, not less.
Explain 'one of those GDPR hearings'. I'm pretty familiar with the GDPR and everything that comes with it, I've never even heard of the term 'GDPR hearing'.
Maybe it is not handled the same way depending on where you operate from but where I live you are supposed to report any potential data leak of GDPR sensitive data to the local authorities. I do not know what the criteria is that makes you end up being questioned about it though. I been at such meetings over silly things.
The reporting requirements are the same all over Europe, it's a 'better safe than sorry' mechanism put in place after companies routinely wiped their data breaches under the rug in the hope that nobody would notice.
Once you get to that stage there can be a follow up to gather more information, and that follow up then usually results in some advice or no further action. In rare cases - typically the ones where gross negligence or willful transgression of the rules was established - there will be a fine and if it is a repeat occurrence that fine can be quite substantial.
Also you don't have to report 'potential leaks', only actual leaks.
So I think that in the case of your hypothetical end-point the logging isn't a GDPR requirement, but if your endpoint ends up leaking data then the log can help you to establish if and if so how much data was exfiltrated. But that does not mean that the GDPR requires you to have logs, though, having access logs for your endpoints is a fairly standard thing and not having them is going to raise a few eyebrows, especially if you are also reporting a breach.
Where there are logging requirements: data retention laws, SOX, fintech, tax regulation, AML.
At companies I worked with repute is key. Companies with employee count in the range of 10 to 50k. The legal departments will report any potential breach. As a developer you can decide what is logged but not much about anything else in the business. You as a developer or tech lead will most likely be held accountable.
There are countless ways that this vulnerability could be an issue irrespective of whether user-provided data are being logged. Attackers achieving RCE on a machine in your infrastructure is likely extremely bad for most realistic architectures.
I think the main takeaway for all of these is that tricky and possibly unexpected behavior should always be explicitly enabled. And it's not the first time that we are learning that particular lesson so there really are no excuses.
It is pretty valuable to my debugging and general observability to log the URLs requested (including query component), which is user-provided data. I'm not saying I couldn't maintain my app operations without logging URI's including query strings... but it would definitely be more challenging.
Then there's the user-agent header, which also can be pretty valuable to debugging and other operations. Although this one is easier not to log.
In my current logs, I'm seeing attempts to compromise log4j vulnerability in both URL query string and user-agent. (Which of course I would not know about if I weren't logging them). This particular application has no Java involved, so is not vulnerable.
It's not difficult to take user-entered data and just dump it to a log file. I'm pretty positive none of these vulnerabilities exist with System.out.println.
The fundamental problem is that, pretty surprisingly, log4j was running the same parsing and logic on user-entered strings as they did format strings. That's essentially the root cause of all this - log4j shouldn't be attempting to parse this data at all.
So many severe vulnerabilities are due to the complexities of parsing potentially malicious user input - just look at all the severe iMessage bugs. While many other apps do have a fundamental requirement to parse this input, there is no reason a logging library needs to do it.
> It's not difficult to take user-entered data and just dump it to a log file. I'm pretty positive none of these vulnerabilities exist with System.out.println.
yes, but there can be (and have been) vulnerabilities in terminal emulators, terminal multiplexers, text editors, databases, web renderers and all manner of tools that can be used to store the contents of or view that file. your argument makes sense that the logging library should not be attempting to parse the log strings, but think about all the other millions of lines of code that could potentially try to parse that data as well. patch log4j one day and the next you find a bug in some colorizer javascript library or in the sixel support for the terminal emulator. people add parsing everywhere.
i think ideally that all user supplied data should be sanitized in the strictest possible manner before anything is done with it, including logging.
sure. i don't think i'm articulating my real point well enough.
web browsers and all of their components get audited. because they process potentially malicious data as their core function, they see A LOT of attention in terms of hardening.
the point that i'm trying to make is: the amount of software that doesn't immediately and apparently touch potentially malicious data is absolutely enormous and the number of paths that data can take to touch that software is even more enormous. yes, one can put forth good principles for "building codes" for software and yes, one should... but there's an awful lot of software out there that is not up to code (where the craft is so young that the idea of a code isn't even close to being finalized) and it won't be for a very long time.
that being the actual on the ground situation, what is one simple thing that individual developers can do to help prevent catastrophes? always sanitize inputs. moreover, it's much easier to find and audit every place where data is accepted from untrusted sources than it is to find every place where data may be parsed.
sort of raises an interesting question. rust has memory safety, but it's still possible to write code that "isn't up to code." i wonder if maybe rust+1 will be the first language/environment/ecosystem that has application development security principles built in. that would mean, like, not just taking the compiler forward in terms of memory safety, but also codifying a set of principles, enforcing those that can be statically checked in the compiler with the compiler, and aggressive curation for a singular library archive (similar to how some internet library archives require unit tests or style, etc).
> user supplied data should be sanitized in the strictest possible manner before anything is done with it, including logging
Oh come on, how would you even do that if you don’t know how the data is going to be interpreted by parsers that nobody knows they’re using? WAFs have been trying to do this since forever, but it’s always been security theater that separates gullible companies from their money and randomly blocks users that use “special” characters like slash or quotation marks.
I guess you could base64 encode everything, but that wouldn’t save you from the terminal bugs you mentioned. At some point you’re gonna have to undo the “sanitizing” and look at the text.
User data shouldn't be processed at all. You never know where there will be a vulnerability. So you turn off logging so you have no idea what pages your customers are now requesting (phew!) but you can still be vulnerable to SQL injection attacks, buffer overflow attacks, etc. when you are processing the user data. Best to just reject it all.
Here's what I don't get: What is so complicated about logging that you can't implement the necessary functionality in your own codebase, and instead have to adopt a third-party library?
The latter option represents a liability, which in this case (as with others) has shown can be a tremendous risk. Is the time saving really worth it, at the cost of risking disasters like this?
I get that the tradeoff it is worth it for complicated things (e.g. crypto libraries). But logging, really?
Software development culture today is too quick to adopt a huge tree of dependencies of unknown quality, rather than thinking about how to minimize dependencies to only those truly necessary. The leftpad fiasco was but an extreme example of this, but I see it all the time, and it seems probable that there are hundreds (maybe even thousands) of similarly severe problems out there in widely used dependencies that we just don't know about yet.
Whenever I write C, the goal is to have no or very few dependencies. When I see projects written in C, one of the features is that it has no dependencies. This is a good thing. I do not know why we are steering away from this.
It's pretty easy to understand really; it allows engineers to focus on other things and do more. It's not nefarious or because people are always lazy, but the choice between spending time on a business feature or a logging library when one exists that is widely used is hard to sell.
Problem comes when you have a lot of various parts that want to log.
Lets say you have a message processing system getting messages from http and throwing that in a db and keeping counts on an external API. Now every module can use a log library and you, final user of the system can decide where the db log goes, where the http client debug message go, how much detail you want on any of these etc. So in the world of small applications and services, you can come up with a simple logging function. On large projects it is much more complicated.
because Log4J is also used by libraries. Which means you can configure the log level and capture logs from libraries and dependencies you use in a global configuration.
You really shouldn't use log4j in libraries. Instead use slf4j in a library and log4j (or whatever implementation for the facade you prefer) in the application using the library.
Main thing is being able to share the logging stuff between libraries so something like slf4j and then various actual implementations of that facade (which log4j is one of) in jvm land.
There is also lots of stuff you might want a logging library to do that while not very hard would be annoying to do yourself for every project like log file rotation, switching log level while the application is running, optimizations (like in c/c++/rust you might compile for INFO level and drop all the DEBUG and TRACE level logging from the binary), different log levels for different parts of the app (including dependencies), etc
> Here's what I don't get: What is so complicated about logging that you can't implement the necessary functionality in your own codebase, and instead have to adopt a third-party library?
Transitive dependencies for one thing. I rather enjoy having my DB driver, connection pool, kafka client, and web framework all emit their own logs, which is not possible if I wrote my own logging library.
> I get that the tradeoff it is worth it for complicated things (e.g. crypto libraries). But logging, really?
Logging at scale is more complicated than people give it credit for. Personally I've had my services suffer noticeable performance degredations due to poor logging framework config. We've ended up with a fairly complicated logging config that allows us to carefully balance collecting as much developer useful information as possible against the need to drop excessive logs during periods of heavy activity. Oh, and it's gotta be converted into JSON for aggregation into our observability tools, and decorated appropriately so that our logs can be correlated with our application traces, because that is incredibly useful during incidents.
Oh, and it all needs to work no matter what threading configuration you throw at it. That matters a lot too.
Do we need all those things? Arguably no, but then again they are helpful tools to have. Without them, the team would have a harder time developing on our systems. You have to understand the use case that people need to fulfill before dismissing the entire thing as unnecessary and pointless.
> Software development culture today is too quick to adopt a huge tree of dependencies of unknown quality, rather than thinking about how to minimize dependencies to only those truly necessary. The leftpad fiasco was but an extreme example of this, but I see it all the time, and it seems probable that there are hundreds (maybe even thousands) of similarly severe problems out there in widely used dependencies that we just don't know about yet.
I see this argument a lot, and I think it's a great example of "grass is greener" syndrome. People don't think through the practical consequences of drastically reducing their library usage. I personally worked on a project that had basically no external libraries due to the language choice, and we wasted inordinate amounts of time trying to chase down bugs we wrote in things like our database library, localization library, and yes, our logging library. Looking back I can say with very strong confidence that what we were doing was a massive waste of the company's resources, and rewriting it into Java or C# would have been the right call, even with the complexities those ecosystems bring to the table.
Yes, using external libraries comes with tradeoffs. Fixing bugs like this is one unpleasant possibility. But let's be clear that the alternative is a massive drop in developer productivity to write basic functionality that already is implemented elsewhere. Having been on both sides of this equation, I can confidently say that you'll spend way less time fixing security issues as they arise than you will reinventing the metaphorical wheel.
(I do agree that leftpad is an example of going way too far with libraries, but I think the library culture for NPM and the Java ecosystem are quite different).
It's because your program probably depends upon other libraries, which all need to log their own info, and your life will be significantly easier if you can view, save, and control logging from one centralized place at the top of your program. If every library rolled their own logging, configuration would be a huge mess, and your program would need to specify log files/formats/levels with each API call (or at least on API initialization).
Logging facades like slf4j help a lot, but then you still need to implement the required interfaces. For something like the specific Log4J CVE, things would be significantly worse if it weren't centralized in a library. They have a bug in deserialization when talking to LDAP servers, allowing a compromised LDAP server to execute code remotely. It's likely that a subset of libraries would likely have rolled this functionality independently, and then you have many libraries that all need to be patched and updated instead of one.
Consistency of configuration is a big thing — I can configure my Spark, internal framework, and application logs in the same file, for example. That’s actually one thing I liked about Java, how easy configuring logging via Log4j was and how widely adopted it was by most dependencies.
In terms of features, one that comes to mind log rotation.
I dunno, I suppose all together I’d answer your third paragraph to the affirmative — I’m more productive and the libraries improve significantly within crucibles like this.
The issue is that java did not initially ship with logging capabilities, and by the time it was part of the java.util package, a handful of logging systems had already been introduced by library vendors. JBoss, Jakarta Commons, Log4j and ultimately slf4j all were introduced to address these shortcomings.
The larger issue is in javas dependency system are built with dependencies on these libraries, so if there is no commons logging (or bridge, ala slf4j), then the program won't run.
There are a few other techinical traps with the jvm, like using system.out logging can cause issues with threaded code, or having one giant static logger being imported all over the place conveniently (this is before inversion-of-control became the dominant paradigm) had a few shortcomings, especially as programs grew. Ultimately, though, if you were going to use some third party libraries or app servers, you were almost certainly going to get pulled into the commons-logging vortex, which means you're now configuring that AND your homespun logger.
There is also the case for things like SOLR, where it might be just being used out of the box, but the distribution includes the affected JARs, even if you're just posting and retrieving documents with a python script doing print-logging.
Honestly, I didn't even know log4j2 was a thing, I've moved as much over to slf4j/logback as I could ages ago because of the madness of the JDK logging ecosystem.
When Java finally added a logging library, they added one that no-one had ever heard of before, and which had fundamental flaws. That's why no-one uses it. At the time, log4j existed and was in wide use, but was not represented on the committee.
The author of log4j eventually decided to go a different way and created slf4j / logback. This offering is compatible with log4j and considerably simpler in many ways.
Having not been in the Java world for many years, I was surprised log4j was still in widespread use. But maybe I shouldn't be surprised - whenever we have to interact with Java teams, we always have to tamp down on the tendency towards complexity.
It's a bit of conservative culture, and a bit the result of success. There is a lot of Java code out there, and a lot of it is quite old. Finding and replacing every single instance of log4j out there would be a lot of work, and before this incident of relatively low benefit to everyone involved.
Very well explained. About log4j2, I think it came to exist because log4j fell far behind logback, and many veteran dev didn't like the inherent push for slf4j over logback. While logback can be used standalone, snobs would yell at you. And since figuring out how to config slf4j over it can easily be less than half a day of work, many simply drooped in log4j2 and told snobs to go away.
Note: In case someone is tempted to say that it is trivial to plug slf4j over logback, it is for greenfield work. Replacing an existing logging library, or the use of multiple logging libraries in favor of slf4j over one lib is a confusing matter, mostly because the error thrown at runtime are misleading.
> What is so complicated about logging that you can't implement the necessary functionality in your own codebase, and instead have to adopt a third-party library?
(1) Coordinating with dependencies requires at a minimum a common API (which, in a statically typed language, means a library providing at least the interface).
(2) Configurability to log to different destinations and with different formats, which you eventually tend to want on major or long-lived projects, can be fiddly, as can the specifics of different formats and targets. Each of the bits is simple, but added up it's a lot over time, and reinventing the wheel, the axle, the tire, and the air pump for it just doesn't make a lot of sense. Especially given the need for a library for #1, it's best to get the core functionality together with it, and the peripheral pieces either together or as plugins, and only solving unique problems, if you have them.
We asked the same thing about log4j over 10 years ago. We spent an hour or two and switched to slf4j / logback. It's written by the original author of log4j, and has a compatible api. Much simpler and much more understandable.
> implement the necessary functionality in your own codebase
Sir, you might have just extended all of our careers by at least a decade. We salute you.
I agree that developers are too quick to put themselves at the mercy of 3rd parties, but it's a risk-dollar trade off we'll continue to make because reinventing the wheel each time would sap our margins.
It is not complicated. Structured or semi-structured logging is the only way to log things and still be compliant with all the demands.
A lot of these libs separate logging into layouts and parameters. If you log input parameters as layouts. You might break the logging. I have seen this at every company I worked with.
The only really surprising thing about log4j is the remote code execution.
> Software development culture today is too quick to adopt a huge tree of dependencies of unknown quality,
Log4J has been around 20 years. Log4J inspired and was lifted into the JDK. JDK logging is essentially an inspired copy of Log4J. The Sun coders were no better or worse than the Apache coders. I use JDK logging personally simply because it's one less dependency. JDK logging hasn't change since JDK 1.4, and is weaker than, well, pretty much anything else. But, with a simple wrapper I've used for 15 years, it does most everything I want from a logger.
That said.
Log4J has been, and is still, a boon to the Java community. Arguably, Log4J is the root of a tree of vast array of logging frameworks, across languages. Java server developers essentially live and die by their logs. It's routine for developers in dark rooms with screen lit faces to pouring through logs with endless stack traces. Thank heavens for Java stack traces.
Logging is part and parcel to the Java server side experience, and even the client side, and we can place much of that on the shoulders of giants like Log4J because it made logging easy and set the stage. It's so helpful, so useful, so flexible (obviously, perhaps, a bit too flexible), and so powerful.
Because Log4J inspired the other loggers like the JDK Logger, we have logging shims. Shims like Commons Logging, that act as intermediaries that can have adapters written so that we can use other logging frameworks. Log4J was one of the first, was, and is still, dominant in the community, but it's not alone, and thus a bad choice for things like libraries. Instead, those choose the shims like Commons Logging that developers can use to configure to route through Log4J or JDK logging or any of the others. That said, even programs that use Log4J directly can be routed to other loggers.
This is all entrenched. It's part of the flavor of server side Java, configuring the different logging shims to write to your logger of choice on your system. Just the way it is. But, that's what happens when you don't live in a mono-culture. Feature, not a bug, and it helps empower the vast array of software that millions of developers and applications rely on everyday. Nobody designed it this way. It didn't start this way, it just evolved this way. It's a very "Java" thing.
I watch Stack Overflow questions and 90+% of the time when someone asks "How do I" what they mean is "What library do I need", not "How can I write this". This is the sign of the times, and Java is not alone. The beauty of Java is that it made this kind of sharing REALLY easy. REALLY REALLY easy.
Apache has a solid reputation for good projects and good code and good stewardship. It's not a back alley transaction to grab an Apache Java jar file and shove it in your project. Is it all perfect? No, but what is?
You of course have a point here, the massive dependency trees of modern apps carries a certain risk.
The problem with rolling your own is that, for a non trivial technology like a producionised logging framework or some encryption library, you're highly likely to write a massive security hole yourself. The great thing about battle-tested open source software, used by 1000s, is that years of real world use, development, bug fixes and general scrutiny from a large number of people makes for a high level of reliability normally.
Obviously here we have a problem with this approach but avoiding third party dependencies carries its own risks and obvious costs too. Let's focus on how reliable so many of these libraries seem to be and how rare problems of this scale are.
I think a bigger problem here is the fact that so many people are using so few libraries. Maybe we need more open source libraries to counter single points of failure. As with how we deploy our code on the same clouds and so on, single points of failure abound these days. Things fail however well they're engineered and therefore we need diversification of everything.
It’s worth noting that log4j does a lot more than, say, JavaScript’s console.log function. It’s dynamically configurable, supports a wide range of output formats and destinations, easily filterable, and highly optimized to reduce performance impact (in part through delayed evaluation and multi-threading).
Rolling your own logging library to reimplement it is not trivial, even if you ignore the more esoteric output formats.
Sure, in retrospect they went too far with some of the dynamic logging features in log4j2, but it is no exaggeration to say that log4j revolutionized logging in the Java world when it first came out, and heavily shaped later solutions like logback and java.util.logging.
Not to mention a wide range of destinations, too. Want to log to a database table? You’re covered. What about log messages to a chat server (xmpp, slack) or email but only fatal errors? You’re covered.
Now roll your own logging system that supports those destinations safely and get back to me to do pen testing. I bet we’ll find some vulns in your code.
This is the job of a log collector, not a log producer/library. The logging library itself should have no mechanism for shipping files anywhere except a file / stdout / stderr.
Hindsight bias. Keep in mind that Log4j is older than most log aggregators by a solid decade. This is a solid reason to not use log4j going forward, but an ahistoric charge against it in the era when it was first written and adopted.
Most of us have log4j in our system because it's used by some various dependency of one type or another, and up until now it has generally just worked.
It's a logging library. You don't reconfigure logging, you just make another logger and use it. And by the way, you take the configurations from your global configuration system, "dynamic configurable" is a feature of the configuration system, not of the logging.
> supports a wide range of output formats and destinations
Yeah, like network aware templates. (But the destinations are a feature, multiple destinations are one of the few things a logging library should support.)
Despite what people claim, the Java culture of loving complexity never did go away.
> It's a logging library. You don't reconfigure logging, you just make another logger and use it.
Prior company had a shared plugin that let you turn up the logging for a given logger remotely for 30 minutes (it self reset). Extremely helpful for incident debugging.
Now we can have an interesting discussion about whether that capability is worth the complexity, but that’s a very different discussion than unilaterally declaring that other people’s use cases just don’t exist.
> Despite what people claim, the Java culture of loving complexity never did go away.
Do you develop in Java? Because I do, and your takes don’t match my daily experience one bit. My experience going from Spring 3 to Spring Boot 2 is in fact one of reducing complexity (goodbye XML bean configuration!) and to more “it just works” situations than before.
This turns gigabytes per hour of logs to hundreds of terabytes per hour during normal operation, in many cases it's an inferior solution for problems which pop up in a fraction of infrastructure several times a month.
Ok? One you need to disable debug logging and two I still don’t see a problem with that amount of data. You also don’t need to keep logs more than a month or so. How much log data do you think cloudwatch and the other majors filter per hour? much more than terabytes.
Log emission can often have a negative impact on performance, especially since peak logs and peak traffic tend to coincide. I’ve seen P95 latencies suffer just because of logs.
> How much log data do you think cloudwatch and the other majors filter per hour?
How much do you think the major providers pay for that infrastructure per hour, and are you willing to shoulder that cost too?
(Hint: log ingestion is $0.50 per gigabyte in cloudwatch. So a terabyte of logs an hour is $512/hr)
> Log emission can often have a negative impact on performance, especially since peak logs and peak traffic tend to coincide. I’ve seen P95 latencies suffer just because of logs.
Only if the emissions of said logs is doing more than stdout or fs writing and it’s not done in another thread.
> Log emission can often have a negative impact on performance, especially since peak logs and peak traffic tend to coincide. I’ve seen P95 latencies suffer just because of logs.
Yes and you need to fix that terabyte per hour garbage as that is well beyond typical for a single service. We’re also talking about java so 7000 page stack traces is where i’d start…
Why is there any problem in restarting your logging system? There is nothing in a logging system that could be harmed by a restart. Restarting it is exactly how you should handle changes, because it deals with plenty of stuff that isn't reconfigurable on a lower level.
The only way to make a "reconfigurable" logging system is by having a stable facade that restarts everything behind it. But that extra layer does not belong on a logging system, there are plenty of other things you need to make "reconfigurable", and now you are adding an extra layer into each of them, instead of having only one on your main entry point.
Pushing that feature down into the logging system is the choice that maximizes complexity and minimizes functionality.
You act as if “restarting” the logging system is easy and doesn’t require the same facade that you decry for a reconfigurable one. With dependency injection, there are loggers everywhere in the system, possibly in sections of the code you didn’t even rewrite. Restarting such a system while continuing to run everything else is just as complex as reconfiguring, possibly more complex.
Also, I noticed you didn’t answer my question about whether you’re a Java developer.
> I was once, but haven't used it professionally for a while.
This is why I ask. Java has changed a lot since ~1.8, and it is super common to see ex or non java developers making grandiose and sweeping statements about a community and language that they are no longer part of. Having developed in Java on and off since ~1.7 and Spring 3, I can certainly say that I have seen a strong movement towards less complexity and more convention. It's not perfect by any stretch, but I think saying that the space is "in love with complexity" is incorrect.
You of course don't have to go and re-use Java on my suggestion. We all have our preferences, and that's ok! But I would generally recommend against such sweeping and dismissive general statements about a culture that you are no longer part of. Things change, and your experience back then is probably not representative of things today.
> Oh, and now that you described it with so many details, the attachment to complexity does look more like addiction than love.
There's a fine line between disagreement and insult, and you are over it.
If you turn on debug logging to see why somewhere weird is happening, and the only way to do that is to completely restart the entire system - good luck figuring out why a large class of weird things happen!
I think the mere fact that use of log4j is so wide-spread answers your first question. It's not that simple to do it (well) yourself.
That said, I do agree that there is a problem with large dependency trees. It seems building a simple hello-world app with Maven ends up downloading about 1/2 the internet. :)
It's been a minute since I primarily developed in Java, but back then we had switched away from log4j in favor of slf4j / logback. These were written by the author of log4j. We switched because they were significantly simpler than log4j.
Switching over merely involved changing the jar files we used, and redoing the config.
It's amazing that this library is used by so many large enterprises yet none of them made an effort to thoroughly audit the code (actually, it looks like Alibaba reported the first one. But the point stands that the code was used for so long and by so many.)
Each enterprise doesn't only use this common library, it uses a few more hundreds. If the point of pulling libraries is to save tremendous time in not having to write them, I can assure you managers aren't keen in wasting their gains in looking deeply into what's inside.
Spotting the flaws, even with careful reviewing is still unlikely to happen, and would benefit the community at wide over their selfish interests.Enterprises use of open source doesn't adhere to the open source étiquette.
Not only are they not interested in spending time auditing code, your typical enterprise lacks the skillset to do so effectively. They couldn't do it if they wanted to, short of contracting it out.
Thats the point. Giving back money for the libraries used might not fix anything and can be difficult in many cases. But the most and important way of "giving back" would be testing and review of the libraries used. If especially the large companies using this library had used some of the dev hours for testing and review, the bug should have been found early on.
Code auditing is extremely expensive. I recently did some napkin math and my estimate is that for our relatively small project (a company of ~10) it would cost between 2 and 12 million dollars to have our rust dependencies audited, with Rust making up ~60% of our codebase.
And that's a point in time audit. To maintain that value we'd have to redo the audit periodically.
Dealing with the current vulnerabilities is very likely cheaper than auditing each and every dependency with the thoroughness needed to actually catch those vulnerabilities. Also, nobody within the by affected enterprises gets blamed for those external vulnerabilities, nor are the enterprises liable for preventing them, so there is little incentive to prevent them proactively.
I did this years ago with an open-source and reasonably popular non-Apache library. The results were so horrifying my boss quickly concluded such audits were outside our responsibility and the time I spent on the audit was written off.
It was especially difficult for us because we’d shipped so much code that used the library, and replacing the library was unthinkable.
I assume at this point that this library has been audited many, many times by skilled professionals, none of whom thought to abuse the template + JNDI + LDAP chain in this way.
It's just not as simple as "security audit finds all the vulnerabilities, then you fix them." You invest X in the review, you get the results that X/(hourly rate) finds. This is a lot of software with a ton of configurability-- that's a lot of variations to review and test.
Now that someone found the first lump of gold and gave it away, there are thousands of eyes searching for the next one. These recent findings are all abuses of this same chain of functionality, just along different sets of settings. In another month we might have half-a dozen more of varying severity and scope. That _still_ won't prove that the overall library is then safe, but we will probably have a a little more confidence in this particular bit of crazy template formatting flexibility. Maybe not as much as we had had three weeks ago, but more than we do now.
Yes, and it’s only applicable in very specific and likely quite rare circumstances. If not for the initial RCE vulnerability, no one would blink an eye.
They are generally high quality and implemented by cryptographers. Parts of the implementations are certified by NIST.
Log4j is not part of the JDK and is an open source project with only few regular contributors. There is really no reason to infer any correlation to other parts of the Java ecosystem.
I see plenty of comments about security teams that should be doing this or that.
I do not know about your security teams, but my security teams are now sitting 9 days straight trying to stop that shitshow. With teams complaining that they have to patch instead of doing Great Things.
After that these security teams will be back to invisible work and forgotten again. Until the next issue.
If your company has a truly different approach to security, tell it here - you may get really talented people knocking in because they are fed up with the security theater at their current job.
The most reliable place to find new bugs and security holes is next to others you just found out about.
Bugs come in bunches. If you have fixed just one or two, more lurk right there. And, anyplace else that coder worked. The more you have found, the more remain to be found. Look at other places that coder worked that week. Or year.
Your implication that specific bad coders are responsible for bugs is dangerous. Anyone can write a bug, and thinking that you're too good to do it is cocky.
Every bug is coded by a specific coder. We know there are huge differences in productivity of different coders. We know they repeat mistakes. Thus, the best place to look for another bug is in other code the same one wrote.
But looking in your own code is probably next best. Ask yourself: if there is a bug somewhere in my code, where would it be? You may be surprised at how immediately the answer comes to mind. Look there.
Yes, but when a coder makes a specific mistake, it's likely they made it elsewhere. Similarly, when their mistake made it past code review, it likely made it past review elsewhere. They're not the only person making that mistake, but their diffs are a great place to start looking.
A big subset of mistakes people make when writing software are the result of people not having the same understanding of a concept, policy or practice.
It really depends on the subject area of the code in specific. “Mistake” is a very broad term, in most cases when it’s used it’s not the implication that the writer didn’t know what they were doing, rather they wrote erroneous code—which ultimately everyone does at some point. What it is however indicative of, is a lack of thorough testing and review that could have potentially caught the error. Obscure errors in places such as parsers, rendering logic, complex algorithms, memory management, etc. are particularly tough areas because building a good exhaustive testing workflow can be a challenge.
Even a good writer will make mistakes, otherwise we wouldn’t need proofreaders.
Is this still about malicious payloads in serialized Java packets? I don't use Java so maybe I'm being unjust, but I'm smiling slightly because Python went through this 20 years ago with regard to the pickle protocol. It was originally intended to be ok for transporting hostile data, and it took a fair amount of persuasion and a few security incidents to turn the community's thinking around about that. ast.literal_eval was invented partly for this reason, but the generally preferred way to serialize now is usually json.
So, again, why is log4j "evaluating" stuff in log strings? This is the root cause that people should be addressing
Why are people still using JDNI? It is frail enough to have logs depend on a dns lookup, I can't imagine depending on a LDAP lookup. This is not such a fundamental issue as the above but it is pretty dangerous.
Oh yes and of course one of the worse and least pythonic package in the Python standard library was "inspired" by the log4j monstrosity
Because "java:comp/env" is the way for a J2EE web application to get values or objects from the configuration outside its container. Yes, nobody actually uses the remote parts of JNDI anymore, but the local variant (within the same process) is still in use.
And that's what led to this vulnerability: someone wanted to be able to get these local values within the logging configuration, which is a valid use case. Not realizing that this opened the door to remote values and objects.
I maintain `debug` for Node.js and the Log4J devs are living my worst nightmare when it comes to that library. It's used just about everywhere and any sort of zero-day would have far reaching implications.
This isn’t a RCE right? Compared to the previous one I wouldn’t call a denial of service high severity at this point. It might have been high severity two weeks ago.
But to be clear: nested/recursive template expansion and expansion of user provided strings were never by design, correct, and the removal shouldn’t just be by configuration or considered breaking - it should simply be corrected (just like the jndi should be removed and not even be optionally possible)?
Okay, more seriously... this is why I am currently staying away from Rust. Because sometimes I don't even know if the random number generation crate I imported for my banking app came from Rust team or some random dude in zanzibar.
(Sorry, I know HN loves Rust, so do I. But I currently that the cargo/crate system is a bit too easy to use and misuse).
I was pretty horrified to learn that log4j makes network calls, was even surprised to hear it parses log strings.
Fortunately our log4j was too old and JVM too young to be vulnerable to the first exploit, but it seemed best to rip out log4j and replace it with something simpler anyway.
java.util.logging seemed the obvious choice, but the API is different, so would have been a lot of s/debug/fine and so on.
In the end, we:
- wrote a new class with the same 'api' as the part of log4j that we use
- globally replaced 'import org.apache.log4j.Logger' with our new class
- fixed any compile errors by adding new methods to our class
- wrote a test to make sure it rolled correctly and could handle lots of threads etc.
Here's the code if you want to do something similar (don't know how long it will be accessible from there): <https://ideone.com/XKg5M9>
334 comments
[ 4.8 ms ] story [ 276 ms ] threadI hope you step on a Lego.
The disclosure process was terrible, they clearly had no time to perform a thorough validation of the hotfixes, and this is the result. Why didn't anybody else step up to help them? Did any of the large companies using log4j pour some thousand dollars into the Apache foundation to let a couple of security-oriented engineers take a complete review of the patches?
But then, it's log4j developers and Apache that should be blamed. Bah. That's SO pretentious.
Um, yeah. They wrote the idiotic code that allowed this to happen. Who should get the blame then, the users? Duh, no. Should the hackers taking advantage of the exploit? Maybe, some, but they wouldn't be able to do it if the code wasn't so badly wrritten.
>Why didn't anybody else step up to help them?
To me, this is the pretentious part. I didn't write the code, I don't use the code, so why in the world should I be expected to fix the code? Do you use the code? Why didn't YOU fix it?
> who should get the blame? The users?
If you leverage a ton of open source deps, shit is going to happen. You should not blame anyone.
But if you must, yes: blame the users that choose such piss-poor software.
People keep singing the praises of FOSS, but yet whenever this kind of shit happens, it's always the user's fault for choosing poorly which packages are being used. I'm saying that the original devs are criminals, but they did the original sin here. The fault ultimately lies at their feet. Why is this such a unrespected point of view?
Come to think of it, they wouldn't even have to search for any exploits. Just ask devs nicely to put the backdoor in and be done with it.
What original sin? What fault? You’re basically saying that log4j developers lured you into temptation, otherwise you would have coded your own logging library, which, for sure, would have been better.
I haven’t seen 1/100 of this hatred for Apple iMessage vuln that led to NSO zero-click exploits, and I don’t get why.
I think there are a few reasons for this, and most of them aren’t based on logic, but emotion.
First, as bad as the NSO zero-click exploits are, they aren’t things that developers were potentially injecting into their own projects. So there is an easier force to “blame” — Apple (for making the mistakes in the first place — and to be clear, all software can have bugs) and NSO Group (for being pieces of shit) — whereas if you either chose Log4j for your own project, or more likely, chose a project/product that chose to use Log4j, the “who to blame” question becomes a lot less comfortable to answer.
At least with the NSO thing, there is a face of who the bad guy is. So it’s easy to sort of not hate Apple, provided Apple patched exploits as soon as they found them. That doesn’t mean that Apple, a company that has used its “better” security relative to its competitors one of its marketing messages, gets off easy. It’s a black mark for them for sure and will make it harder for the company to argue that it is so much more secure than x, y, z service or platform.
With Log4j, similar to OpenSSL, people have to grapple with the reality that they don’t really know a lot about a lot of the code they run — and even worse, acknowledge that they aren’t capable of understanding or auditing that code themselves. I certainly don’t understand a lot of the code and libraries I rely on for things I build. And if I really stop to think about that, that’s scary. Fortunately, my own projects are personal and aren’t taking data from others. But facing that reality can be a difficult pill to swallow. And Log4j is even worse than OpenSSL in a way, because one could be forgiven for having trust in a well-respected cryptographic library, and for not having a deep understanding of how that library works. But having an insecure logging library? That almost feels avoidable — even if it isn’t.
So people want to blame someone. And when it looks like the code — made by volunteers or not — wasn’t the best written, that’s an easy thing to go to. Rather than self-reflecting about all the libraries we use every day that we don’t think about the security or code quality of.
It doesn’t help that the disclosure, for a host of reasons, wasn’t good. That isn’t me blaming the Log4j team, it just is what it is. Disclosure wasn’t great and the fury to patch made some mistakes and now that there is outsized attention on the library, even more vulnerabilities have been found. Which is all understandable and indicative of what often happens in high stress situations.
Ultimately, we should all accept that mistakes happen and that this wasn’t done out of malice. We all have some level of culpability for what we choose to use in our projects. But mistakes happen.
And as you say, discussion is OK but disrespect isn’t. But I think we see lashing out because people want to find someone to blame.
No, I would never use a logging library, so there was no luring. If you want to call those that did use it lured, then you're already coming at it that the original software devs were coniving.
That’s an unbelievable stretch. I wonder how your mind works.
And released it with NO WARRANTY. And no mega corp thought it’d be a good idea to try and buy one.
> To me, this is the pretentious part. I didn't write the code, I don't use the code, so why in the world should I be expected to fix the code? Do you use the code? Why didn't YOU fix it?
If you don’t use it, then noone’s expecting you to fix anything. But those out there using it for free, complaining that it’s broken, and not doing anything to help? They should hush their hush holes. Or open pull requests. But mostly hush their hush holes.
> Who should get the blame then, the users? Duh, no.
And then you say:
> I don't use the code, so why in the world should I be expected to fix the code? Do you use the code? Why didn't YOU fix it?
Just take a step back, stop trying to throw shade at anyone and everyone, and look at the situation analytically.
I don't really see the point of these comments. The people who make a contribution to the world are the people who see a problem and take practical steps to fix it and improve the situation, rather than the people who see a problem and immediately descend into finger-pointing and shouting.
Decent logging is surprisingly complicated, but it's not "We need an arrrchitecturreee! for infinitely configurable logging or we are utterly doomed" complicated.
Even with proper disclosure, patching can be harder than it seems at face value. Take shellshock (the bash RCE from 2014) for example, the first patch was released a week after discovery and publication happened after the patch was released. Yet there were still a flurry of further CVEs found and further emergency patches required as a result.
The problem is once a major vulnerability is discovered, you then have a the worldwide attention focused on that specific vulnerability looking for other ways to break it. So it’s not really a surprise that you find multiple CVEs for most major vulnerabilities.
How could they be expected to? Reviewing all of the source code for log4j is not a walk in the park. Even if you thoroughly reviewed all the source code, would this specific exploit have crossed your mind?
The game theory and assumptions around a "logging" library probably led a lot of organizations to not even consider it a potential threat and focus their efforts elsewhere.
There is also the transitive dependency angle which turns this into more of a plague than a blame assignment session.
Back a long time ago, companies were extremely wary of including any open source library (let alone thousands!) into their product. It took a very long time with audits, lawyers, etc to get a single library approved.
Yes, it was kind of a pain.
So the world has swung into the other extreme these days. Just pull in a couple thousand libraries from the internet, nobody knows what they even are or what's in them. All in the name of speed. So it has its benefits.
But the fundamental concerns of long ago are still there. Bringing in unreviewed third party code is a vector for vulnerabilities, supply chain attacks and all that joy.
Traditionally if something is too large to audit then you shouldn’t use it, which few exceptions (like Linux). Maybe this should be a red flag in the future, especially for a logging library that can almost be replaced with a call to stdout and a log collector.
The point is to keep you secure, not be a slow news relay. In that regard, GP is just really bad at his job.
We won't hold a grudge against you; open source means collaboration, and you don't blame hard-working people that give away their software for free for honest mistakes. Your software still did far more good than bad to the software world.
Thanks!
I thus enjoyed a development career (mostly) free from this kind of needless pain and suffering.
It took a simple problem and turned it into a clusterfuck of configuration and complexity.
It is worth mentioning, that any even very basic solution always has a possibility to shoot in the leg, and only professionalism of the developers of the client code can avoid it, so the real reason why this is happening now on a massive scale is that Java world is big and has probable hundreds of thousands, if not millions of developers, who do not have enough knowledge or experience to write better code, understanding all the possible side effects and other consequences of their decisions.
This was going to happen eventually. It will happen again, but maybe next time it will be a Python or TypeScript library. Would it be right to call their ecosystems toxic? I honestly do not know. But let's at least have some respect for people who have built it, almost for free.
Why should I show respect for software I hate using and that causes problems for the entire tech community? I appreciate that they built it for free. I write open source code too, and glady accept criticism if I release something that sucks (which I do often)
This saga is a lesson in choosing the right tech, and switching off libraries built for use cases 100x more complex than what 99.9% users need.
What knowledge do you have, about this library which you've never used, which qualifies you to say that it's especially prone to security vulnerabilities - other than hindsight bias?
You aren't expected to. You're expected to show respect for the medium with which you are expressing those thoughts (HN), and the other people you are communicating with on that medium, by keeping the discussion civil and doing your part to keep it from devolving I to a flame-fest.
Almost nobody cares that you have and express dislike for this software. They care that you're doing it in a way which encourages additional discourse which isn't useful and wastes everyone's time.
At extremes it's the difference between staying you think a person is immoral or untruthful with reasons and calling someone an asshole. One of those is easier to have an actual discussion over than the other.
Yes, it's free. No, the vast majority of users never actively opted into it. No, it's not disrespectful to suggest that a log library that interpolates parameters was never fit for purpose.
This class of problems was back then called DLL hell, and guess where the „DLL“ comes from? Hint: not Java. In any case, it is quite rare by now and calling ecosystem „toxic“ because of that?.. Pretty sure you haven’t seen the others long enough.
The other example for those that are curious is micrometer. The basic API and the implementations (prometheus, statsd, etc.) are separate jars.
https://news.ycombinator.com/newsguidelines.html
This wasn't just a predictable scenario, it was predicted. Or more accurately, it has occurred already repeatedly in the NPM ecosystem, but for some mysterious reason those incidents were simply ignored by security teams world wide.
Instead of chilling them to the bone, they simply shrugged their shoulders and said "Well, we don't use NPM... I think. Probably?" and went on with their paper-pushing or whatever it is CISOs do these days.
If you think Log4j is bad, wait until Rust gets popular and has something similar happens with a commonly used crate. How would you scan for a vulnerability in compiled code that doesn't even have separate module files?
I had to help an organisation with log4j that had on the order of 5K distinct executables/applications across 3K servers on two clouds and three on-prem networks. At least with log4j it's a simple matter of finding JAR files and scanning their contents. They're just zip files! With languages that output a single binary by default such as Rust and Go, we would have been screwed. No way to scan, no way to self-help update.
Introspection and post-shipping updates for security are mandatory. Rust and Go like to pretend they aren't, because they originate from organisations that are in total, end-to-end control of the software on their own network. Google famously uses a monorepo and can build everything they run from scratch in short order. The shortcuts they can take with their post-release management will never work for ordinary organisations. Never.
So let this be a lesson: Log4j was made for a language that at least allowed us all to find the issue and fix it ourselves. We have Sun's forward-thinking and the enterprise-friendliness of the Java ecosystem to thank for that.
> I lay any blame squarely at the feet of IT security of large organisations that were entirely unprepared to update a widely used dependency that wasn't an operating system or a runtime.
If there is the possibility to plan ahead from a sufficiently strong bargaining position, you will not end up in the situation you described. The seller should either continue to support it with a good SLA, or you get the source code with a turn-key build system if the seller discontinues its support. Or you switch out the software if the SLA ends. Should you not be able to agree on such terms, I don't think you would have enough influence to tell the seller to write their software in such a way that dependencies are easy to hotfix/upgrade.
Ok, after putting the unicorns herd to bed, an event happens and congratulations, you now own a license to PeopleSoft v.whatever source code. Oh yeah, you also got the management platform for your network provider too.
Now what?
There is typically no link back from a deployed system to its source, even if you compiled the binary within your organisation.
If it came from an external organisation, things are exponentially harder.
Modern deployment systems are largely "one-way", with no way to trigger a full recompile from the deployment end of things. If you have a VM with "SomeRandomBinary.exe" running on it that has a vulnerability in it, you don't have many options.
IMHO, typical IT as done in the wild suffers from this write-once attitude, where the chain of provenance is too easily broken.
Companies like Google have a very non-typical setup that can be copied only by very small orgs like startups. Typical enterprises look nothing like a startup or a FAANG.
IDK where this idea that no one can know what's in a dependency tree is coming from but is not ops best practice for more than a decade.
I’d guess people running commercial non source available binaries in production. And outside startups and unicorns, that’s almost everybody.
How do you find the dependancy tree for your on prem Oracle db, or your self hosted Atlassion stuff, or your non cloud ServiceNow or PeopleSoft stuff, or your Huawei network management stuff, or or or…
And what can you do for any of your cloud hosted saas stuff, beyond telling legal “I dunno, their website hasn’t said whether they’re vulnerable to exploit de jour yet”?
Depending on the product, maybe forever.
I have some bad news for you: viruses and hackers don't care about your support contracts and the delays they cause.
Actually, I tell a lie: the hackers love them.
a) It's very rare to patch things manually
b) It's very rare to develop patches yourself (for 3rd party software)
I'm sure some companies out there with many decades old deployments have to do stupid shit like that, but it's hardly the standard.
If it’s some important piece of software you also can’t stop or the business stops - well, you and everyone else is just going to take it in the pants.
Many enterprises are in exactly this bind. And support contracts only get you so far.
Many people don’t even realize how long it can take to get a real issue fixed even with a super sweet support contract. For instance, I once was a wet behind the ears Oracle DBA, and when creating a schema i dumbly used a feature in the documentation without checking to see how long it had been released. Unfortunately, it had been released about 6 months before, as I found out later.
It took 6 months for them to figure out that it was the cause of our sporadic ORA600 errors (aka core dumps), by which point we’d already migrated to MySQL for many of the workloads because the database was having at best 95% reliability because of it.
Once we migrated the table off with the feature, somehow our problems ceased, but that was still well before they told us what was going on.
Moreover he is arguing that java, as opposed to fat binaries with potentially stripped symbols, made things solvable. He fears that many companies would be much worse off with a big vulnerability in e.g. a rust crate.
I don't think we should optimize for garbage companies with garbage practices.
I think we should optimize security for say the 80th percentile company as far as not following best practices (i.e. 80% of companies have better practices than what I suggest targeting).
The vast majority of cases, and this has been the case for decades, will be that you take responsibility for your own dependencies, and you let your vendors take responsibility for theirs.
> The go command now embeds version control information in binaries including the currently checked-out revision and a flag indicating whether edited or untracked files are present.... Additionally, the go command embeds information about the build including build and tool tags (set with -tags), compiler, assembler, and linker flags (like -gcflags), whether cgo was enabled, and if it was, the values of the cgo environment variables (like CGO_CFLAGS).
https://go.googlesource.com/proposal/+/master/design/draft-v...
At the time that Microsoft started doing it, paranoiacs thought it was some kind of anti-piracy software fingerprinting mechanism, embedding the compiling computer's MAC address or something. But really, it serves exactly the same purpose that Golang's effort does here: to let you map compiled-artifact back to inputs.
Yes...? What's the problem?
> why they don't instead hash all the source files and then embed that hash.
This is almost exactly what the git sha is. If you're arguing that README.md files shouldn't be included in the checksum, that's subjective. Many people would argue otherwise. And it doesn't hurt to be more accurate than less.
Do you not have this? Our docker images are tagged with the git hash they were built from, so at any point, for any of our envs, I can pull up the lock file of that build.
Our deployment config also describes everything that is running the relevant env, and nobody has machine access- going through the pipeline is the only way to get access, so there’s nothing deployed that’s a mystery.
Imagine you are Mr SecOps guy, and you've just ran some sort of Log4j tool across literally three thousand servers. Of those, several hundred came back positive.
Those included about a dozen flavours of Linux, a smattering of manually built(!) containers, and every version of Windows from 2008 R1 to 2022. Most of the code was built by third parties, some under support contract, some not. Most was built and installed manually, with developers using RDP or SSH to edit config files an whatnot directly on servers.[1]
So what you have now is literally just a string to a path, something like "D:\apps\foo\bar\baz\libs\stuff\thingie\log4j-core.jar" or the Linux equivalent.
Now what?
No, seriously, now what do you do? You're in SEC OPS. Not dev ops. You're certainly not in the dev team with access to the Git repo of some random vendor product like Tableau, or JIRA, or whatever[2]. You didn't deploy it. It got installed by a contractor during a short-term project three years ago.
A random hash string is totally useless to you. Even a repo URL and the commit hash will more than likely just end with an "Access Denied" URL, assuming you even have a network route to the Super Secure SCM Server.
There is no way you can figure out who needs to do what to make this go away. Not at this kind of scale at any rate. Not without first-class automation for literally everything. Which you can't have, because third-party software just doesn't play nice with any one tooling you'd like to use.
Containerisation? Bahaha... haha... snort. You're dealing with vendors that literally advertise "now with 64-bit support" and are unable to comprehend the concept of unattended command line installers. Vendors that insist on USB dongles for licensing. License that expire. Annually. And are tied to CPUID values. And on, and on.
[1] Oh, you think you can dictate release methodologies to these people? They're bureaucrats and they play politics better than you. Any word of changing their workflow in any way will immediately bring their boss, their bosses' boss, and maybe a few more levels up down upon your lowly head. You will have people literally screaming at you that your fanciful notions of build pipelines is "too much" and would "impact the work". That's the end of the conversion. I said THE END, and good day sir.
[2] Just get them to update it under their support contract? Ha-ha. Ha. Haaaa... We had vendors straight up lie about the vulnerability of their software. Then another vendor said that updating the JVM already mitigates the issues and hence they're not going to release an update. (Narrator: JVM updates aren't sufficient.)
Goodbye and thanks for all the fish. Run!
Now what?
The remediation (until you can get an update from the vendor) is to remove the JndiLookup.class file from that jar. It's been fairly well publicised, as has the way to do it.
I can't see how this is going to work as a strategy. Ad hoc cleverness? Sure, and that's always good to have when you're smart and lucky enough to pull it off. But it'd be nice to have a strategy that works without relying on being smart, willing and able to engage with tricky ad hoc solutions, and somewhat lucky.
Your company has an inventory of hosts to teams which you can look up and contact. In addition the package manager tracks which package the file belongs to. Therefore you raise a ticket with the team from the inventory tracker that package XYZ on host ABC contains vulnerable file /x/y/z/log4j-xyz.jar (The reverse is also true, for each package we can find the corresponding repo and set of hosts its deployed on). Fixing it is now that team's problem, and secs ops guy's problem is now just to verify when they claim to have fixed it.
If the software is JIRA? Doesn't matter, still needs a package built before it gets deployed on our machine. "No really, the vendor's only supported installation method is sudo curl | bash" - into a container it goes, the owning team of the container can be identified much the same way as the owning team of the host.
Now the problem you find is sometimes the result is that the host belongs to team XYZ and you find the team was reorged by some exec's great idea in 2019 and only one guy who used to be on the team is still in the company, but he left the team in 2018 and has no idea what that team did since he left, but the application has been still runnning and power $millions of real revenue. That one is much harder to fix by automation. The business reality is the security team can't enforce to the execs to not lay anyone off or disband any teams without a concrete transition plan for their systems.
ELF binaries are just collections of another sort. You can scan for symbol names or assembled instructions common to a dependency, and you might be able to update with an LD_PRELOAD library that patches the symbol table.
Just look at what game modders have accomplished without access to source code.
> Just look at what game modders have accomplished without access to source code.
Game modding is usually done on Windows given the target market (up until recently of course) and thus usually means Windows PE's, which do not have debug symbols attached - ever. Windows debug symbol databases are emitted as PDB files and are almost always omitted from game releases.
This means modding has to do a sigscan[1] in most cases in order to find something interesting - especially in the case of ASLR[2]. Then whatever modding framework (or hack) can set itself up and hook into the game.
These techniques are NOT what we should be encouraging, and are certainly not common, even for vulnerability mitigation. Re-compilation and re-deployment should be the defacto mode of operation in production, especially for systems that have sensitive information. Relying on bin-patching is not something many security specialists would regard as a "good" mitigation.
[0] https://en.wikipedia.org/wiki/Strip_%28Unix%29
[1] https://wiki.alliedmods.net/Signature_Scanning
[2] https://en.wikipedia.org/wiki/Address_space_layout_randomiza...
Library .so files are not harder to scan than jar files. They just need different tools, which everyone has. And Java systems are not easier to patch than others.
Of course they are. You can trivially reverse engineer Java bytecode back to Java, with almost everything except generic annotations due to erasure, and a few other small things. Tools have been around for ages that do this[0], and while they're not perfect, they can re-generate nearly perfectly compilable source code.
.so files are ELF files, which means while they have sections, those sections don't typically reveal much about the internals of the code. This means you need to convert machine code (assembly) back to a higher level language - which is not a trivial transformation. There are tools that do do this (e.g. hexrays[1]) but they are far from perfect, especially for optimized machine code.
Java systems are very, very easy to crack, RE and patch. There are virtually no protections. JAR injection is still a valid means of patching a Java application. No, manifest files and integrity checks are not a security measure but an error detection measure, and are trivially bypassed.
If you asked me to choose between patching this vulnerability in a .JAR vs a .SO, I would choose the .JAR any day of the week.
[0] http://java-decompiler.github.io/
[1] https://hex-rays.com/
So, more advocacy BS.
All the public entry points are spelled out exactly by name, and all the external symbol dependencies.
That also assumes that the binary wasn't linked statically - in which case, there are no external symbols to even look for.
Lastly, please stop calling someone educating you "BS".
Binary patching, JAR patching, disassembly, library interception, syscall interception, whatever, these are all tools that are on the table. You do what it takes to get the job done, knowing and weighing the pros and cons.
...why? You don't need code to be in an HLL representation in order to patch it. Where by "patching" I mean "automatic insertion of a JMP into the subroutine prologue, that branches to your replacement code."
You don't even need to write any assembly in order to do that, either; you can write your replacement subroutine in C or whatever. The replacement subroutine just needs to codegen as a C-ABI subroutine with the same register preconditions/postconditions as the subroutine it's replacing.
Imagine the vulnerability would have been in the vendor code instead of in the log4j jar. You’d. still need a way to fix and rebuild it.
Note that this doesn’t have to mean a public MIT repo on GitHub. You can have more restrictive gray box licenses between your b2b partners only.
What are you doing about it?
With log4j you have a brutal combination of:
1. RCE
2. Exposure
RCE happens frequently but often attackers don't have an easy time getting to the exploitable code. With log4j it's trivial - every app can be owned.
But here are some questions:
1. Why do those apps have the ability to make network requests?
2. For the apps that need to make those requests internally, why aren't those over mTLS?
3. For the apps that need to make them externally, why isn't that going through egress proxying?
4. Why are there credentials spewed all over your environment variables across your services? Are they short lived?
It's actually not super hard to have a worst-case scenario vulnerability like log4j be not that bad for your organization. A bit of hardening and even if something like this happens you're in a good position to wait, monitor, and patch.
not just for security either. i can't count the number of times i've traced a performance problem or bug to a piece of code that was doing some io when it didn't need to
That's a bummer, but I suspect many on HN can in fact build software that isn't total garbage.
> What these applications are supposed to connect to are likely unknown. This is a major problem at enterprises
Only if they fail to consolidate vendor integrations and/or have already insufficient staff. (Which is often indeed a problem.)
Simple, if brutal solution; Infosec audits logs and works with application teams to confirm existing traffic and pre-document new traffic out. With properly staffed teams you should run out of surprises within a year.
> Simple, if brutal solution ... properly staffed teams
The recruitment part is not simple, maybe bordering to impossible? for some larger companies. I say, based on my past experiences in how clueless big companies can be concerning what technical people they hire.
Depends on how you game OKRs. Implement auditing Q1. Contact all teams Q2. Have 75% of teams Q3. 90% of teams Q4. Just an example but its all in how you sell it. Big IT understands icebergs, you just have to give a good/accurate roadmap and execute if they agree on it.
You might be correct that a lack of insight on this sort of thing is an organizational problem. My experience indicates that if you can't get a list of dependencies based on a series of e-mails down an org chart, something is wrong. Either someone is managing too many services for the level of tooling you have, or there's a complete breakdown in org comms (which happens more in larger enterprises.)
I find it quite ironic to praise the forward-thinking of the company that has been instrumental to bring us into this mess via their vision of loading dependencies at runtime from an online repository. Sun imagined that the code and its configuration does not have to worry about how to fulfill its dependencies, but instead JNDI [1] could magically fetch the appropriate objects from wherever, don't worry.
I'd say I would even find it ironic to praise the Java ecosystem, which, due to its excessive over-engineering, has produced best practice frameworks where fully documented behavior has the latent potential to just be catastrophically exploitable by accident, as nobody is able to reasonably understand how all of it plays together. I have a hard time to imagine how you could enable loading of external code in Rust or Go by accident using run-of-the-mill logging frameworks.
[1] https://en.wikipedia.org/wiki/Java_Naming_and_Directory_Inte...
Designs like JNDI were widespread in the IT software industry at the time.
The academic prototype for all this was CORBA, the Common Object Request Broker Architecture. An object-oriented was to transmit data between systems running on separate processes, CPUs, remote servers.
CORBA was a next-gen RPC (Remote Procedure Call) framework, the data format on the wire closely follows the format for C structs using Sun-RPC, but was way better because these CORBA things were discoverable and described at run-time, rather than compile-time libraries.
That's right: RPC was initially a Sun UNIX thing.
Microsoft needed a way to do this, so they had COM, the Component Object Model, for run-time discovery and linking (late binding).. Microsoft got COM to reach across network connections, "DCOM", for Distributed systems.
Meanwhile, corporate IT was still pulling text out of databases and shoving it into C structs. It sucked.
Microsoft's object model was better, and Java was better than better, because you didn't have to license it to use it. Well, you did; it was like an MIT License.
Reason why IBM writes huge Java systems. And why Oracle owns Java. For some value of ownership.
Yay objects!
In these Java attacks the code is downloaded and run locally, with all secrets and data access of the attacked program.
It’s funny because when I worked in .gov IT, we were all characterized as stupid donkeys by because we weren’t able to just use NPM, etc to do cool kid stuff.
Good luck getting security people with the power and brass balls to do that in any company.
This works because the binary is compiled against the dynamic library, using the system-provided library. It isn't statically compiled, because even though that frees the developers from worrying about library versions, it makes the sysadmin's job tracking library versions be much harder. It isn't vendored, for the same reason.
Sometimes you can get away with just depending, and that’s nice, but other times there are strong reasons to embed. At that point, you own the code, and can actually do the things you need to do. (Of course, it follows that you own the code - and need to then actually do the things you need to do.)
Both approaches have their advantages, and both have potential for technical debt. Which approach offers the most advantage for the least debt will depend on the individual circumstance; I don’t feel that you can really advocate for one over the other in a general sense.
Our software scans take 3 forms:
1. Searching for file names
2. Running test exploits that have a payload that just logs the vulnerability with a central monitoring system
3. Dependency resolution and checks against a DB (our solution _already_ supports Cargo.lock files.
Rust (and Go, C++, etc.) only prevent option 1. Thanks to bundlers it also can miss it in JS, or shaded fatjars it can also fail kn Java too. So it's not a new challenge
I think this attitude is one of the main issues for this kind of incidents in larger organisations. I'm not sure why you would be willing to lay blame at your colleagues for this. Most of the time, development teams don't tend to like some governance over the code and the dependencies they are pulling in. Not just because they know of course what they are doing, but also because they are under pressure to deliver features. That is what matters for business. Also, convincing management that budget is needed for correct tooling to track all stuff deployed; is also not as straight forward as you seem to suggest.
In this case, the problem goes even beyond just the code of your own dev teams. This is embedded in countless software packages deployed all over your organisation. Same here, people want to buy and use whatever they want. And all processes to keep some form of control over it, are mostly seen as overhead.
And I'm sure they are lots of "security" people who are just producing documents and policies which are complete detached from reality. But developers who consider security completely as somebody else his responsibility, are a problem as well.
The only good think I see coming from this mess, is that security teams probably will get the means to try to get more control and insight over this. For the coming weeks/months at least. After that, everyone in management of dev will be forgotten about it. But something tells me the security people who are working on this right now, won't.
no, best practices include checking in your Cargo lock file
They found a way to make several catastrophic mistakes while writing a logging solution (one of the most trivial kinds of software) in a memory-safe language.
It's surprising they manage to breathe properly and thus survive.
Githubs of the world could just gate downloads, pull requests etc. behind a payment to see what is the real valuation of open source software; I imagine it'd mostly settle around $0 excluding couple of big projects.
Saying things like "No amount of money is going to impart good taste and best practices onto these projects" is insulting the people who wrote it with no real basis. It's not helpful and it makes you (and some other people in the thread) sound like a jerk.
If that's true, do you think that having money might possibly be helpful to that effort?
Use the standard java logger and move on, it's that easy.
If you don't want to work on it anymore that's OK too. Open source libraries are generally not held to particularly stringent requirements by the vast majority of people - perhaps accepting an obvious patch in a critical situation, at most.
You obviously can pay people. They're called employees. The thing about employees is that your money translates to business value, which is much broader than a single library. This means that when a single library is done the developer doesn't have to think "ok but what's next for this library" and start adding more code. They can stop and move on to other tasks.
Paying for a single library is nice. I support multiple open source developers because I appreciate their work. I recommend everyone does so.
I don't at all believe it's going to help security though.
[0] https://www.dimastr.com/redemption/home.htm
[1] https://www.devexpress.com/
[2] https://www.aspose.com/
But just be be sure, we should ask legal!
If we can't find a way to pay Google, there is no hope for some random developer in Nebraska.
Nebraska is a reference to an XKCD Comic (https://xkcd.com/2347/). The idea is that there are nameless people all over the world contributing to critical open source infrastructure.
Back to the point.
Running Google Maps on one of my low traffic sites, would generate at most, several dollars a month in charges. It would be a rounding error in our budget, and we can't pay it. Our purchasing rules are not flexible enough to work with google.
Paying Google is hard enough, an independent oss developer with no company would be impossible. I would get laughed at if I suggested throwing a few bucks at a developers Patreon.
Security teams are stuck with securing the tire fire; they didn't choose the library or platform. If anyone should be advocating for supporting open source, it's the developers who benefit by using open source libraries.
Log4j2 partially exists because of pushback on adding features to v1
http://logback.qos.ch/
Honestly the biggest problem of package managers is finding a reputable package that one can trust.
The best one can do to aid this situation is reviewing, vetting and warning of which packages that can be trusted and not. Automatic scanners to find code smells and vulnerabilities.
“Security rating: 2/10, This package seems to use JNDI loading, are you sure you want to continue (y/n).”
Logging as it's understood today should ship with most STD libraries. Things like logback an log4j really should just eventually be rolled into the std.
---- This wasn’t a process failure, the vendor did everything right. Mozilla has a mature, world-class security team. They pioneered bug bounties, invest in memory safety, fuzzing and test coverage.
NSS was one of the very first projects included with oss-fuzz, it was officially supported since at least October 2014. Mozilla also fuzz NSS themselves with libFuzzer, and have contributed their own mutator collection and distilled coverage corpus. There is an extensive testsuite, and nightly ASAN builds.
I'm generally skeptical of static analysis, but this seems like a simple missing bounds check that should be easy to find. Coverity has been monitoring NSS since at least December 2008, and also appears to have failed to discover this.
Until 2015, Google Chrome used NSS, and maintained their own testsuite and fuzzing infrastructure independent of Mozilla. Today, Chrome platforms use BoringSSL, but the NSS port is still maintained.
Did Mozilla have good test coverage for the vulnerable areas? YES.
Did Mozilla/chrome/oss-fuzz have relevant inputs in their fuzz corpus? YES.
Is there a mutator capable of extending ASN1_ITEMs? YES.
Is this an intra-object overflow, or other form of corruption that ASAN would have difficulty detecting? NO, it's a textbook buffer overflow that ASAN can easily detect. ----
[1]: https://googleprojectzero.blogspot.com/2021/12/this-shouldnt...
OTO, people write software for free and licence it in a way that allows above companies to use it without giving them a single penny. And then they complain about not getting a penny whilst somebody else makes billions.
This whole thing is just incomprehensible to me.
Once you get to that stage there can be a follow up to gather more information, and that follow up then usually results in some advice or no further action. In rare cases - typically the ones where gross negligence or willful transgression of the rules was established - there will be a fine and if it is a repeat occurrence that fine can be quite substantial.
Also you don't have to report 'potential leaks', only actual leaks.
So I think that in the case of your hypothetical end-point the logging isn't a GDPR requirement, but if your endpoint ends up leaking data then the log can help you to establish if and if so how much data was exfiltrated. But that does not mean that the GDPR requires you to have logs, though, having access logs for your endpoints is a fairly standard thing and not having them is going to raise a few eyebrows, especially if you are also reporting a breach.
Where there are logging requirements: data retention laws, SOX, fintech, tax regulation, AML.
Then there's the user-agent header, which also can be pretty valuable to debugging and other operations. Although this one is easier not to log.
In my current logs, I'm seeing attempts to compromise log4j vulnerability in both URL query string and user-agent. (Which of course I would not know about if I weren't logging them). This particular application has no Java involved, so is not vulnerable.
The fundamental problem is that, pretty surprisingly, log4j was running the same parsing and logic on user-entered strings as they did format strings. That's essentially the root cause of all this - log4j shouldn't be attempting to parse this data at all.
So many severe vulnerabilities are due to the complexities of parsing potentially malicious user input - just look at all the severe iMessage bugs. While many other apps do have a fundamental requirement to parse this input, there is no reason a logging library needs to do it.
yes, but there can be (and have been) vulnerabilities in terminal emulators, terminal multiplexers, text editors, databases, web renderers and all manner of tools that can be used to store the contents of or view that file. your argument makes sense that the logging library should not be attempting to parse the log strings, but think about all the other millions of lines of code that could potentially try to parse that data as well. patch log4j one day and the next you find a bug in some colorizer javascript library or in the sixel support for the terminal emulator. people add parsing everywhere.
i think ideally that all user supplied data should be sanitized in the strictest possible manner before anything is done with it, including logging.
web browsers and all of their components get audited. because they process potentially malicious data as their core function, they see A LOT of attention in terms of hardening.
the point that i'm trying to make is: the amount of software that doesn't immediately and apparently touch potentially malicious data is absolutely enormous and the number of paths that data can take to touch that software is even more enormous. yes, one can put forth good principles for "building codes" for software and yes, one should... but there's an awful lot of software out there that is not up to code (where the craft is so young that the idea of a code isn't even close to being finalized) and it won't be for a very long time.
that being the actual on the ground situation, what is one simple thing that individual developers can do to help prevent catastrophes? always sanitize inputs. moreover, it's much easier to find and audit every place where data is accepted from untrusted sources than it is to find every place where data may be parsed.
Oh come on, how would you even do that if you don’t know how the data is going to be interpreted by parsers that nobody knows they’re using? WAFs have been trying to do this since forever, but it’s always been security theater that separates gullible companies from their money and randomly blocks users that use “special” characters like slash or quotation marks.
I guess you could base64 encode everything, but that wouldn’t save you from the terminal bugs you mentioned. At some point you’re gonna have to undo the “sanitizing” and look at the text.
The latter option represents a liability, which in this case (as with others) has shown can be a tremendous risk. Is the time saving really worth it, at the cost of risking disasters like this?
I get that the tradeoff it is worth it for complicated things (e.g. crypto libraries). But logging, really?
Software development culture today is too quick to adopt a huge tree of dependencies of unknown quality, rather than thinking about how to minimize dependencies to only those truly necessary. The leftpad fiasco was but an extreme example of this, but I see it all the time, and it seems probable that there are hundreds (maybe even thousands) of similarly severe problems out there in widely used dependencies that we just don't know about yet.
yea literally any library/dependency can introduce risk, including stuff coded internally for those purposes
There is also lots of stuff you might want a logging library to do that while not very hard would be annoying to do yourself for every project like log file rotation, switching log level while the application is running, optimizations (like in c/c++/rust you might compile for INFO level and drop all the DEBUG and TRACE level logging from the binary), different log levels for different parts of the app (including dependencies), etc
Transitive dependencies for one thing. I rather enjoy having my DB driver, connection pool, kafka client, and web framework all emit their own logs, which is not possible if I wrote my own logging library.
> I get that the tradeoff it is worth it for complicated things (e.g. crypto libraries). But logging, really?
Logging at scale is more complicated than people give it credit for. Personally I've had my services suffer noticeable performance degredations due to poor logging framework config. We've ended up with a fairly complicated logging config that allows us to carefully balance collecting as much developer useful information as possible against the need to drop excessive logs during periods of heavy activity. Oh, and it's gotta be converted into JSON for aggregation into our observability tools, and decorated appropriately so that our logs can be correlated with our application traces, because that is incredibly useful during incidents.
Oh, and it all needs to work no matter what threading configuration you throw at it. That matters a lot too.
Do we need all those things? Arguably no, but then again they are helpful tools to have. Without them, the team would have a harder time developing on our systems. You have to understand the use case that people need to fulfill before dismissing the entire thing as unnecessary and pointless.
> Software development culture today is too quick to adopt a huge tree of dependencies of unknown quality, rather than thinking about how to minimize dependencies to only those truly necessary. The leftpad fiasco was but an extreme example of this, but I see it all the time, and it seems probable that there are hundreds (maybe even thousands) of similarly severe problems out there in widely used dependencies that we just don't know about yet.
I see this argument a lot, and I think it's a great example of "grass is greener" syndrome. People don't think through the practical consequences of drastically reducing their library usage. I personally worked on a project that had basically no external libraries due to the language choice, and we wasted inordinate amounts of time trying to chase down bugs we wrote in things like our database library, localization library, and yes, our logging library. Looking back I can say with very strong confidence that what we were doing was a massive waste of the company's resources, and rewriting it into Java or C# would have been the right call, even with the complexities those ecosystems bring to the table.
Yes, using external libraries comes with tradeoffs. Fixing bugs like this is one unpleasant possibility. But let's be clear that the alternative is a massive drop in developer productivity to write basic functionality that already is implemented elsewhere. Having been on both sides of this equation, I can confidently say that you'll spend way less time fixing security issues as they arise than you will reinventing the metaphorical wheel.
(I do agree that leftpad is an example of going way too far with libraries, but I think the library culture for NPM and the Java ecosystem are quite different).
Logging facades like slf4j help a lot, but then you still need to implement the required interfaces. For something like the specific Log4J CVE, things would be significantly worse if it weren't centralized in a library. They have a bug in deserialization when talking to LDAP servers, allowing a compromised LDAP server to execute code remotely. It's likely that a subset of libraries would likely have rolled this functionality independently, and then you have many libraries that all need to be patched and updated instead of one.
In terms of features, one that comes to mind log rotation.
I dunno, I suppose all together I’d answer your third paragraph to the affirmative — I’m more productive and the libraries improve significantly within crucibles like this.
The larger issue is in javas dependency system are built with dependencies on these libraries, so if there is no commons logging (or bridge, ala slf4j), then the program won't run.
There are a few other techinical traps with the jvm, like using system.out logging can cause issues with threaded code, or having one giant static logger being imported all over the place conveniently (this is before inversion-of-control became the dominant paradigm) had a few shortcomings, especially as programs grew. Ultimately, though, if you were going to use some third party libraries or app servers, you were almost certainly going to get pulled into the commons-logging vortex, which means you're now configuring that AND your homespun logger.
There is also the case for things like SOLR, where it might be just being used out of the box, but the distribution includes the affected JARs, even if you're just posting and retrieving documents with a python script doing print-logging.
Honestly, I didn't even know log4j2 was a thing, I've moved as much over to slf4j/logback as I could ages ago because of the madness of the JDK logging ecosystem.
When Java finally added a logging library, they added one that no-one had ever heard of before, and which had fundamental flaws. That's why no-one uses it. At the time, log4j existed and was in wide use, but was not represented on the committee.
The author of log4j eventually decided to go a different way and created slf4j / logback. This offering is compatible with log4j and considerably simpler in many ways.
Having not been in the Java world for many years, I was surprised log4j was still in widespread use. But maybe I shouldn't be surprised - whenever we have to interact with Java teams, we always have to tamp down on the tendency towards complexity.
It's Java; the ecosystem is, politely, rather conservative and slow moving.
Note: In case someone is tempted to say that it is trivial to plug slf4j over logback, it is for greenfield work. Replacing an existing logging library, or the use of multiple logging libraries in favor of slf4j over one lib is a confusing matter, mostly because the error thrown at runtime are misleading.
(1) Coordinating with dependencies requires at a minimum a common API (which, in a statically typed language, means a library providing at least the interface).
(2) Configurability to log to different destinations and with different formats, which you eventually tend to want on major or long-lived projects, can be fiddly, as can the specifics of different formats and targets. Each of the bits is simple, but added up it's a lot over time, and reinventing the wheel, the axle, the tire, and the air pump for it just doesn't make a lot of sense. Especially given the need for a library for #1, it's best to get the core functionality together with it, and the peripheral pieces either together or as plugins, and only solving unique problems, if you have them.
Sir, you might have just extended all of our careers by at least a decade. We salute you.
I agree that developers are too quick to put themselves at the mercy of 3rd parties, but it's a risk-dollar trade off we'll continue to make because reinventing the wheel each time would sap our margins.
A lot of these libs separate logging into layouts and parameters. If you log input parameters as layouts. You might break the logging. I have seen this at every company I worked with.
The only really surprising thing about log4j is the remote code execution.
Log4J has been around 20 years. Log4J inspired and was lifted into the JDK. JDK logging is essentially an inspired copy of Log4J. The Sun coders were no better or worse than the Apache coders. I use JDK logging personally simply because it's one less dependency. JDK logging hasn't change since JDK 1.4, and is weaker than, well, pretty much anything else. But, with a simple wrapper I've used for 15 years, it does most everything I want from a logger.
That said.
Log4J has been, and is still, a boon to the Java community. Arguably, Log4J is the root of a tree of vast array of logging frameworks, across languages. Java server developers essentially live and die by their logs. It's routine for developers in dark rooms with screen lit faces to pouring through logs with endless stack traces. Thank heavens for Java stack traces.
Logging is part and parcel to the Java server side experience, and even the client side, and we can place much of that on the shoulders of giants like Log4J because it made logging easy and set the stage. It's so helpful, so useful, so flexible (obviously, perhaps, a bit too flexible), and so powerful.
Because Log4J inspired the other loggers like the JDK Logger, we have logging shims. Shims like Commons Logging, that act as intermediaries that can have adapters written so that we can use other logging frameworks. Log4J was one of the first, was, and is still, dominant in the community, but it's not alone, and thus a bad choice for things like libraries. Instead, those choose the shims like Commons Logging that developers can use to configure to route through Log4J or JDK logging or any of the others. That said, even programs that use Log4J directly can be routed to other loggers.
This is all entrenched. It's part of the flavor of server side Java, configuring the different logging shims to write to your logger of choice on your system. Just the way it is. But, that's what happens when you don't live in a mono-culture. Feature, not a bug, and it helps empower the vast array of software that millions of developers and applications rely on everyday. Nobody designed it this way. It didn't start this way, it just evolved this way. It's a very "Java" thing.
I watch Stack Overflow questions and 90+% of the time when someone asks "How do I" what they mean is "What library do I need", not "How can I write this". This is the sign of the times, and Java is not alone. The beauty of Java is that it made this kind of sharing REALLY easy. REALLY REALLY easy.
Apache has a solid reputation for good projects and good code and good stewardship. It's not a back alley transaction to grab an Apache Java jar file and shove it in your project. Is it all perfect? No, but what is?
The problem with rolling your own is that, for a non trivial technology like a producionised logging framework or some encryption library, you're highly likely to write a massive security hole yourself. The great thing about battle-tested open source software, used by 1000s, is that years of real world use, development, bug fixes and general scrutiny from a large number of people makes for a high level of reliability normally.
Obviously here we have a problem with this approach but avoiding third party dependencies carries its own risks and obvious costs too. Let's focus on how reliable so many of these libraries seem to be and how rare problems of this scale are.
I think a bigger problem here is the fact that so many people are using so few libraries. Maybe we need more open source libraries to counter single points of failure. As with how we deploy our code on the same clouds and so on, single points of failure abound these days. Things fail however well they're engineered and therefore we need diversification of everything.
Rolling your own logging library to reimplement it is not trivial, even if you ignore the more esoteric output formats.
Sure, in retrospect they went too far with some of the dynamic logging features in log4j2, but it is no exaggeration to say that log4j revolutionized logging in the Java world when it first came out, and heavily shaped later solutions like logback and java.util.logging.
Not to mention a wide range of destinations, too. Want to log to a database table? You’re covered. What about log messages to a chat server (xmpp, slack) or email but only fatal errors? You’re covered.
Now roll your own logging system that supports those destinations safely and get back to me to do pen testing. I bet we’ll find some vulns in your code.
Most of us have log4j in our system because it's used by some various dependency of one type or another, and up until now it has generally just worked.
> It’s dynamically configurable
It's a logging library. You don't reconfigure logging, you just make another logger and use it. And by the way, you take the configurations from your global configuration system, "dynamic configurable" is a feature of the configuration system, not of the logging.
> supports a wide range of output formats and destinations
Yeah, like network aware templates. (But the destinations are a feature, multiple destinations are one of the few things a logging library should support.)
Despite what people claim, the Java culture of loving complexity never did go away.
Prior company had a shared plugin that let you turn up the logging for a given logger remotely for 30 minutes (it self reset). Extremely helpful for incident debugging.
Now we can have an interesting discussion about whether that capability is worth the complexity, but that’s a very different discussion than unilaterally declaring that other people’s use cases just don’t exist.
> Despite what people claim, the Java culture of loving complexity never did go away.
Do you develop in Java? Because I do, and your takes don’t match my daily experience one bit. My experience going from Spring 3 to Spring Boot 2 is in fact one of reducing complexity (goodbye XML bean configuration!) and to more “it just works” situations than before.
And you really believe this should be a feature of you logging library, and not of you configuration system?
> How much log data do you think cloudwatch and the other majors filter per hour?
How much do you think the major providers pay for that infrastructure per hour, and are you willing to shoulder that cost too?
(Hint: log ingestion is $0.50 per gigabyte in cloudwatch. So a terabyte of logs an hour is $512/hr)
Only if the emissions of said logs is doing more than stdout or fs writing and it’s not done in another thread.
> Log emission can often have a negative impact on performance, especially since peak logs and peak traffic tend to coincide. I’ve seen P95 latencies suffer just because of logs.
Yes and you need to fix that terabyte per hour garbage as that is well beyond typical for a single service. We’re also talking about java so 7000 page stack traces is where i’d start…
The only way to make a "reconfigurable" logging system is by having a stable facade that restarts everything behind it. But that extra layer does not belong on a logging system, there are plenty of other things you need to make "reconfigurable", and now you are adding an extra layer into each of them, instead of having only one on your main entry point.
Pushing that feature down into the logging system is the choice that maximizes complexity and minimizes functionality.
Also, I noticed you didn’t answer my question about whether you’re a Java developer.
I was once, but haven't used it professionally for a while. It's on my "I'd rather not" list, but not strongly so.
Oh, and now that you described it with so many details, the attachment to complexity does look more like addiction than love.
This is why I ask. Java has changed a lot since ~1.8, and it is super common to see ex or non java developers making grandiose and sweeping statements about a community and language that they are no longer part of. Having developed in Java on and off since ~1.7 and Spring 3, I can certainly say that I have seen a strong movement towards less complexity and more convention. It's not perfect by any stretch, but I think saying that the space is "in love with complexity" is incorrect.
You of course don't have to go and re-use Java on my suggestion. We all have our preferences, and that's ok! But I would generally recommend against such sweeping and dismissive general statements about a culture that you are no longer part of. Things change, and your experience back then is probably not representative of things today.
> Oh, and now that you described it with so many details, the attachment to complexity does look more like addiction than love.
There's a fine line between disagreement and insult, and you are over it.
That said, I do agree that there is a problem with large dependency trees. It seems building a simple hello-world app with Maven ends up downloading about 1/2 the internet. :)
Switching over merely involved changing the jar files we used, and redoing the config.
And that's a point in time audit. To maintain that value we'd have to redo the audit periodically.
It's just not gonna happen.
It was especially difficult for us because we’d shipped so much code that used the library, and replacing the library was unthinkable.
It's just not as simple as "security audit finds all the vulnerabilities, then you fix them." You invest X in the review, you get the results that X/(hourly rate) finds. This is a lot of software with a ton of configurability-- that's a lot of variations to review and test.
Now that someone found the first lump of gold and gave it away, there are thousands of eyes searching for the next one. These recent findings are all abuses of this same chain of functionality, just along different sets of settings. In another month we might have half-a dozen more of varying severity and scope. That _still_ won't prove that the overall library is then safe, but we will probably have a a little more confidence in this particular bit of crazy template formatting flexibility. Maybe not as much as we had had three weeks ago, but more than we do now.
We use a product that uses JCE. The log4j vulnerabilities worry me about the security of software in Java ecosystem in general.
Log4j is not part of the JDK and is an open source project with only few regular contributors. There is really no reason to infer any correlation to other parts of the Java ecosystem.
https://issues.apache.org/jira/browse/LOG4J2-3230
It's pretty interesting technically, and it's not really affecting 2.16.0 much except in weird edge-cases.
I do not know about your security teams, but my security teams are now sitting 9 days straight trying to stop that shitshow. With teams complaining that they have to patch instead of doing Great Things.
After that these security teams will be back to invisible work and forgotten again. Until the next issue.
If your company has a truly different approach to security, tell it here - you may get really talented people knocking in because they are fed up with the security theater at their current job.
Bugs come in bunches. If you have fixed just one or two, more lurk right there. And, anyplace else that coder worked. The more you have found, the more remain to be found. Look at other places that coder worked that week. Or year.
But looking in your own code is probably next best. Ask yourself: if there is a bug somewhere in my code, where would it be? You may be surprised at how immediately the answer comes to mind. Look there.
A big subset of mistakes people make when writing software are the result of people not having the same understanding of a concept, policy or practice.
Even a good writer will make mistakes, otherwise we wouldn’t need proofreaders.
No, it’s not.
Why are people still using JDNI? It is frail enough to have logs depend on a dns lookup, I can't imagine depending on a LDAP lookup. This is not such a fundamental issue as the above but it is pretty dangerous.
Oh yes and of course one of the worse and least pythonic package in the Python standard library was "inspired" by the log4j monstrosity
Because "java:comp/env" is the way for a J2EE web application to get values or objects from the configuration outside its container. Yes, nobody actually uses the remote parts of JNDI anymore, but the local variant (within the same process) is still in use.
And that's what led to this vulnerability: someone wanted to be able to get these local values within the logging configuration, which is a valid use case. Not realizing that this opened the door to remote values and objects.
Hugs for all of you. This can't be fun.
But to be clear: nested/recursive template expansion and expansion of user provided strings were never by design, correct, and the removal shouldn’t just be by configuration or considered breaking - it should simply be corrected (just like the jndi should be removed and not even be optionally possible)?
Okay, more seriously... this is why I am currently staying away from Rust. Because sometimes I don't even know if the random number generation crate I imported for my banking app came from Rust team or some random dude in zanzibar.
(Sorry, I know HN loves Rust, so do I. But I currently that the cargo/crate system is a bit too easy to use and misuse).
(Unless some hobbits have 3 breakfasts?)
Fortunately our log4j was too old and JVM too young to be vulnerable to the first exploit, but it seemed best to rip out log4j and replace it with something simpler anyway.
java.util.logging seemed the obvious choice, but the API is different, so would have been a lot of s/debug/fine and so on.
In the end, we:
- wrote a new class with the same 'api' as the part of log4j that we use
- globally replaced 'import org.apache.log4j.Logger' with our new class
- fixed any compile errors by adding new methods to our class
- wrote a test to make sure it rolled correctly and could handle lots of threads etc.
Here's the code if you want to do something similar (don't know how long it will be accessible from there): <https://ideone.com/XKg5M9>