> When it was first revealed in early December 2021, the Log4Shell bug was described as one of the most severe security vulnerabilities ever.
> The Apache Software Foundation, which maintains the open-source tool, quickly released a patch...
Apache horribly mismanaged this and did not release a patch until it was already widely known and being exploited in the wild. They also messed up and had to release several subsequent patches to actually fix the vulnerability. This greatly contributed to "[the] mad scramble to plug the gaps".
Saying that "Apache mismanaged" it when talking about addressing a vuln is quite awkward. Probably fueled by lack of understanding of what the foundation does. The fact that every project have to have apache stamped in their name doesn't help either.
So is saying "The Apache Software Foundation... quickly released a patch", but the average person doesn't care about the inner workings of the foundation because, as you mentioned, everything is stamped with "Apache".
It may be technically incorrect, but most people wouldn't understand if you got into the minutiae.
They might be able to understand, but they wouldn't care. If there's a bug in Microsoft Office there's no discussion which engineer was responsible. Microsoft put their name on it and are thus first recipient of praise and blame alike. Apache Log4J2 is no different.
Apache provides infrastructure and some general organizational rules for projects, but the people who actually maintain any given Apache project are largly independent and autonomous, they are not "Apache".
You're right, but this misses the point of my comment.
It's a distinction without a difference for most people, and trying to clarify the structure of the foundation to executives and senior leaders (technical and non-technical) only makes their perception of any Apache-branded or open source projects worse.
> My point is one shouldn't bring up Apache in the first place when talking about a project like Log4j. It's similarly irrelevant as "hosted on GitHub".
That isn't comparable.
- the name of the project is "Apache Log4J"
- the tag-line is "*Apache Log4j* 2 is an upgrade to Log4j that provides significant improvements over its predecessor..."
- it's hosted under the Apache organization
- the documentation, release notes, and bug tracker is under the apache.org domain.
- the artifacts are published under the org.apache namespace
- etc.
No competent person in the industry is going to think that GitHub is associated with the projects it hosts. There are also hundreds of projects stewarded by Eclipse, Red Hat, and other similar organizations that avoid this confusion by using sensible branding.
What you're saying is more equivalent to "no reasonable person would think that Vitamin Water contains vitamins and is healthy".
The fact that Log4j is a project under the Apache foundation has practically zero bearing regarding Log4Shell. What relevance do you think Apache has here? Apache isn't a software development organization, it's a dev community umbrella organization. And therefore I think it's not useful to refer to the product as an Apache product.
> The fact that Log4j is a project under the Apache foundation has practically zero bearing regarding Log4Shell. What relevance do you think Apache has here? Apache isn't a software development organization, it's a dev community umbrella organization.
FWIW, I agree that the branding on the project pages can be misleading. My point is, we as developers who know what Apache is, shouldn't carry that misleading branding over when discussing such a project. In most contexts it doesn't make sense to talk about such a project as an Apache project, as if Apache was the entity managing the content of that project, very much like it doesn't make sense to talk about it as a GitHub project in that capacity.
That doesn't make any sense. You (and the OT) are effectively misleading people to associate "Apache" with the folks that screwed it up with log4shell.
If the concern is that "most people" will have a wrong perception, I don't see how talking losely about Apache will help. Apache is irrelevant in this context.
Is your goal to make "most people" steer away from anything Apache?
It was a complete misfeature from start to finish.
They put in an "attempt to evaluate strings" feature. Literally as bad as someone doing a `console.log(eval(input))`.
Not only is it that bad, but also it's something with just a tiny bit of inconvenience you can get with either String.format or the likes of slf4j's formatting ( `logger.info("{} value", foo.get());` )
They took the worst case for an injection attack and applied it everywhere.
Not to mention that `log.info(eval(input))` made it effectively impossible to determine if you were compromised. If you weren't vulnerable you'd see "eval(input)" in your log, however, if you were vulnerable you wouldn't see anything.
In such cases I'd normally apply Hanlon's razor ( https://en.wikipedia.org/wiki/Hanlon%27s_razor ) but this security gap is really huge and rather obvious if you have an informed look at the code.
So this is one of the cases where I'd personally would not be surprised if in the far future someone admits on their death bed that malice was involved.
This particular bug has manifested in many places now, and in all those cases I find it plausible that it was a confluence of several independent teams (as in, not even working on the same code) adding features until, whoops, it turns out the combination is catastrophic. You find it obvious after the fact, but I don't think it is always obvious at the time.
It's really easy for someone to see a feature like "interpolate strings better than ever" in their string library, and say to themselves, "Hey, that sounds good, let's turn it on!" without realizing that "better than ever" means "with arbitrary lookup from a string into a class of some sort", and then chain a few things together and hey presto you've got an arbitrary code execution vulnerability.
I personally think allowing arbitrary string -> class lookup for all classes in the runtime is automatically a security code smell at best if not outright antipattern, but I don't think this is at all well understood yet. (While the general pattern is hard to avoid, you should have to register all classes/structs/values/whatever you want to automatically load from somehow. This prevents things like "I can use the OS object to execute arbitrary shell code" from automatically and unexpectedly creeping in.) Dynamic languages are all but based on such a capability, with their "eval" support. It's probably still going to be decades before this is generally understood "programmer wisdom", partially because the particular confluence of features that enables this bug is not common. It's just that when it does happen, it's catastrophic. But it's not actually that common.
>I personally think allowing arbitrary string -> class lookup for all classes in the runtime is automatically a security code smell at best if not outright antipattern, but I don't think this is at all well understood yet.
Spot on. Case in point: the maintainer of snakeyaml furiously arguing that a similar vulnerability in his code is the fault of absolutely everything (client code, any code that can be used as a deserialization gadget, "low quality tooling") except his code: https://bitbucket.org/snakeyaml/snakeyaml/issues/561/cve-202...
I’m so confused about how many times it’s reiterated that _no one_ is parsing untrusted YAML with the library. It’s so unlikely as to be laughable.
Then later when someone says they are, he doubles down saying that parsing arbitrary YAML provided from people on the internet isn’t parsing “untrusted” YAML because they’re authenticated. Apparently it’s reasonable that anyone that signs up for a SaaS service gets code execution privileges on the prod infrastructure?
I’m pretty sure that thread had me shaking my head hard enough to give me a concussion.
The whole thing stems from developers wanting "easy" to use logger and mixing "a formatting string" with "actual thing being formatted". The vulnerability is bad enough on its own but having API of library allowing to pass one argument that is then treated as format string practically ensues someone will do on accident
if library have interface being
.log(format,value...)
.log(format)
It is kinda easy to assume second invocation would not apply formatting and just act as plain value, because it makes zero sense to have format string with no inputs, yet that's what Log4J does.
Separating it for "log" and "log with formatting" calls helps a bunch, at the very least makes easier to search for potentially vulnerable code.
Absolutely. I saw a lot of bad advice from people who thought it was only exploitable when misusing the API, similar to SQL injection, and I can see why they would think that. Unfortunately, they were wrong.
> They put in an "attempt to evaluate strings" feature.
You have to evaluate to strings eventually though. A shell console/command prompt has no concept of SQL-style bind-variables. If you are logging "username: {}" that eventually has to be evaluated to a string sometime.
> It was a complete misfeature from start to finish.
No, there was a legitimate demand for this at the start. If you want to log instance-specific information (like JVM container name) the facility provided for that in Java is JNDI. And obviously it would be nice to have "what's the instance that produced this particular message" in the error logs, that's self-evidently helpful and desirable.
The problem is that JNDI is a foot-howitzer, it can also be used to load and execute arbitrary code from remote locations among many other things. Similar to how binary deserialization from remote sources basically cannot be used safely in Java (because a deserialized object can be in an "illegal" logical state and bypasses constructors and other assertions, an object type cannot be known before it is deserialized (so it can be of any class-type), and deserializers can run arbitrary attacker-controlled code before it hits any validation layer) this is not immediately apparent just from the fact that you are using JNDI, but, it's safe because we're not going to call attacker-controlled remote code through JNDI, right? Why would we attack ourselves?
And then the final piece is that (as you note) they fucked up the order of evaluation. JNDI is evaluated at the very end of the stringification process, meaning an attacker can include their own JNDI calls in the string and then it will be evaluated like developer-controlled JNDI. That is the fundamental problem here - this is bog-standard SQL-injection-attack style stuff, you should never be able to escape an injected/bound variable, but JNDI evaluation takes place after stringification.
All that really needed to be fixed was the last part. But honestly the dogma of the Java world has moved away from this whole approach in the first place - people were doing multi-container-per-JVM back in 2000s when memory was scarce, so you wanted to know that instance information so you could tell what instance was generating a message. But nowadays people do 1-container-per-JVM (and many JVMs scaled in a cluster) and you can handle that with runtime variables/etc that let the instance know where it is from external oracles. So there is no need to touch that in the first place.
It's all very much functionality that was demanded by real-world users... in the early 2000s. But Java refuses to clear some of the cruft from their APIs because it would produce (gasp) breaking changes in legacy applications... sort of the exact opposite of the Python 3 mindset. And JNDI is one of those interfaces that was just way too powerful when it was created. As mentioned, binary deserialization is another... you can't really use that in almost any scenario because the attack surface is just too large.
(JNDI plus deserialization is a good one too, up until a couple years ago you could specify the class/deserializer as living on a attacker-controlled remote source, because JNDI can do anything!)
We dropped an entire feature from our product because of this crap. We had a sidecar java process that handled certain PDF conversion tasks, but the lack of in-house java wizardry meant that we couldn't trust that small binary with regard to a spiraling vulnerability situation. Our customers were very nervous at the time.
Making something work in Java is easy. Proving it is secure is not.
It is almost certain that we over-reacted and shot ourselves in the foot with this, but our customers were happy with the way in which we responded. Better safe than sorry was the theme throughout.
Perhaps — explaining that to people in procurement and convincing them is months of work. Corporate procurement doesn't actually care about the risk in many cases, they care about having someone to blame.
Is there any interest in bringing back that feature? No wizard here, but I dealt with Log4shell for our product. Reach out if you'd like someone to take a look.
> but the lack of in-house java wizardry meant that we couldn't trust that small binary with regard to a spiraling vulnerability situation.
I've never experienced the problem of "nobody in my company knows Java well enough to trace the dependencies of a JAR". Do you all works in Rust and Javascript or something?
I'm not a cybersecurity expert, but i follow security news as i'm often working alongside secops teams.
When i read here about Log4J, is was a real "Holy shit moment". It was so easy to use, it could've been a nebula exercise.
I contacted my PO asap, we found out that we had no inventory of what our VM were and what java versions were installed (worked at a bank btw). We then contacted the secops team, who just found out, and triggered all alarms. It was all hands on deck time. Quite a fun two weeks tbh.
Product Owner (from Scrum)? Likely an org where dev teams must get permission from product people, who own the backlog. Less objectionably it might be the leader of a technical product.
It means "product Owner". Basically someone who know the inside of the applications you're presenting, understand the tech debt and your release cycles, and discuss with the users about new features and potential issues.
It's basically the same managment level as the tech lead/lead dev.
I understand this sound like scrum/agile gibberish, but honestly this was like having a technical manager/salesman who could also help you debug stuff, probably the most usefull post of "agile at scale"
(Note: I'm the person that coined the term "Log4Shell")
You may be surprised when I tell you what the Apache Software foundation's yearly budget is. You'd think for software that is used by practically every Fortune 500 company and most governments, it would be something reasonable. Maybe a few hundred million dollars a year to pay for a reasonable full-time staff, right?
It turns out... it's about $2 million a year. (Wikipedia[0])
This helps explain to me why the devs of Log4j directly uploaded the file "JNDIExploit.java" (the POC) to GitHub while they were patching. (Here is a full analysis about what happened[1].)
They're not security people. They're volunteers working on this in addition to their full-time job.
What kind of brave soul wants to trudge through and maintain log4j in their spare time for zero compensation? I appreciate the people that are capable of doing that, but I think they are rare!
This whole entire vulnerability was eye opening for everybody and I have actually spent the last year building tooling on GitHub to help fix the problems that Log4Shell exposed.
If you have 2 seconds to try that out or just Star the repo[2], it would be very helpful!
Absolutely, but if I had to rank Open Source projects by "enjoyment level to work on", I would rank a 10+ year old Java project primarily used by enterprise companies quite low on my list!
Compare that to something like Bun.js[0] which is "sexy" and written in a "cool" programming language (Zig). Or Wasp[1] which is built with Haskell and is trying to define a new programming language designed to make common dev patterns less painful.
Those projects are naturally going to soak up smart people that have extra energy to share because they hate their day job but need to pay their bills. (imo)
Who is left that wants to bang their head against a legacy codebase like Log4j? Maybe somebody that feels there is "clout" to be had from it? (Spitballing here, I honestly don't know!)
I imagine that someone who is good at that style of Java might enjoy the challenge of working on it, particularly if they are already familiar with the codebase.
I bet a big chunk of them have contracts that prevent them from contributing to open source. Atleast without going through a painful vetting process that drains all the enjoyment from it.
Usually that does more than drain enjoyment, but tarpits it to the point it won’t happen - and is only to be a trap for employees so they think it’s an option.
> What kind of brave soul wants to trudge through and maintain log4j in their spare time for zero compensation?
It's not clear to me as an outsider what exactly the Apache foundation is doing for these projects. It feels like Apache is willing to accept code donations from anyone and is willing to attach the foundation's name to code that isn't widely used, actively maintained, or may just be abandonware.
I have soooo much more confidence in CNCF projects. The conditions for graduating as a CNCF project include criteria like that your project must be in use by multiple real companies, have maintainers who are (paid) employees of multiple different companies, and get a professional security audit.
>It feels like Apache is willing to accept code donations from anyone and is willing to attach the foundation's name to code that isn't widely used, actively maintained, or may just be abandonware
That's why I'm allergic to Apache software. A lot of it is overengineered, insecure, legacy abandonware.
> It feels like Apache is willing to accept code donations from anyone and is willing to attach the foundation's name to code that isn't widely used, actively maintained, or may just be abandonware.
That’s incorrect. Projects need to report quarterly and need a Project Management Committee of at least three people, or they are retired. Retired projects may not make releases.
(Source: past ASF board member, who used to review those reports each month.)
There are a fair number of retired projects, and others that may become retired within the near-to-medium term. The ASF has been around for a while, and every software project has a life cycle. Those are still associated with the ASF brand because Google, whatcha gonna do? An explicit retirement policy overseen by a board is still superior to how the vast majority of open source projects approach end-of-life.
In theory. Open Office shows that the process of retiring semi-abandoned projects leaves a lot to be desired.
The project has few, if any, volunteers, and there are security problems known to be actively exploited, yet the ASF is not willing to work to find a viable solution.
Open Office losing popularity and having a shortage of developers makes some sense to me given all the progress in web-based document editors.
Something I have a harder time understanding is how it came to be that Apache Thrift and Facebook Thrift both exist as competing implementations of the same software originated by the same company.
The implied point with Open Office was not the users habits shifting, but that there was a fork in name only. The project is still under active development with a diverse set of developers but under the name Libre Office.
Only a skeleton crew of paid developers stayed with Open Office, enough to cut releases regularly but not even to fix the security issues actively exploited. All distributions moved with the developers, but there is a discoverability problem which has led to mostly Windows continuing to install the unmaintained version.
The ASF could have fixed this quickly, either by helping out with the trademark issues, moving with the developers, or at least moving the unmaintained version to the attic and steering new users towards the actively developed version.
But they collectively decided to sit on their hands as users continued to install unmaintained software rather than take the slightest risk of offending one of their members. From an outside perspective, all of this was completely unnecessary.
The Thrift situation is another example where some active stewardship could have made a difference.
The ASF didn't have to say yes, and didn't have to allow "Apache OpenOffice" to decay for years afterwards without putting a stop to it.
The current Apache OpenOffice "Vice President" ie the leader of its developers such as they are, is Jim Jagielski, who was also an Apache Director back when a previous "Vice President" suggested the project should be wound up altogether. ASF should have accepted, they did not.
If Linux, OSS, and the like is a firmament of modern society, AND IT IS, why would it be left to adhoc donations or the whims of corporate donations?
There should be billions in funding of OSS from governments each year. Think that's ridiculous? Consider how much Microsoft makes in support payments from governments around the world. The US government has a 92 billion dollar software budget.
If it's that important, just make a new tax and stop dicking around with this "oh it would be so nice if companies would just donate out of the goodness of their hearts hint hint" nonsense. It's pretty clear that 99+% of for-profit companies have zero interest in paying for things that developers will (apparently) also develop for free.
Society figured out a way of forcing people to pay for common goods a long time ago, and it's taxation.
The core infrastructure team is paid, and that’s been the case since fairly early on — there’s too much work not suitable for a purely volunteer crew. Infra is half the budget.
I wanna give props to all the teams who have developed great features to mitigate these types of security incidents. Props to GitHub and Amazon for both producing great sets of vulnerability scanners. Amazon gets bonus points for finding a shaded dependency embedded in a jar released though one of our docker images.
A couple of clarifications that i expect everyone knows, but still should be said.
In all practical sense, no one "at Apache" has released any remediations for log4j. There's just a bunch of random people, probably like 3 around the world, who released these patches.
"how reliant many companies are on third-party and open source code over which they have little control or visibility" - that's crap. They have complete visibility, and a good deal of control. Companies just choose not to spend money doing so. Which is true of their own code, as well.
What the hell is "log4jshell" that is a completely stupid name. It confers some meaning that isn't there.
I apologize for accidentally creating a meme here. (I got angry when I saw the recent "ProxyNotShell" vulnerability in the media...)
It is an interesting exercise in the power of language though. The name "Log4Shell" was literally picked because there wasn't a CVE and people running searches for "log4j RCE" were seeing a vuln in Log4j v1 from 2016. There needed to be a unique identifier that people could search for and tag on Twitter.
The name just... sort of got picked arbitrarily while I was very tired after doing many hours of security research and reading through Enterprise Java code lol. (You're right that "shell" was used to express RCE, ala Shellshock or other major vulns.)
I think the subsequent vulns (Spring4Shell, Text4Shell) just use the format of the name to be more catchy and ride the coat tails of Log4Shell's notoriety. (Like a meme!)
Your blog posts were instrumental in response efforts, and it was honestly a great name (in response to a real problem, as you've mentioned), so I'll forgive you. ;)
> I think the subsequent vulns (Spring4Shell, Text4Shell) just use the format of the name to be more catchy and ride the coat tails of Log4Shell's notoriety. (Like a meme!)
100%. The problem is that it's not only lazy (like how every controversy is X-gate), it also makes executives and senior leaders shit their pants. Log4Shell was a serious wake-up call for many companies and forced them to take security and software seriously. Subsequent clout-chasing or memed vulnerabilities that evoke its name risk creating "alarm fatigue" — "The Boy Who Cried '4Shell'", so to speak.
The big problem here is really the library distribution system for java. Most jvm based projects use binary dependencies that are mostly sourced from maven central. A build tool could automatically block downloads of, or construction of a classpath including, binaries with known vulnerabilities. But then you might find yourself with a broken build that you can't fix yourself leaving you at the mercy of the developers of an upstream dependency that you inadvertently depend on (like log4j). I think the situation would be a lot better if java projects defaulted to source dependencies instead of binary dependencies and build tools were good at supporting remote binary caching so that you could build your own cache of binary artifacts rather than rely on a central one (so that not everyone in your org needs to burn cpu rebuilding libraries with each update). Then even if you indirectly depend on log4j, you could just grep your codebase for log4j, bump the version, rebuild everything and move on with your life. This also could make life easier for maintainers since downstream users might just fix things for them.
Logging is a cross-cutting concern, and most code bases employ a logger such that it violates the DRY principle. Any particular logger shouldn't be difficult to replace in a system. The issue is that developers conflate "logging" with "something happened that must be known." The former being a text message, the latter being an event. The advantage of using an event is that it decouples how the logging transpires from the information about the event. When done well, this can isolate logging to a single line of code in the code base, which is far easier to maintain than countless repetitious logger.debug(...) statements.
Log4shell raised something else that isn't being discussed much: Staying up to date on latest versions of dependencies is not always a good idea.
Log4j 1.x didn't have the log4shell vulnerability. Log4j 2.x did. Those who stayed behind on 1.x never had the log4shell issue, even though the log4shell issue raised some other new vulnerability issues on the old code.
Worse yet, when that happened, the Apache project wouldn't reopen the 1.x branch to allow for fixes, necessitating a fork called reload4j. What could have been an easy version update overriding old versions became more troublesome. Now one must add exclusions for the old log4j to any dependency that has log4j 1.x transitively, and then declare the reload4j dependency to replace them. It is a very suboptimal solution which requires constant vigilance instead of a one and done fix.... all in an apparent effort to force dependency consumers to update to their new version which added major vulnerabilities.
> bring to light how reliant many companies are on third-party and open source code over which they have little control or visibility
All the log4j code is open. All the discussion are open. All the commit log is open. This is the case for the overwhelming majority of "open source code" projects that many companies. It's hard to see how more "control or visibility" could be reasonably enacted for a project like log4j.
All you have to do is actually get involved instead of being an apathetic consumerist and then start crying when it's not all perfect.
77 comments
[ 3.0 ms ] story [ 84.3 ms ] thread> The Apache Software Foundation, which maintains the open-source tool, quickly released a patch...
Apache horribly mismanaged this and did not release a patch until it was already widely known and being exploited in the wild. They also messed up and had to release several subsequent patches to actually fix the vulnerability. This greatly contributed to "[the] mad scramble to plug the gaps".
You can watch the chaos unfold by reading through the GitHub PR. Remember, this was disclosed to Apache in November. https://github.com/apache/logging-log4j2/pull/608#issuecomme...
It may be technically incorrect, but most people wouldn't understand if you got into the minutiae.
It's a distinction without a difference for most people, and trying to clarify the structure of the foundation to executives and senior leaders (technical and non-technical) only makes their perception of any Apache-branded or open source projects worse.
That isn't comparable.
- the name of the project is "Apache Log4J"
- the tag-line is "*Apache Log4j* 2 is an upgrade to Log4j that provides significant improvements over its predecessor..."
- it's hosted under the Apache organization
- the documentation, release notes, and bug tracker is under the apache.org domain.
- the artifacts are published under the org.apache namespace
- etc.
No competent person in the industry is going to think that GitHub is associated with the projects it hosts. There are also hundreds of projects stewarded by Eclipse, Red Hat, and other similar organizations that avoid this confusion by using sensible branding.
What you're saying is more equivalent to "no reasonable person would think that Vitamin Water contains vitamins and is healthy".
https://www.businessinsider.com/coca-cola-glacau-vitaminwate...
I am not sure how to make my point any clearer.
If the concern is that "most people" will have a wrong perception, I don't see how talking losely about Apache will help. Apache is irrelevant in this context. Is your goal to make "most people" steer away from anything Apache?
They put in an "attempt to evaluate strings" feature. Literally as bad as someone doing a `console.log(eval(input))`.
Not only is it that bad, but also it's something with just a tiny bit of inconvenience you can get with either String.format or the likes of slf4j's formatting ( `logger.info("{} value", foo.get());` )
They took the worst case for an injection attack and applied it everywhere.
So this is one of the cases where I'd personally would not be surprised if in the far future someone admits on their death bed that malice was involved.
It's really easy for someone to see a feature like "interpolate strings better than ever" in their string library, and say to themselves, "Hey, that sounds good, let's turn it on!" without realizing that "better than ever" means "with arbitrary lookup from a string into a class of some sort", and then chain a few things together and hey presto you've got an arbitrary code execution vulnerability.
I personally think allowing arbitrary string -> class lookup for all classes in the runtime is automatically a security code smell at best if not outright antipattern, but I don't think this is at all well understood yet. (While the general pattern is hard to avoid, you should have to register all classes/structs/values/whatever you want to automatically load from somehow. This prevents things like "I can use the OS object to execute arbitrary shell code" from automatically and unexpectedly creeping in.) Dynamic languages are all but based on such a capability, with their "eval" support. It's probably still going to be decades before this is generally understood "programmer wisdom", partially because the particular confluence of features that enables this bug is not common. It's just that when it does happen, it's catastrophic. But it's not actually that common.
Spot on. Case in point: the maintainer of snakeyaml furiously arguing that a similar vulnerability in his code is the fault of absolutely everything (client code, any code that can be used as a deserialization gadget, "low quality tooling") except his code: https://bitbucket.org/snakeyaml/snakeyaml/issues/561/cve-202...
...
> 100% of the application which use SnakeYAML do not parse data from untrusted sources.
Wow, what a trainwreck of a thread. I don't know Jonathan but I would love to buy him a beer.
Thank you for sharing. ;)
Then later when someone says they are, he doubles down saying that parsing arbitrary YAML provided from people on the internet isn’t parsing “untrusted” YAML because they’re authenticated. Apparently it’s reasonable that anyone that signs up for a SaaS service gets code execution privileges on the prod infrastructure?
I’m pretty sure that thread had me shaking my head hard enough to give me a concussion.
if library have interface being
It is kinda easy to assume second invocation would not apply formatting and just act as plain value, because it makes zero sense to have format string with no inputs, yet that's what Log4J does.Separating it for "log" and "log with formatting" calls helps a bunch, at the very least makes easier to search for potentially vulnerable code.
You have to evaluate to strings eventually though. A shell console/command prompt has no concept of SQL-style bind-variables. If you are logging "username: {}" that eventually has to be evaluated to a string sometime.
> It was a complete misfeature from start to finish.
No, there was a legitimate demand for this at the start. If you want to log instance-specific information (like JVM container name) the facility provided for that in Java is JNDI. And obviously it would be nice to have "what's the instance that produced this particular message" in the error logs, that's self-evidently helpful and desirable.
The problem is that JNDI is a foot-howitzer, it can also be used to load and execute arbitrary code from remote locations among many other things. Similar to how binary deserialization from remote sources basically cannot be used safely in Java (because a deserialized object can be in an "illegal" logical state and bypasses constructors and other assertions, an object type cannot be known before it is deserialized (so it can be of any class-type), and deserializers can run arbitrary attacker-controlled code before it hits any validation layer) this is not immediately apparent just from the fact that you are using JNDI, but, it's safe because we're not going to call attacker-controlled remote code through JNDI, right? Why would we attack ourselves?
And then the final piece is that (as you note) they fucked up the order of evaluation. JNDI is evaluated at the very end of the stringification process, meaning an attacker can include their own JNDI calls in the string and then it will be evaluated like developer-controlled JNDI. That is the fundamental problem here - this is bog-standard SQL-injection-attack style stuff, you should never be able to escape an injected/bound variable, but JNDI evaluation takes place after stringification.
All that really needed to be fixed was the last part. But honestly the dogma of the Java world has moved away from this whole approach in the first place - people were doing multi-container-per-JVM back in 2000s when memory was scarce, so you wanted to know that instance information so you could tell what instance was generating a message. But nowadays people do 1-container-per-JVM (and many JVMs scaled in a cluster) and you can handle that with runtime variables/etc that let the instance know where it is from external oracles. So there is no need to touch that in the first place.
It's all very much functionality that was demanded by real-world users... in the early 2000s. But Java refuses to clear some of the cruft from their APIs because it would produce (gasp) breaking changes in legacy applications... sort of the exact opposite of the Python 3 mindset. And JNDI is one of those interfaces that was just way too powerful when it was created. As mentioned, binary deserialization is another... you can't really use that in almost any scenario because the attack surface is just too large.
https://cheatsheetseries.owasp.org/cheatsheets/Deserializati...
(JNDI plus deserialization is a good one too, up until a couple years ago you could specify the class/deserializer as living on a attacker-controlled remote source, because JNDI can do anything!)
https://www.veracode.com/blog/research/exploiting-jndi-injec...
Making something work in Java is easy. Proving it is secure is not.
It is almost certain that we over-reacted and shot ourselves in the foot with this, but our customers were happy with the way in which we responded. Better safe than sorry was the theme throughout.
Ok, so which language it is easy to prove secure?
Any use of Java will trigger additional diligence work to prove it does not use vulnerable log4j. So in this sense almost anything else is better.
But, we do have sufficient resources to justify and defend decisions made with our primary development language.
In this context, "prove" is more like "convince".
I've never experienced the problem of "nobody in my company knows Java well enough to trace the dependencies of a JAR". Do you all works in Rust and Javascript or something?
When i read here about Log4J, is was a real "Holy shit moment". It was so easy to use, it could've been a nebula exercise.
I contacted my PO asap, we found out that we had no inventory of what our VM were and what java versions were installed (worked at a bank btw). We then contacted the secops team, who just found out, and triggered all alarms. It was all hands on deck time. Quite a fun two weeks tbh.
Not familiar with the term PO in the way you are actually using it. What management level does that represent?
It's basically the same managment level as the tech lead/lead dev.
I understand this sound like scrum/agile gibberish, but honestly this was like having a technical manager/salesman who could also help you debug stuff, probably the most usefull post of "agile at scale"
You may be surprised when I tell you what the Apache Software foundation's yearly budget is. You'd think for software that is used by practically every Fortune 500 company and most governments, it would be something reasonable. Maybe a few hundred million dollars a year to pay for a reasonable full-time staff, right?
It turns out... it's about $2 million a year. (Wikipedia[0])
This helps explain to me why the devs of Log4j directly uploaded the file "JNDIExploit.java" (the POC) to GitHub while they were patching. (Here is a full analysis about what happened[1].)
They're not security people. They're volunteers working on this in addition to their full-time job.
What kind of brave soul wants to trudge through and maintain log4j in their spare time for zero compensation? I appreciate the people that are capable of doing that, but I think they are rare!
This whole entire vulnerability was eye opening for everybody and I have actually spent the last year building tooling on GitHub to help fix the problems that Log4Shell exposed.
If you have 2 seconds to try that out or just Star the repo[2], it would be very helpful!
0: Log4j revenue https://en.wikipedia.org/wiki/The_Apache_Software_Foundation
1: "How to Discuss and Fix Vulnerabilities in Open Source" https://www.lunasec.io/docs/blog/how-to-mitigate-open-source...
2: GitHub project building better dependency patching tools https://github.com/lunasec-io/lunasec
Can't this sentence be changed for almost any open source project?
Compare that to something like Bun.js[0] which is "sexy" and written in a "cool" programming language (Zig). Or Wasp[1] which is built with Haskell and is trying to define a new programming language designed to make common dev patterns less painful.
Those projects are naturally going to soak up smart people that have extra energy to share because they hate their day job but need to pay their bills. (imo)
Who is left that wants to bang their head against a legacy codebase like Log4j? Maybe somebody that feels there is "clout" to be had from it? (Spitballing here, I honestly don't know!)
0: https://github.com/oven-sh/bun
1: https://github.com/wasp-lang/wasp
There are a LOT of Java SWEs floating around...
There are people that like to clean messes and make people life better from the shadows. These tend to be called engineers.
You may be interested in https://www.fordfoundation.org/work/learning/research-report...
It's not clear to me as an outsider what exactly the Apache foundation is doing for these projects. It feels like Apache is willing to accept code donations from anyone and is willing to attach the foundation's name to code that isn't widely used, actively maintained, or may just be abandonware.
I have soooo much more confidence in CNCF projects. The conditions for graduating as a CNCF project include criteria like that your project must be in use by multiple real companies, have maintainers who are (paid) employees of multiple different companies, and get a professional security audit.
That's why I'm allergic to Apache software. A lot of it is overengineered, insecure, legacy abandonware.
That’s incorrect. Projects need to report quarterly and need a Project Management Committee of at least three people, or they are retired. Retired projects may not make releases.
(Source: past ASF board member, who used to review those reports each month.)
There are a fair number of retired projects, and others that may become retired within the near-to-medium term. The ASF has been around for a while, and every software project has a life cycle. Those are still associated with the ASF brand because Google, whatcha gonna do? An explicit retirement policy overseen by a board is still superior to how the vast majority of open source projects approach end-of-life.
The project has few, if any, volunteers, and there are security problems known to be actively exploited, yet the ASF is not willing to work to find a viable solution.
Something I have a harder time understanding is how it came to be that Apache Thrift and Facebook Thrift both exist as competing implementations of the same software originated by the same company.
Only a skeleton crew of paid developers stayed with Open Office, enough to cut releases regularly but not even to fix the security issues actively exploited. All distributions moved with the developers, but there is a discoverability problem which has led to mostly Windows continuing to install the unmaintained version.
The ASF could have fixed this quickly, either by helping out with the trademark issues, moving with the developers, or at least moving the unmaintained version to the attic and steering new users towards the actively developed version.
But they collectively decided to sit on their hands as users continued to install unmaintained software rather than take the slightest risk of offending one of their members. From an outside perspective, all of this was completely unnecessary.
The Thrift situation is another example where some active stewardship could have made a difference.
Hey Oracle.
— Larry Ellison, probably
The current Apache OpenOffice "Vice President" ie the leader of its developers such as they are, is Jim Jagielski, who was also an Apache Director back when a previous "Vice President" suggested the project should be wound up altogether. ASF should have accepted, they did not.
There should be billions in funding of OSS from governments each year. Think that's ridiculous? Consider how much Microsoft makes in support payments from governments around the world. The US government has a 92 billion dollar software budget.
I mean, look at this: https://www.reuters.com/article/us-usa-cyber-microsoft-exclu...
Society figured out a way of forcing people to pay for common goods a long time ago, and it's taxation.
If Apache tried to use it without doing it in the context of a formal deal or sponsorship, it wouldn’t last long.
No such thing as a free lunch, or whatever.
The core infrastructure team is paid, and that’s been the case since fairly early on — there’s too much work not suitable for a purely volunteer crew. Infra is half the budget.
In all practical sense, no one "at Apache" has released any remediations for log4j. There's just a bunch of random people, probably like 3 around the world, who released these patches.
"how reliant many companies are on third-party and open source code over which they have little control or visibility" - that's crap. They have complete visibility, and a good deal of control. Companies just choose not to spend money doing so. Which is true of their own code, as well.
What the hell is "log4jshell" that is a completely stupid name. It confers some meaning that isn't there.
It's a pun on Log4J and describes the vulnerability: all you need to get reverse shell is for the application to log input.
What meaning do you feel it confers that isn't there?
(The trend of subsequent vulnerabilities being dubbed "...4Shell" is infuriating, though.)
It is an interesting exercise in the power of language though. The name "Log4Shell" was literally picked because there wasn't a CVE and people running searches for "log4j RCE" were seeing a vuln in Log4j v1 from 2016. There needed to be a unique identifier that people could search for and tag on Twitter.
The name just... sort of got picked arbitrarily while I was very tired after doing many hours of security research and reading through Enterprise Java code lol. (You're right that "shell" was used to express RCE, ala Shellshock or other major vulns.)
I think the subsequent vulns (Spring4Shell, Text4Shell) just use the format of the name to be more catchy and ride the coat tails of Log4Shell's notoriety. (Like a meme!)
> I think the subsequent vulns (Spring4Shell, Text4Shell) just use the format of the name to be more catchy and ride the coat tails of Log4Shell's notoriety. (Like a meme!)
100%. The problem is that it's not only lazy (like how every controversy is X-gate), it also makes executives and senior leaders shit their pants. Log4Shell was a serious wake-up call for many companies and forced them to take security and software seriously. Subsequent clout-chasing or memed vulnerabilities that evoke its name risk creating "alarm fatigue" — "The Boy Who Cried '4Shell'", so to speak.
https://en.wikipedia.org/wiki/Alarm_fatigue
https://xeraa.net/blog/2021_mitigate-log4j2-log4shell-elasti...
I've blogged about this to mixed reception:
https://dave.autonoma.ca/blog/2022/01/08/logging-code-smell/
Log4j 1.x didn't have the log4shell vulnerability. Log4j 2.x did. Those who stayed behind on 1.x never had the log4shell issue, even though the log4shell issue raised some other new vulnerability issues on the old code.
Worse yet, when that happened, the Apache project wouldn't reopen the 1.x branch to allow for fixes, necessitating a fork called reload4j. What could have been an easy version update overriding old versions became more troublesome. Now one must add exclusions for the old log4j to any dependency that has log4j 1.x transitively, and then declare the reload4j dependency to replace them. It is a very suboptimal solution which requires constant vigilance instead of a one and done fix.... all in an apparent effort to force dependency consumers to update to their new version which added major vulnerabilities.
All the log4j code is open. All the discussion are open. All the commit log is open. This is the case for the overwhelming majority of "open source code" projects that many companies. It's hard to see how more "control or visibility" could be reasonably enacted for a project like log4j.
All you have to do is actually get involved instead of being an apathetic consumerist and then start crying when it's not all perfect.