I’m amazed on how many websites today still don’t use script hash or nonce and happily reference random CDNs with unsafe-inline. Not saying this then wouldn’t be an issue, but at least you have a failsafe.
Great work finding this out, it was really fascinating read. Just wondering how do you publish compromised package to npm? CDN had to install some npm package right? Publishing some package is easy, but how do you publish something that is downloaded by Cloudflare? If it is so easy to publish compromised npm package and it ends up in cloudflare CDN it is quite scary, even if underlying security loophole related to overriding paths by tar is fixed.
cdnjs and similar CDNs should be considered mirrors of what they are serving, so they don't do any security checks on it. It is best not to use these services, especially since cache-partitioning and http2 makes them obsolete.
The vulnerability is the ability to poison the cache of some popular library once I steam the WORKERS_KV token. I could choose the most popular library, say Bootstrap, and change it to something else. People who loaded the script with SRI on browsers that support SRI would notice a problem, but could still target millions of others.
Do they? Their website https://cdnjs.com/libraries says
"Couldn't find the library you're looking for?
You can make a request to have it added on our GitHub repository."
I'm just trying to find the root cause. I see that is PR is merged by a contributor "xtuc". Root question is that why was this PR merged? Apart from reading/writing to other files in the build pipeline instance (which is a really good catch), primary problem I see here is that anyone is able to deploy any arbitrary code/javascript to serve it from cdnjs. Just send a PR, then some contributor will merge it without checking the release package content. Don't you think that is open to a lot of issues?
Just to clarify, we won't just mirror any npm package. We have a set list of git repos and npm packages that we pull from, and adding a new one requires a PR to our packages repo which a human must review.
or maybe there's another exploit to publish a bad version of someone else's package to npm that is already approved to have new versions mirrored by cdnjs robots ( i have no evidence that this was done in this case, but an attacker might be able to do it. there's a lot of surface area to attack in build & release pipeline of 4000+ packages )
Great job on finding the vulnerability. Anyone who uses a CDN link should use the integrity attribute, which can be generated by https://www.srihash.org/.
If you click the "copy script tag" icon on a library page (such as https://cdnjs.com/libraries/lodash.js), it will include the integrity attribute for you.
What would likely happen if the SRI hash check triggered? The devs would have to be pretty savvy to suspect CloudFlare has actually been breached and refrain from just remaking the hash for the new updated content, in the middle of a production outage.
I think the issue is that a lot of devs won't care, they are contractors and some project leader calls and says something is down, they see a failed hash, download it again and restart.. and the hash check in the first place was probably a copy/paste from a SO question..
I'm sorry for the cynicism but I'm pretty sure this is the reality for the bulk of internet companies out there.. :/
>What other conclusion could you come to apart from a CDN breach?
"Ah fuck, hundreds of users are suddenly calling/emailing/tweeting at us saying they can't load the site, and I see a bunch of errors in the console while testing from a fresh browser. Looks like it's only affecting browsers that don't currently have this CDN JS file in cache. The CDN must have broke something, but things seem to work okay when including the script with a standard script tag. I'll just remove this SRI attribute thing for now to stop the bleeding."
Or, if they have a bit more knowledge/do a bit more research:
"Looks they might have pushed a change that minifies the JS file slightly differently or something, or maybe the script got upgraded to patch some issue? And I guess the change probably broke the SRI thing? I don't know if the file is going to change again in the near future so I'll just remove this annoying thing for now until they provide some recommendations." (Or: "I'll just hash the current version of the file and change it to that, and prepare to change it again ASAP if the errors re-appear.")
Someone more experienced/knowledgeable would obviously not operate that way, but given how popular cdnjs is, seems fairly likely at least a few administrators would do something like that. Especially if they don't fully understand what the intitialism really means or what a hash is or how integrity is defined in this context and what the implications might be. Their instinct might just be to see something breaking the site for lots of users and to "fix the problem" as quickly and easily as possible.
Wouldn't it be better to switch to a self-hosted copy of the file and retain the SRI attribute ("reasonable fallbacks" as robin_reala said), until you get time to investigate it thoroughly?
To be specific, that should be the default :) By “reasonable fallbacks”, I meant more that your application should fail gracefully when modules / components fail to load. Escalators become stairs as their failure mode.
Yes, that would be the appropriate thing to do, for sure. The parent poster rhetorically asked about other responses people might have, so I was just giving a few examples of what a less experienced person might try.
I don’t recall ever seeing ‘don’t use SRI on non versioned resources’ - don’t get me wrong, it makes sense, but I’m a pentester and my clients know less than I do
> What other conclusion could you come to apart from a CDN breach?
The majority of alerts from security systems and mechanisms are in the end false positives in that the reason is a misconfiguration or some pther non malicious reason. Sad but true.
They should also consider whitelisting libraries individually in their CSP header (or meta tag if the header gets too long), instead of a blanket whitelist for an entire domain/subdomain or not having a CSP at all...
One aspect that I think isn't common knowledge: You shouldn't use javascript libraries from CDNs. It's less secure, has less privacy and is slower than selfhosting.
Why is it less secure? You're adding an additional point of attack. Someone could compromise the CDN or they could stop operating and the domain will fall to someone else. (And it doesn't really matter if your CDNs security is better or worse than your own hosting security - if you use a CDN it adds security risk and does not take away the risk from whatever other hosting you're using.)
Why privacy? Your CDN can at the very least read the IPs of your visitors (+ they could add cookies, which will usually not happen but still is a possibility) and you may need to worry about legal issues (e.g. adding it to your GDPR statement).
Finally performance: I guess many people will use a CDN "because it's faster" - except it's not.
In the past there used to be the argument that the JS may already be cached. However that's no longer the case as for security reasons browsers partition the cache based on the origin.
It's always faster to fetch a resource over the same connection compared to opening up a new connection to a different host.
So: Don't fetch your JS from CDNs, just put it in your webspace and fetch it from there.
On performance it is also good to mention that these sort of cdns where mostly conceived before http2, so then you ran into the risk of filling your connection quota to the origin server. Nowadays with http2/http3 that is no longer a problem.
Also reliability: by using a 3rd party service for important parts of your page/site/app for performance reasons is essentially RAID0 for web requests. You now have two (or more) points of failure (your infrastructure and the CDN) each of which can single-handedly break your stuff. And the break doesn't have to happen at the CDN, it could for instance be a failed DNS lookup, or an odd routing error at an ISP which affects the CDN but not your main location, a plugin such as noscript, or many other possibilities.
Yes, there are methods that allow you to fall-back to your own local copy if the CDN copy fails to arrive, but that is adding further complication and could effectively slow down your page/app by getting in the way of some “make the initial render as early as possible” techniques.
I never liked the idea of loading code from CDNs, or anything else for that matter, even when the benefits seemed more real than they do now.
> Finally performance: … there used to be the argument that the JS may already be cached
For larger CDNs with the resources to have many widely distributed points of presence there is a potential benefit that is still relevant even with partitioned caching: multi-view DNS or any-cast routing can result in a user's request being served from a topologically closer host than otherwise.
Though I suspect that is you care that much about this small benefit you are probably running your own multi-homed infrastructure using the same technique(s) so CDNs aren't needed for you to benefit from them. And back to reliability: such methods further abstract away from the simple “make request, get response from expected host” arrangement and make diagnosing problems further down the line more difficult.
"archives that contain filename like ../../../../../../../tmp/test may overwrite arbitrary files on the system"
Maybe paths like that and symlinks should be behind flags that are disabled by default? Specially if you are extracting content that you didn't generate.
gnu tar does disable extracting paths like that by default. You need to explicitly add the '--absolute-names' flag to gnu tar in order to extract such a malicious archive.
I think all tar utilities that are in common usage disable extracting those sorts of paths by default.
It just happens that the go stdlib includes a tar implementation that was written from scratch (the go authors are allergic to re-using any C code and will NIH as much as they can).. and they probably don't want to fix it now because the go1 promise heavily discourages making changes like that, even if security is an exception to it.
> It just happens that the go stdlib includes a tar implementation that was written from scratch (the go authors are allergic to re-using any C code and will NIH as much as they can).. and they probably don't want to fix it now because the go1 promise heavily discourages making changes like that, even if security is an exception to it.
It's not about NIH, it's about this being a low-level tar parsing library, not a ready-to-use unpack tool or even a library with unpack functionality. I would never expect such a library to do any sort of automagic path mangling for me unless explicitly stated!
I see the same behaviour in libarchive (C library), microtar (C library), and calccrypto/tar (C _tool_ that is vulnerable to this). How would have re-using any C code helped here, if the top three (from Google search results) C tar libraries have the exact same behaviour? The same behaviour is in Python's tarfile, too.
Plus, even with Go1 stability guarantees, there's nothing preventing the Go standard library from adding a struct method like .SafePath() if it's deemed necessary - standard library types regularly get new fields/methods added to them.
> the most popular tar extraction library in rust has a secure default for this
... only on the high-level unpack methods. The low-level unsafe stuff is still there: https://docs.rs/tar/0.4.35/tar/struct.Entry.html (.path, .unpack). And that's basically the API level which Go's archive/tar provides. If that crate didn't provide unpack functionality, it would have been the exact same kind of footgun.
> It's not about NIH, it's about this being a low-level tar parsing library, not a ready-to-use unpack tool or even a library with unpack functionality. I would never expect such a library to do any sort of automagic path mangling for me unless explicitly stated!
I can appreciate this but I disagree; the defaults should be the exact reverse -- do the safest thing by default and people who "know what they're doing" (famous last words IME) should have a path to do the less safe thing.
I lean towards having programming languages that are developed with the most mistake-prone 20-40% of programmers in mind, rather than the absolute best software engineers, though I certainly understand the hesitance to have languages come with training wheels by default.
Automagically rewriting/removing some files based on path fragments is not universal safety. There is no safe subset of extracting archives, IME. You might fix this particular bug class (where someone implements a poor file extraction routine), but you can introduce tons of other classes of bugs, like, off the top of my head:
a) someone uses this library for validating whether files contained are conforming a given layout, then passes the tarball to tar - differences in behaviour (tar strips leading .., while eg. the Rust module ignores all files with '..', no matter the position in the path) might cause a security issue.
b) silent removal of 'invalid' paths might cause unexpected behavior for when these paths actually are valid and expected (you might not have used ../ in tar, but I have); can imagine one hell of a debugging session coming from that one. I can also imagine codebases that filter tars for malware, and if they happen to skip ../-paths (because of the tar parser silently skipping them), they could let through the tarball unaffected, with some files unscanned, but extractable in GNU tar.
c) silent rewriting of 'invalid' paths might cause inconsistency between the behavior of different tar libraries, or might cause internal inconsistency due to two 'invalid' paths sanitizing to a single valid path; imagine someone relying on ../-stripping behaviour at some point, then that behavior changing due to switching libraries or a rearrangement of the files in the source tarball. Sanitizing is a complex issue, and any kind of universal logic might actually introduce more bugs than it solves.
d) false sense of security for more complex classes of bugs, like extracting a tarball with a symlink pointing outside of the root of the archive. What should the default behaviour be then, and why? Sometimes symlinking to `/etc` is exactly what the user needs, sometimes it's going to cause a bug further down the line. What about relative symlinks? What about cross-filesystem symlinks? What about the setuid bit? If you make users expect the library to do the right thing, it might cause them to think even less about possible security implications of more tricky situations. Writing user-controlled data to the filesystem is _always_ dangerous and there is no right way to do it, it all depends on the context.
I agree that the default behaviour of an _extraction_ library should probably be to sanitize paths. However, I cannot agree that this should be the default behavior of a thin file parser library, like archive/tar. The only thing Go should do, IMO, is explicitly state in documentation that acting on user-supplied tarballs is dangerous and great consideration should be taken before blindly extracting files to the local filesystem.
I can think of dozens of edge cases in all kinds of security controls that remain security weaknesses despite a large chunk of problems being addressed. The point is to address the bulk of the cases and not allow perfect to become the enemy of good.
> a) someone uses this library for validating whether files contained are conforming a given layout, then passes the tarball to tar - differences in behaviour (tar strips leading .., while eg. the Rust module ignores all files with '..', no matter the position in the path) might cause a security issue.
I would argue that any library that does not already fully resolve relative paths to do this is already faulty and this makes it no less safe. To draw an example, if at the time I pass in a relative path to say, "all files should exist under this directory" and provide "/tmp/somedir/../../usr/lib/" as that parameter, I would expect that this already resolves to "/usr/lib/" before doing any processing. This would eliminate the entire problem presented here.
> b) silent removal of 'invalid' paths might cause unexpected behavior for when these paths actually are valid and expected (you might not have used ../ in tar, but I have); can imagine one hell of a debugging session coming from that one. I can also imagine codebases that filter tars for malware, and if they happen to skip ../-paths (because of the tar parser silently skipping them), they could let through the tarball unaffected, with some files unscanned, but extractable in GNU tar.
I agree that silent removal is a problem. There should be errors/warnings attached to indicate that this has been done.
This isn't a good counterpoint to actually having a sane default though. The very issue here is we have a clear security bug introduced in part due to the fact that the library does something seemingly unexpected precisely because it does not line up with existing tooling.
> c) silent rewriting of 'invalid' paths might cause inconsistency between the behavior of different tar libraries, or might cause internal inconsistency due to two 'invalid' paths sanitizing to a single valid path; imagine someone relying on ../-stripping behaviour at some point, then that behavior changing due to switching libraries or a rearrangement of the files in the source tarball. Sanitizing is a complex issue, and any kind of universal logic might actually introduce more bugs than it solves.
Again, I'm not advocating it be an invisible process, just that the default should be the safer of the two options for the bulk of cases.
> d) false sense of security for more complex classes of bugs, like extracting a tarball with a symlink pointing outside of the root of the archive. What should the default behaviour be then, and why? Sometimes symlinking to `/etc` is exactly what the user needs, sometimes it's going to cause a bug further down the line. What about relative symlinks? What about cross-filesystem symlinks? What about the setuid bit? If you make users expect the library to do the right thing, it might cause them to think even less about possible security implications of more tricky situations. Writing user-controlled data to the filesystem is _always_ dangerous and there is no right way to do it, it all depends on the context.
What you describe is not a code problem but a programmer / documentation problem, and is really about the idea that fixing one problem does not inherently fix another problem that it's not necessarily aimed at fixing.
Clear documentation that outlines the steps that are taken, as well as the option to disable the feature if truly needed (as I suggested) would solve this entirely.
Also, this is a weird argument to make given this already exists with existing tooling. I expect if I name a file "../../../../etc/cron.hourly/evil.sh&qu...
thanks a ton for the link. This looks way better than I would have expected.
Though also, "we've seen this cause exploits in the wild multiple times, let's just let this footgun sit for 3 years".
They could have at least added a documentation note to the archive/tar page that there's a security issue you have to handle, which is what the unsafe version of the rust unarchive methods do. 3 years of not even updating docs is kinda unfortunate.
Unlike the go library, on the unsafe methods, the rust library includes a note explaining the security footgun.
It also provides 'unpack_in' on entry, so there's a secure and insecure variant.
The go library should at least include a note, but does not. The rust library does not have the same footgun because it is clearly documented and provides a secure alternative.
After checking the contents of the file to send the report, I was surprised.
Surprisingly, clearly sensitive information such as GITHUB_REPO_API_KEY and WORKERS_KV_API_TOKEN was displayed. I couldn’t understand what happened for a moment, and when I checked the command log, I found that I accidentally put a link to /proc/self/environ instead of /proc/self/maps.
[…]
It seems that GitHub notified Cloudflare immediately because GITHUB_REPO_API_KEY (API key of GitHub) was included in the repository, and Cloudflare started incident response immediately after the notification.
[…]
April 6, 2021 20:30 cdnjs processed the file
At the same time GitHub sent an alert to Cloudflare
At the same time Cloudflare started an incident response
Within minutes Cloudflare finished revocation of credentials
April 6, 2021 20:40 I sent an initial report
This a nice Github feature. Also kudos to Cloudflare for quickly reacting to this.
You might have strings that match one of their secrets regex's that are not an actual secret. It would be extremely annoying if they blocked you from pushing a commit in that case.
Github doesn't check if the secrets are real or not. Any string that matches one of the regexs gets shared with the appropriate API provider. The provider checks to see if the its a valid credential or not and revokes the key if necessary.
It's more to avoid false positives. I've had bug bounty hunters flag secrets in my projects before looking for a bounty, but in actuality it was some random string in a variable named AWS_ACCESS_KEY etc that we used in a unit test (which mocked out the SDK but still expected the config values to exist at init)
I'm wondering if this couldn't be used as a way to confirm or search for secrets: push huge files with many "secrets", if it's rejected, then you know it has a valid one, then rince and repeat with a binary search to find the one.
It is possible they received a reward, it just isn't publicized.
"The decision to pay a reward is entirely at our discretion. You must not violate any law. You are responsible for any tax implications or additional restrictions depending on your country and local law. We reserve the right to cancel this program at any time."
That commit log should give you a better sense of what happened if you (like me) didn't understand how cdnjs works. Apparently robocdnjs will just pull arbitrary packages and unzip and commit them into the cdnjs repo which then gets served all over the internet. Crazy!
> April 6, 2021 20:30 [JST] cdnjs processed the file
you can see commits in the cdnjs/cdnjs git log adding an absurdly named test package from one of the maintainers to test packages containing symlinks within 24 hours after the exploit
CDN can protect your servers from DDOS and countless attacks, bots, vulnerabilities. It tends to be patched pretty quickly when there is an issue. It can reduce cost by offloading, improve performance by distributing content (with or without computing) from a closer location, optimise delivery in many ways, scale with limited cost...
Extra cookies and scripts are related to specific features. A good CDN will let you configure exactly what you want.
Cloudfront is a CDN.
EDIT: The issue is more related to using a third party domain in order to load assets on you website. I agree, that should be done from first party when possible.
It’s often the lowest-friction way to include a library in a project. You’ll often see “getting started“ documentation using cdnjs.com because it’s so easy. That’s not a good reason, but probably a very real one.
> 1) The biggest benefit of using cdn was that scripts stylesheets were often loaded from cache. This is no longer the case anymore.
This is indeed true. However, most websites (despite the relative ease of activating a CDN service) only host their website in a single location. In those cases, library CDNs (to differentiate from other uses of CDNs) can still benefit the website, even when you consider optimisation strategies such as HTTP/2. Additionally, the stress on their servers are decreased because it's handled by their library CDN provider.
> 2) It's adding an unnecessary vulnerability as seen from this case.
> 3) CDN scripts can be (possibly) used to track your website traffic.
Good points and something that I can't rebut easily. It is indeed on the onus of the website to check the reputation of library CDNs, and it may violate privacy guarantees (you're trusting Google/CloudFlare/Microsoft/Amazon/Fastly/MaxCDN etc.) Specifically on 2) however, they knew that they are large: the speed of fixes (as shown) here proves that library CDNs can benefit websites' security, which tends that its software stack is updated slower (by the website operator's fault). You can also leverage subresource integrity (https://developer.mozilla.org/en-US/docs/Web/Security/Subres...) to fail hard.
> 4) They often drop third party cookies which decreases your website load speed among other things.
Citation needed here. As far as I checked (and just verified as I post this, to ensure that this is still true), every library CDNs I knew (even in weird places such as Japan- and China-centric library CDNs) do not place cookies in the library URLs themselves (some do on their websites themselves, but it is usually separated by domain names).
> It's best to just host your script on Digital ocean spaces or S3 and use a Cloudfront proxy.
If you're a large website or basically need to ensure the privacy is guaranteed, sure! I see that large websites do in fact use their own choice of CDN providers (most websites including Amazon themselves uses Fastly though). Some even do a step-up: some websites (for example) use Fastly for their American traffic, KeyCDN for European traffic and Quantil for Asian traffic. However, I only knew of two CDN providers which provides a flat-rate and reasonable costs: CloudFlare (uhh, CDNJS operator) and DDOS-Guard (which some consider shady).
A CDN is really only useful if you store your entire website with all it's content on it. Or if you serve a lot of data (petabyte egress) where the CDN have better peering deals for lower bandwidth cost. If you operate a "normal" static website, make sure http2 is activated, and that assets that never change are cached. And if possible use a hosting provider in the same country your users are located.
HTTP/2 suffers from Head-of-Line blocking, which may impede rather than help speed up the connection. In most cases, this doesn't happen, but when they do happen it's consistent. Testing is required, because there are cases where HTTP/2 backfires spectacularly.
> That assets that never change are cached
Fair point, I also strongly recommend this. Plus, if your server supports statically compressing resources ahead of time it pays serious dividends (and activating Brotli, if you can custom compile applications or lucky to use a distribution that built it).
> And if possible use a hosting provider in the same country your users are located.
In Europe, this is relatively easy. In the US and Russia, probably two servers (for East and West coast/okrugs). In Asia? Surprisingly difficult, unless you're in East Asia (I'm not discussing why here since there's plenty of resources to learn more about why it's a pain in the rear). Africa? Actually, despite improvements in connectivity I strongly recommend to make the pages small and simple (as the requirements allow).
I've been running several sites off Firebase Hosting the last 4 years and have seen surprisingly high downtime, cache clearing errors, and unexplained 502 errors from Firebase Hosting.
For users: Decentraleyes (https://decentraleyes.org/) is a browser extension that will intercept requests to common CDN library services and serves them from local storage instead if at all possible, kinda like enforcing aggressive Cache-Control for all CDN resources.
It's a privacy-oriented extension first, but it may still protect from an eventual zero-day like this.
cdnjs is a CDN for javascript libraries. cdnjs has a "library update server", in which an "auto-update script" runs periodically to update the libraries that the CDN offers. If it finds a new version of a library, it downloads the files from the library's git repository, and it publishes them on the CDN.
There can be symlinks in git repositories, but the "auto-update script" doesn't handle them correctly. And that's the core of the vulnerability.
If you own the git repository of a library published by cdnjs, you could add a symlink to the git repository. The symlink could point to a file that you want to access on the cdnjs "library update server". Then, as soon as the "auto-update script" runs, the file's contents will be public to everyone on the CDN.
But how would you manage to create a malicious library and have it approved by cdnjs? That's the part of the exploit that sounds the hardest, and I don't see a reasonable way to go about it.
> But how would you manage to create a malicious library and have it approved by cdnjs? That's the part of the exploit that sounds the hardest, and I don't see a reasonable way to go about it.
1. Hijack an existing library to publish the malicious version
2. or, Have us approve what looks like a legitimate library, which later publishes a malicious version
Hijack or buy outright an existing library. IIRC, there have been many such instances where existing projects have been sold for large sums to questionable parties.
As I understand it. Every library in npm registry (they library you can install using npm) are also automatically approved ( which sounds reasonable you don't want to approve every library by hand).
And that's what the author exploits by registering a library that have a symlink in it. Which then got excuted.
I was a bit disappointed by Cloudflare's HackerOne-based disclosure / bug bounty program.
I submitted a working proof-of-concept that would allow you to use Cloudflare's page-customizing preview function to create a Cloudflare phishing site.
The issue was closed as "accepted as the nature of the cloudflarepreview playground".
Then they went ahead and fixed the issue.
Not the end of the world, but definitely didn't inspire confidence in the program.
Last year I submitted a working proof of concept with detailed explanations of how you can steal passwords and details of users as an app maker on Shopify, and it got denied for a similar reason (though I dont think it's been fixed). I've since stopped wasting my time on such pursuits.
No joke, have you thought about selling your exploits to other actors?
Exploitation has such a rich history between sellers, buyers, and users; the marketplace is way more sophisticated than what it was in the 90s.
You can probably get $5k to $20k for the exploit to Shopify. There are plenty of nationstate actors that create honeypots for a variety of targets. I'm sure Shopify exploits are worth something to someone.
Selling exploits use to be way more common prior to 2010. Previously many companies wouldn't pay for bug fixes and would periodically sue developers for disclosing issues.
There eventually became a large movement within the cyber security groups about no longer giving bugs away for free and demanding fair market value for exploits (imagine the NSA paying $100k for a zero-day, versus the company giving you $500):
Nowadays the highly sought out targets like Android or iOS have premiums for zero-days (upwards to $1mil), in-fact it wasn't until Google started paying more for exploits was when Android security started getting better. But if some companies aren't willing to pay fair market value for their exploits, why should it matter if you sell it to someone else?
102 comments
[ 3.0 ms ] story [ 78.3 ms ] threadI've read the pricing models are different between the two and pentesters seem to work with both.
I'm a bit confused... how did they get their updates pushed to a repo they didn't control?
They didn't update any part of cdnjs itself. Instead, they triggered cdnjs on their package.
Amusingly you can see the "hey-sven" library they added to test the fix: https://github.com/cdnjs/packages/pull/695
If you download the tarball directly, https://registry.npmjs.org/hey-sven/-/hey-sven-1.0.2.tgz, then tar -ztvf hey-sven-1.0.2.tgz, you can see
-rw-r--r-- 0 ryotak staff 204 Jun 2 16:21 package/package.json
-rw-r--r-- 0 ryotak wheel 10 Jun 2 16:21 ../../../../../../../../../../tmp/ryotak
-rw-r--r-- 0 ryotak wheel 10 Jun 2 16:22 ../../../../../../../../../../tmp/ryotak.sh
both which just have the contents "Retesting"
Because if he did that symlink should at least raised some eyebrows at least
from 2019/12/20 cloudflare blog post about cdnjs project: https://blog.cloudflare.com/an-update-on-cdnjs/
or maybe there's another exploit to publish a bad version of someone else's package to npm that is already approved to have new versions mirrored by cdnjs robots ( i have no evidence that this was done in this case, but an attacker might be able to do it. there's a lot of surface area to attack in build & release pipeline of 4000+ packages )
What’s harder is persuading devs to architect systems with reasonable fallbacks in case CDN loading fails.
I'm sorry for the cynicism but I'm pretty sure this is the reality for the bulk of internet companies out there.. :/
"Ah fuck, hundreds of users are suddenly calling/emailing/tweeting at us saying they can't load the site, and I see a bunch of errors in the console while testing from a fresh browser. Looks like it's only affecting browsers that don't currently have this CDN JS file in cache. The CDN must have broke something, but things seem to work okay when including the script with a standard script tag. I'll just remove this SRI attribute thing for now to stop the bleeding."
Or, if they have a bit more knowledge/do a bit more research:
"Looks they might have pushed a change that minifies the JS file slightly differently or something, or maybe the script got upgraded to patch some issue? And I guess the change probably broke the SRI thing? I don't know if the file is going to change again in the near future so I'll just remove this annoying thing for now until they provide some recommendations." (Or: "I'll just hash the current version of the file and change it to that, and prepare to change it again ASAP if the errors re-appear.")
Someone more experienced/knowledgeable would obviously not operate that way, but given how popular cdnjs is, seems fairly likely at least a few administrators would do something like that. Especially if they don't fully understand what the intitialism really means or what a hash is or how integrity is defined in this context and what the implications might be. Their instinct might just be to see something breaking the site for lots of users and to "fix the problem" as quickly and easily as possible.
The majority of alerts from security systems and mechanisms are in the end false positives in that the reason is a misconfiguration or some pther non malicious reason. Sad but true.
Why is it less secure? You're adding an additional point of attack. Someone could compromise the CDN or they could stop operating and the domain will fall to someone else. (And it doesn't really matter if your CDNs security is better or worse than your own hosting security - if you use a CDN it adds security risk and does not take away the risk from whatever other hosting you're using.)
Why privacy? Your CDN can at the very least read the IPs of your visitors (+ they could add cookies, which will usually not happen but still is a possibility) and you may need to worry about legal issues (e.g. adding it to your GDPR statement).
Finally performance: I guess many people will use a CDN "because it's faster" - except it's not. In the past there used to be the argument that the JS may already be cached. However that's no longer the case as for security reasons browsers partition the cache based on the origin. It's always faster to fetch a resource over the same connection compared to opening up a new connection to a different host.
So: Don't fetch your JS from CDNs, just put it in your webspace and fetch it from there.
Yes, there are methods that allow you to fall-back to your own local copy if the CDN copy fails to arrive, but that is adding further complication and could effectively slow down your page/app by getting in the way of some “make the initial render as early as possible” techniques.
I never liked the idea of loading code from CDNs, or anything else for that matter, even when the benefits seemed more real than they do now.
> Finally performance: … there used to be the argument that the JS may already be cached
For larger CDNs with the resources to have many widely distributed points of presence there is a potential benefit that is still relevant even with partitioned caching: multi-view DNS or any-cast routing can result in a user's request being served from a topologically closer host than otherwise.
Though I suspect that is you care that much about this small benefit you are probably running your own multi-homed infrastructure using the same technique(s) so CDNs aren't needed for you to benefit from them. And back to reliability: such methods further abstract away from the simple “make request, get response from expected host” arrangement and make diagnosing problems further down the line more difficult.
If the CDN server is compromised and the library edited then the resource no longer loads.
Maybe paths like that and symlinks should be behind flags that are disabled by default? Specially if you are extracting content that you didn't generate.
I think all tar utilities that are in common usage disable extracting those sorts of paths by default.
It just happens that the go stdlib includes a tar implementation that was written from scratch (the go authors are allergic to re-using any C code and will NIH as much as they can).. and they probably don't want to fix it now because the go1 promise heavily discourages making changes like that, even if security is an exception to it.
As a random data point, since go and rust are often compared, the most popular tar extraction library in rust has a secure default for this: https://docs.rs/tar/0.4.35/tar/struct.Archive.html#method.un...
It's not about NIH, it's about this being a low-level tar parsing library, not a ready-to-use unpack tool or even a library with unpack functionality. I would never expect such a library to do any sort of automagic path mangling for me unless explicitly stated!
I see the same behaviour in libarchive (C library), microtar (C library), and calccrypto/tar (C _tool_ that is vulnerable to this). How would have re-using any C code helped here, if the top three (from Google search results) C tar libraries have the exact same behaviour? The same behaviour is in Python's tarfile, too.
Plus, even with Go1 stability guarantees, there's nothing preventing the Go standard library from adding a struct method like .SafePath() if it's deemed necessary - standard library types regularly get new fields/methods added to them.
> the most popular tar extraction library in rust has a secure default for this
... only on the high-level unpack methods. The low-level unsafe stuff is still there: https://docs.rs/tar/0.4.35/tar/struct.Entry.html (.path, .unpack). And that's basically the API level which Go's archive/tar provides. If that crate didn't provide unpack functionality, it would have been the exact same kind of footgun.
I can appreciate this but I disagree; the defaults should be the exact reverse -- do the safest thing by default and people who "know what they're doing" (famous last words IME) should have a path to do the less safe thing.
I lean towards having programming languages that are developed with the most mistake-prone 20-40% of programmers in mind, rather than the absolute best software engineers, though I certainly understand the hesitance to have languages come with training wheels by default.
a) someone uses this library for validating whether files contained are conforming a given layout, then passes the tarball to tar - differences in behaviour (tar strips leading .., while eg. the Rust module ignores all files with '..', no matter the position in the path) might cause a security issue.
b) silent removal of 'invalid' paths might cause unexpected behavior for when these paths actually are valid and expected (you might not have used ../ in tar, but I have); can imagine one hell of a debugging session coming from that one. I can also imagine codebases that filter tars for malware, and if they happen to skip ../-paths (because of the tar parser silently skipping them), they could let through the tarball unaffected, with some files unscanned, but extractable in GNU tar.
c) silent rewriting of 'invalid' paths might cause inconsistency between the behavior of different tar libraries, or might cause internal inconsistency due to two 'invalid' paths sanitizing to a single valid path; imagine someone relying on ../-stripping behaviour at some point, then that behavior changing due to switching libraries or a rearrangement of the files in the source tarball. Sanitizing is a complex issue, and any kind of universal logic might actually introduce more bugs than it solves.
d) false sense of security for more complex classes of bugs, like extracting a tarball with a symlink pointing outside of the root of the archive. What should the default behaviour be then, and why? Sometimes symlinking to `/etc` is exactly what the user needs, sometimes it's going to cause a bug further down the line. What about relative symlinks? What about cross-filesystem symlinks? What about the setuid bit? If you make users expect the library to do the right thing, it might cause them to think even less about possible security implications of more tricky situations. Writing user-controlled data to the filesystem is _always_ dangerous and there is no right way to do it, it all depends on the context.
I agree that the default behaviour of an _extraction_ library should probably be to sanitize paths. However, I cannot agree that this should be the default behavior of a thin file parser library, like archive/tar. The only thing Go should do, IMO, is explicitly state in documentation that acting on user-supplied tarballs is dangerous and great consideration should be taken before blindly extracting files to the local filesystem.
I can think of dozens of edge cases in all kinds of security controls that remain security weaknesses despite a large chunk of problems being addressed. The point is to address the bulk of the cases and not allow perfect to become the enemy of good.
> a) someone uses this library for validating whether files contained are conforming a given layout, then passes the tarball to tar - differences in behaviour (tar strips leading .., while eg. the Rust module ignores all files with '..', no matter the position in the path) might cause a security issue.
I would argue that any library that does not already fully resolve relative paths to do this is already faulty and this makes it no less safe. To draw an example, if at the time I pass in a relative path to say, "all files should exist under this directory" and provide "/tmp/somedir/../../usr/lib/" as that parameter, I would expect that this already resolves to "/usr/lib/" before doing any processing. This would eliminate the entire problem presented here.
> b) silent removal of 'invalid' paths might cause unexpected behavior for when these paths actually are valid and expected (you might not have used ../ in tar, but I have); can imagine one hell of a debugging session coming from that one. I can also imagine codebases that filter tars for malware, and if they happen to skip ../-paths (because of the tar parser silently skipping them), they could let through the tarball unaffected, with some files unscanned, but extractable in GNU tar.
I agree that silent removal is a problem. There should be errors/warnings attached to indicate that this has been done. This isn't a good counterpoint to actually having a sane default though. The very issue here is we have a clear security bug introduced in part due to the fact that the library does something seemingly unexpected precisely because it does not line up with existing tooling.
> c) silent rewriting of 'invalid' paths might cause inconsistency between the behavior of different tar libraries, or might cause internal inconsistency due to two 'invalid' paths sanitizing to a single valid path; imagine someone relying on ../-stripping behaviour at some point, then that behavior changing due to switching libraries or a rearrangement of the files in the source tarball. Sanitizing is a complex issue, and any kind of universal logic might actually introduce more bugs than it solves.
Again, I'm not advocating it be an invisible process, just that the default should be the safer of the two options for the bulk of cases.
> d) false sense of security for more complex classes of bugs, like extracting a tarball with a symlink pointing outside of the root of the archive. What should the default behaviour be then, and why? Sometimes symlinking to `/etc` is exactly what the user needs, sometimes it's going to cause a bug further down the line. What about relative symlinks? What about cross-filesystem symlinks? What about the setuid bit? If you make users expect the library to do the right thing, it might cause them to think even less about possible security implications of more tricky situations. Writing user-controlled data to the filesystem is _always_ dangerous and there is no right way to do it, it all depends on the context.
What you describe is not a code problem but a programmer / documentation problem, and is really about the idea that fixing one problem does not inherently fix another problem that it's not necessarily aimed at fixing. Clear documentation that outlines the steps that are taken, as well as the option to disable the feature if truly needed (as I suggested) would solve this entirely.
Also, this is a weird argument to make given this already exists with existing tooling. I expect if I name a file "../../../../etc/cron.hourly/evil.sh&qu...
Though also, "we've seen this cause exploits in the wild multiple times, let's just let this footgun sit for 3 years".
They could have at least added a documentation note to the archive/tar page that there's a security issue you have to handle, which is what the unsafe version of the rust unarchive methods do. 3 years of not even updating docs is kinda unfortunate.
Unlike the go library, on the unsafe methods, the rust library includes a note explaining the security footgun.
It also provides 'unpack_in' on entry, so there's a secure and insecure variant.
The go library should at least include a note, but does not. The rust library does not have the same footgun because it is clearly documented and provides a secure alternative.
Indeed. See https://docs.github.com/en/code-security/secret-security/abo... for more details on how it works.
"The decision to pay a reward is entirely at our discretion. You must not violate any law. You are responsible for any tax implications or additional restrictions depending on your country and local law. We reserve the right to cancel this program at any time."
(https://www.cloudflare.com/disclosure/)
That commit log should give you a better sense of what happened if you (like me) didn't understand how cdnjs works. Apparently robocdnjs will just pull arbitrary packages and unzip and commit them into the cdnjs repo which then gets served all over the internet. Crazy!
> April 6, 2021 20:30 [JST] cdnjs processed the file
you can see commits in the cdnjs/cdnjs git log adding an absurdly named test package from one of the maintainers to test packages containing symlinks within 24 hours after the exploit
1) The biggest benefit of using cdn was that scripts stylesheets were often loaded from cache. This is no longer the case anymore.
2) It's adding an unnecessary vulnerability as seen from this case.
3) CDN scripts can be (possibly) used to track your website traffic.
4) They often drop third party cookies which decreases your website load speed among other things.
It's best to just host your script on Digital ocean spaces or S3 and use a Cloudfront proxy.
Extra cookies and scripts are related to specific features. A good CDN will let you configure exactly what you want.
Cloudfront is a CDN.
EDIT: The issue is more related to using a third party domain in order to load assets on you website. I agree, that should be done from first party when possible.
I mean, CloudFront is a CDN too.
EDIT: Oh, I misread. You weren't dismissing CDNs entirely. Only this type of CDNs. In that, I agree.
> 1) The biggest benefit of using cdn was that scripts stylesheets were often loaded from cache. This is no longer the case anymore.
This is indeed true. However, most websites (despite the relative ease of activating a CDN service) only host their website in a single location. In those cases, library CDNs (to differentiate from other uses of CDNs) can still benefit the website, even when you consider optimisation strategies such as HTTP/2. Additionally, the stress on their servers are decreased because it's handled by their library CDN provider.
> 2) It's adding an unnecessary vulnerability as seen from this case.
> 3) CDN scripts can be (possibly) used to track your website traffic.
Good points and something that I can't rebut easily. It is indeed on the onus of the website to check the reputation of library CDNs, and it may violate privacy guarantees (you're trusting Google/CloudFlare/Microsoft/Amazon/Fastly/MaxCDN etc.) Specifically on 2) however, they knew that they are large: the speed of fixes (as shown) here proves that library CDNs can benefit websites' security, which tends that its software stack is updated slower (by the website operator's fault). You can also leverage subresource integrity (https://developer.mozilla.org/en-US/docs/Web/Security/Subres...) to fail hard.
> 4) They often drop third party cookies which decreases your website load speed among other things.
Citation needed here. As far as I checked (and just verified as I post this, to ensure that this is still true), every library CDNs I knew (even in weird places such as Japan- and China-centric library CDNs) do not place cookies in the library URLs themselves (some do on their websites themselves, but it is usually separated by domain names).
> It's best to just host your script on Digital ocean spaces or S3 and use a Cloudfront proxy.
If you're a large website or basically need to ensure the privacy is guaranteed, sure! I see that large websites do in fact use their own choice of CDN providers (most websites including Amazon themselves uses Fastly though). Some even do a step-up: some websites (for example) use Fastly for their American traffic, KeyCDN for European traffic and Quantil for Asian traffic. However, I only knew of two CDN providers which provides a flat-rate and reasonable costs: CloudFlare (uhh, CDNJS operator) and DDOS-Guard (which some consider shady).
HTTP/2 suffers from Head-of-Line blocking, which may impede rather than help speed up the connection. In most cases, this doesn't happen, but when they do happen it's consistent. Testing is required, because there are cases where HTTP/2 backfires spectacularly.
> That assets that never change are cached
Fair point, I also strongly recommend this. Plus, if your server supports statically compressing resources ahead of time it pays serious dividends (and activating Brotli, if you can custom compile applications or lucky to use a distribution that built it).
> And if possible use a hosting provider in the same country your users are located.
In Europe, this is relatively easy. In the US and Russia, probably two servers (for East and West coast/okrugs). In Asia? Surprisingly difficult, unless you're in East Asia (I'm not discussing why here since there's plenty of resources to learn more about why it's a pain in the rear). Africa? Actually, despite improvements in connectivity I strongly recommend to make the pages small and simple (as the requirements allow).
One thing that has become an issue, is s3 doesn't let add custom headers, eg. CSP, CORP etc.
We are considering firebase hosting or our own origins behind a CDN for more control over headers. Anyone have any other suggestions?
I've been running several sites off Firebase Hosting the last 4 years and have seen surprisingly high downtime, cache clearing errors, and unexplained 502 errors from Firebase Hosting.
It's a privacy-oriented extension first, but it may still protect from an eventual zero-day like this.
So, let me see if I get this straight...
cdnjs is a CDN for javascript libraries. cdnjs has a "library update server", in which an "auto-update script" runs periodically to update the libraries that the CDN offers. If it finds a new version of a library, it downloads the files from the library's git repository, and it publishes them on the CDN.
There can be symlinks in git repositories, but the "auto-update script" doesn't handle them correctly. And that's the core of the vulnerability.
If you own the git repository of a library published by cdnjs, you could add a symlink to the git repository. The symlink could point to a file that you want to access on the cdnjs "library update server". Then, as soon as the "auto-update script" runs, the file's contents will be public to everyone on the CDN.
But how would you manage to create a malicious library and have it approved by cdnjs? That's the part of the exploit that sounds the hardest, and I don't see a reasonable way to go about it.
1. Hijack an existing library to publish the malicious version
2. or, Have us approve what looks like a legitimate library, which later publishes a malicious version
Hijack or buy outright an existing library. IIRC, there have been many such instances where existing projects have been sold for large sums to questionable parties.
And that's what the author exploits by registering a library that have a symlink in it. Which then got excuted.
I submitted a working proof-of-concept that would allow you to use Cloudflare's page-customizing preview function to create a Cloudflare phishing site.
The issue was closed as "accepted as the nature of the cloudflarepreview playground".
Then they went ahead and fixed the issue.
Not the end of the world, but definitely didn't inspire confidence in the program.
Last year I submitted a working proof of concept with detailed explanations of how you can steal passwords and details of users as an app maker on Shopify, and it got denied for a similar reason (though I dont think it's been fixed). I've since stopped wasting my time on such pursuits.
Exploitation has such a rich history between sellers, buyers, and users; the marketplace is way more sophisticated than what it was in the 90s.
You can probably get $5k to $20k for the exploit to Shopify. There are plenty of nationstate actors that create honeypots for a variety of targets. I'm sure Shopify exploits are worth something to someone.
But yeah, it probably is
https://en.wikipedia.org/wiki/Market_for_zero-day_exploits
Selling exploits use to be way more common prior to 2010. Previously many companies wouldn't pay for bug fixes and would periodically sue developers for disclosing issues.
There eventually became a large movement within the cyber security groups about no longer giving bugs away for free and demanding fair market value for exploits (imagine the NSA paying $100k for a zero-day, versus the company giving you $500):
https://threatpost.com/no-more-free-bugs-software-vendors-03...
Nowadays the highly sought out targets like Android or iOS have premiums for zero-days (upwards to $1mil), in-fact it wasn't until Google started paying more for exploits was when Android security started getting better. But if some companies aren't willing to pay fair market value for their exploits, why should it matter if you sell it to someone else?
Not taking part and allowing that system to be exploited is the safest easiest way to fix these programs. The only legal way.
I hope more people just stop taking part so they are forced to change into something honest.