My reading of OP is that it’s less about whether zopfli is technically the best way to achieve a 5% reduction in package size, and more about how that relatively simple proposal interacted with the NPM committee. Do you think something like this would fare better or differently for some reason?
I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes).
It would have been interesting to see what eventually played out if the author hadn't closed the RFC themselves. It could have been the sort of thing that eventually happens after 2 years, but then quietly makes everybody's lives better.
I felt the same. The proposal wasn't rejected! Also, performance gains go beyond user stories - e.g. they reduce infra costs and environmental impact - so I think the main concerns of the maintainers could have been addressed.
They soft-rejected by requiring more validation than was reasonable. I see this all the time. "But did you consider <extremely unlikely issue>? Please go and run more tests."
It's pretty clear that the people making the decision didn't actually care about the bandwidth savings, otherwise they would have put the work in themselves to do this, e.g. by requiring Zopfli for popular packages. I doubt Microsoft cares if it takes an extra 2 minutes to publish Typescript.
Kind of a wild decision considering NPM uses 4.5 PB of traffic per week. 5% of that is 225 TB/week, which according to my brief checks costs around $10k/week!
I guess this is a "not my money" problem fundamentally.
This doesn't seem quite correct to me. They weren't asking for "more validation than was reasonable". They were asking for literally any proof that users would benefit from the proposal. That seems like an entirely reasonable thing to ask before changing the way every single NPM package gets published, ever.
I do agree that 10k/week is non-negligible. Perhaps that means the people responsible for the 10k weren't in the room?
massively increase the open source github actions bill for runners running longer (compute is generally more expensive) to publish for a small decrease in network traffic (bandwidth is cheap at scale)?
I agree, going from 1 second to 2.5 minutes is a huge negative change, in my opinion. I know publishing a package isn't something you do 10x a day but it's probably a big enough change that, were I doing it, I'd think the publish process is hanging and keep retrying it.
Since it's backwards compatible, individual maintainers could enable it in their own pipeline if they don't have issues with the slowdown. It sounds like it could be a single flag in the publish command.
Probably not worth the added complexity, but in theory, the package could be published immediately with the existing compression and then in the background, replaced with the Zopfli-compressed version.
I don't think that's actually a problem, but it would require continuing to host both versions (at distinct URLs) for any users who may have installed the package before the Zopfli-compressed version completed. Although I think you could also get around this by tracking whether the newly-released package was ever served by the API. If not, which is probably the common case, the old gzip-compressed version could be deleted.
> Probably not worth the added complexity, but in theory, the package could be published immediately with the existing compression and then in the background, replaced with the Zopfli-compressed version.
Checksum matters aside, wouldn't that turn the 5% bandwidth savings into an almost double bandwidth increase though?
IMHO, considering the complexity to even make it a build time option, the author made the right call.
> I don't find the cons all that compelling to be honest
I found it reasonable.
The 5% improvement was balanced against the cons of increased cli complexity, lack of native JS zopfli implementation, and slower compression .. and 5% just wasn't worth it at the moment - and I agree.
>or at least I think they warrant further discussion
Yes, but there’s a difference between “this warrants further discussion” and “this warrants further discussion and I’m closing the RFC”. The latter all but guarantees that no further discussion will take place.
"I don't find the cons all that compelling to be honest"
This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm.
You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to make any changes, until you experience your very own "oh man, I really thought this change was perfectly safe and now my entire customer base is trashed" moment, and then suddenly things like "hey, we need to consider how this affects old signatures and the speed of decompression and just generally whether this is worth the non-zero risks for what are in the end not really that substantial benefits".
I do not say this as the wise Zen guru sitting cross-legged and meditating from a position of being above it all; I say it looking at my own battle scars from the Perfectly Safe things I've pushed out to my customer base, only to discover some tiny little nit caused me trouble. Fortunately I haven't caused any true catastrophes, but that's as much luck as skill.
Attaining the proper balance between moving forward even though it incurs risk and just not changing things that are working is the hardest part of being a software maintainer, because both extremes are definitely bad. Everyone tends to start out in the former situation, but then when they are inevitably bitten it is important not to overcorrect into terrified fear of ever changing anything.
I'm saying you probably don't find them compelling because from your point of view, the problems don't look important to you. They don't from my point of view either. But my point of view is the wrong point of view. From their point of view this would be plenty to make me think twice and several times over past that from changing something so deeply fundamental to the system for what is a benefit that nobody who is actually paying the price for the package size seems to be particularly enthusiastic about. If the people paying the bandwidth bill aren't even that excited about a 5% reduction, then the cost/benefits analysis tips over into essentially "zero benefit, non-zero cost", and that's not very compelling.
Or you're not understanding how he meant it: there are countless ways to roll out such changes, a hard change is likely a very bad idea as you've correctly pointed out.
But it is possible to do it more gradually, I.e. by sneaking it in with a new API that's used by new npm version or similar.
But it was his choice to make, and it's fine that he didn't feel enough value in pursuing such a tiny file size change
Yes and no. If I'm paying $5 a month for storage, I probably don't care about saving 5% of my storage costs. If I'm paying $50,000/month in storage costs, 5% savings is a lot more worthwhile to pursue
Doesn't npm belong to Microsoft? It must be hosted in Azure which they own so they must be paying a rock bottom rate for storage, bandwidth, everything.
Maybe, maybe not. If you are on a bandwidth limited connection and you have a bunch of NPM packages to install, 5% of an hour is a few minutes saved. It's likely more than that because long-transfers often need to be restarted.
You'd keep 5c. A significant number of people who find sums up around $2500 give it back unconditionally, with no expectation of reward. Whoever lost $2500 is having a really bad day.
Why do so many people take illustrative examples literally?
I'm sure you can use your imagination to substitute "lunch" and "car" with other examples where the absolute change makes a difference despite the percent change being the same.
Even taking it literally... The 5% might not tip the scale of whether or not I can purchase the car, but I'll spend a few hours of my time comparing prices at different dealers to save $2500. Most people would consider it dumb if you didn't shop around when making a large purchase.
On the other hand, I'm not going to spend a few hours of my time at lunch so that I can save an extra $1 on a meal.
- Doing 1 hour of effort to save 5% on your $20 lunch is foolhardy for most people. $1/hr is well below US minimum wage.
- Doing 1 hour of effort to save 5% on your $50k car is wise. $2500/hr is well above what most people are making at work.
It's not about whether the $2500 affects my ability to buy the car. It's about whether the time it takes me to save that 5% ends up being worthwhile to me given the actual amount saved.
The question is really "given the person-hours it takes to apply the savings, and the real value of the savings, is the savings worth the person-hours spent?"
If you can get the exact same result for less cost (time and money), why not? Things like enjoyment don't factor in since they can't be directly converted into money.
Those lunches could add up to something significant over time. If you're paying $10 per lunch for 10 years, that's $36,500 which is pretty comparable to the cost of a car.
How often are individuals publishing to NPM? Once a day at most, more typically once a week or month? A few dozen seconds of one person's day every month isn't a terrible trade-off.
Even that's addressable though if there's motivation, since something like transcoding server side during publication just for popular packages would probably get 80% of the benefit with no client-side increase in publication time.
In a great example of the Pareto Principle (80/20), or actually even more extreme, let's only apply this Zopfli optimization if the package download total is equal or more than 1GiB (from the Weekly Traffic in GiB column of the Top 5000 Weekly by Traffic tab of the Google Sheets file from the reddit post).
For reference, total bandwidth used by all 5000 packages is 4_752_397 GiB.
Packages >= 1GiB bandwidth/week - That turns out to be 437 packages (there's a header row, so it's rows 2-438) which uses 4_205_510 GiB.
So 88% of the top 5000 bandwidth is consumed by downloading the top 8.7% (437) packages.
5% is about 210 TiB.
Limiting to the top 100 packages by bandwidth results in 3_217_584 GiB, which is 68% of total bandwidth used by 2% of the total packages.
In some scenarios the equation flips, and the enterprise is looking for _more_ scale.
The more bandwidth that Cloudflare needs, the more leverage they have at the peering table. As GitHub's largest repo (the @types / DefinitelyTyped repo owned by Microsoft) gets larger, the more experience the owner of GitHub (also Microsoft) gets in hosting the world's largest git repos.
I would say this qualifies as one of those cases, as npmjs is hosted on Azure. The more resources that NPM needs, the more Microsoft can build towards parity with AWS's footprint.
The pros aren't all that compelling either. The npm repo is the only group that this would really be remotely significant for, and there seemed to be no interest. So it doesn't take much of a con to nix a solution to a non-problem.
Every single download, until the end of time is affected: It speeds up the servers, speeds up the updates, saves disk space on the update servers, and saves on bandwidth costs and usage.
Everyone benefits, the only cost is a ultra microscopic time on the front end, and a tiny cost on the client end, and for a very significant number of users, time and money saved. The examples of compression here...
Plus a few years of a compression expert writing a JS implementation of what was likely some very cursed C. And someone auditing its security. And someone maintaining it.
> I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion
It needs a novel JS port of a C compresison library, which will be wired into a heavily-used and public-facing toolchain, and is something that will ruin a significant number of peoples' days if it breaks.
For me, that kind of ask needs a compelling use case from the start.
Congrats on a great write-up. Sometimes trying to ship something at that sorta scale turns out to just not really make sense in a way that is hard to see at the beginning.
Another personal win is that you got a very thorough understanding of the people involved and how the outreach parts of the RFC process works. I've also had a few fail, but I've also had a few pass! Always easier to do the next time
Comparing to gzip isn't really worth it. Combine pigz (threaded) with zlib-ng (simd) and you get decent performance. pigz is used in `docker push`.
For example, gzipping llvm.tar (624MB) takes less than a second for me:
$ time /home/harmen/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-13.2.0/pigz-2.8-5ptdjrmudifhjvhb757ym2bzvgtcsoqc/bin/pigz -k hello.tar
real 0m0.779s
user 0m11.126s
sys 0m0.460s
At the same time, zopfli compiled with -O3 -march=native takes 35 minutes. No wonder it's not popular.
It is almost 2700x slower than the state of the art for just 6.8% bytes saved.
In my opinion even the 28x decrease in performance mentioned would be a no-go. Sure the package saves a few bytes but I don't need my entire pc to grind to a halt every time I publish a package.
Besides, storage is cheap but CPU power draw is not. Imagine the additional CO2 that would have to be produced if this RFC was merged.
> 2 gigabytes of bandwidth per year across all installations
This must be a really rough estimate and I am curious how it was calculated. In any case 2 gigabytes over a year is absolutely nothing. Just my home network can produce a terabyte a day.
Because the authors mentioned package, Helmet[1], is 103KB uncompressed and has had 132 versions in 13 years. Meaning downloading every Helmet version uncompressed would result in 132*103KB = 13.7MB.
I feel like I must be missing something really obvious.
Props to anyone who tries to make the world a better place.
Its not always obvious who has the most important use cases. In the case of NPM they are prioritizing the user experience of module authors. I totally see how this change would be great for module consumers, yet create potentially massive inconvenience for module authors.
I think "massive" is overstating it. I don't think deploying a new version of a package is something that happens many times a day, so it wouldn't be a constant pain point.
Also, since this is a case of having something compressed once and decompressed potentially thousands of times, it seems like the perfect tool for the job.
Module authors generally have fairly large test suites which are run often- sometimes on each file save. If you have a 1 or 2 second build script its not a huge deal. If that script starts taking 30-60 seconds- you have just hosed productivity. Also you have massively increased the load on your CI server- possibly bumping you out of a free tier.
The fix would then have to be some variation of:
a) Stop testing (so often)
b) Stop bundling before testing
c) Publish to a different package manager
- all of which would affect the overall quality and quantity npm modules.
You have to test that your bundle _actually works_, especially if you are using non-standard compression.
But yes- you could bundle less, but that would be a disadvantage, particularly if a bundle suddenly fails and you don’t know which change caused it. But maybe that’s not a big deal for your use case.
50% size savings isn't important to the people who pay for it. They pay at most pennies for 100% savings (that is somehow all the functionality in zero bytes - not worth anything to those paying the bills)
Size savings translates to latency improvements which directly affects conversion rates. Smaller size isn’t about reducing costs but increased revenue. People care.
Note that this proof-of-concept implementation saves latency on first load, but may add latency at surprising points while using the website. Any user invoking a rarely-used function would see a delay before the javascript executes, without the traditional UI affordances (spinners etc) to indicate that the application was waiting on the network. Further, these secretly-slow paths may change from visit to visit. Many users know how to "wait for the app to be ready," but the traditional expectation is that once it's loaded, the page/app will work, and any further delays will be signposted.
I'm sure it works great when you've got high-speed internet, but might break things unacceptably for users on mobile or satellite connections.
> without the traditional UI affordances (spinners etc) to indicate that the application was waiting on the network.
This part is obviously trivially solvable. I think the same basic idea is going to at some point make it but it’ll have to be through explicit annotations first and then there will be tooling to automatically do this for your code based upon historical visits where you get to tune the % of visitors that get additional fetches. Also, you could probably fetch the split off script in the background anyway as a prefetch + download everything rather than just 1 function at a time (or even downloading related groups of functions together)
The idea has lots of merit and you just have to execute it right.
This would cause the bundler to inject a split point & know how to hide that + know what needs bundling and what doesn’t. GWT pioneered almost 20 years ago although not a fan of the syntax they invented to keep everything running within stock Java syntax: https://www.gwtproject.org/doc/latest/DevGuideCodeSplitting....
The tooling producing such annotaion based on historical visits whould constantly change the annotated set of functions, I suspect, whith app versions evolving.
Note, the inactive code parts are very often in 3rd party libraries.
Some are also in your own libraries shared between your different application. So the same shared library whould need one set of split-annotated functions for one app, and another set for another app. So these conflicting sets of annotations can not live in the library source code simultaneously.
For example the automated system could take your non-split codebase & inject splits at any async function & self-optimize as needed. The manual annotation support would be to get the ecosystem going as an initial step because I suspect targeting auto-optimizing right from the get go may be too big a pill to swallow.
Only the first user who hits a rarely used execution point may experience the noticeable latency, if he also has slow internet, etc.
As soon as the user executes a rarely used function, the information is about this fact is sent to the server and it includes this function into the active set to be send to future users.
In the video I manually initiate re-generation of the "active" and the "rest" scripts, but the most primitive MVP was supposed to schedule re-generation of the scripts when receiving info that some previously unseen functions are executed in browser.
Obviously, if the idea is developed further, the first user's experience may also be improved - a spinner shown. Pre-loading the inactive set of functions in background may also be considered (pros: it allows to avoid latency for users ho invoke rare functionality, cons: we lose the savings of the traffic and browser memory and cpu for compiling the likely unneded code).
(BTW, further development of the idea includes to splitting the code more granularity than just "active" / "inactive". E.g. active in the first 5 seconds after opening the page loaded immediately, likely to be active soon, active but rarely called - the later two these parts definitely need to be loaded in background)
Who? anyone who is on a slow internet connection or who has a slow device can tell you they don't care. Or maybe they do but features are far more important. I guess if things are slow on a top of the line device withia fast connection they would care.
If we consider overall Internet traffic, which is dominated by video and images, the size of javascript transmitted is negligible.
Savings of memory and cpu for end user's browser? Maybe, but our software at all layers is so bloated, without need, just carelessly, that I'm not sure working of such javascript savings is useful - any resources saved will be eaten by something else immediately.
For an application developer or operator, the 50% savings of javascript size are probably not worth dealing with some tool that dynamically prunes the app code, raising questions about privacy and security. I though through the security and privacy questions, but who would want to even spend attention on these considerations?
As I mention in the README at github, a Microsoft researcher investigated the same approach earlier, but Microsoft haven't taken it anywhere.
There was a commercial company offering this approach of javascript minification as a product, complete product.
As well as other optimizations, like image size, etc. Their proxy embedded special "agent" code into the app which inspected the device and reported to server what image sizes are optimal for user, what js functions are invoked. And the server prepared optimized versions of app for various devices. Javascript was "streamed" in batches of only functions needed by the app.
Now the company is dissolved - that's why the website is unavailable. Wikipedia says they were bought out by Akamai - https://en.wikipedia.org/wiki/Instart. But I don't see any traces of this approach in the today's Akamai offerings.
I contacted several companies, in CDN business and others, trying to interest them in the idea and get very modest funding for a couple more months of my time to work on this (I was working on this in the end of a long break from payed work and was running out of savings). Didn't find anyone ready to take part.
This all may be signs that possibility of such an optimization is not valuable enough for users.
I've not done it, but have you considered using `pnpm` and volume-mounting a shared persistent `pnpm-store` into the containers? It seems like you'd get near-instant npm installs that way.
The only time npm install was on the critical path was hotfixes. It’s definitely worth considering. But I was already deep into doing people giant favors that they didn’t even notice, so I was juggling many other goals. I think the only thank you I got was from the UI lead, who had some soda straw internet connection and this and another thing I did saved him a bunch of hard to recover timeouts.
I wonder if it would make more sense to pursue Brotli at this point, Node has had it built-in since 10.x so it should be pretty ubiquitous by now. It would require an update to NPM itself though.
It only doesn't apply to existing versions of existing packages. Newer releases would apply Zopfli, so over time likely the majority of actively used/maintained packages would be recompressed.
That's not actually so straightforward. You pay the 10-100x slowdown once on the compressing side, to save 4-5% on every download - which for a popular package one would expect downloads to be in the millions.
Most packages don‘t publish every CI build. Some packages, such as Typescript, publish a nightly build once a day. Even then a longer compression time dosen‘t seem too bad.
Caching downloads on a CDN helps offload work from the main server, it doesn't meaningfully change the bandwidth picture from the client's perspective.
As the author himself said, just React was downloaded half a billion times; that is a lot of saved bandwidth on both sides, but especially so for the server.
Maybe it would make sense to only apply this improvement in images that are a) either very big or b) get downloaded at least million times each year or so. That would cover most of the savings while leaving most packages and developers out of it.
Assuming download and decompression cost to be proportional to the size of the incoming compressed stream, it would break even at 2000 downloads. A big assumption I know, but 2000 is a really small number.
Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.
You'll also see the benefit when `rm -rf`ing a `node_modules` and re-installing, as pnpm still has a local copy that it can re-link after validating its integrity.
As someone who mostly works in Java it continues to floor me that this isn’t the default. Why does every project I work on need an identical copy of possibly hundreds of packages if they’re the same version?
I also like Yarn pnp’s model of leaving node_modules as zip files. CPUs are way faster than storage, they can decompress on the fly. Less disk space at rest, less disk slack, less filesystem bookkeeping.
Every single filesystem is way faster at dealing with one file than dozens/hundreds. Now multiply that by the the hundreds if does, it add up.
That's on the package publishers, not NPM. They give you an `.npmignore` that's trivially filled out to ensure your package isn't full of garbage, so if someone doesn't bother using that: that's on them, not NPM.
(And it's also a little on the folks who install dependencies: if the cruft in a specific library bothers you, hit up the repo and file an issue (or even MR/PR) to get that .npmignore file filled out. I've helped folks reduce their packages by 50+MB in some cases, it's worth your own time as much as it is theirs)
It's much better to allowlist the files meant to be published using `files` in package.json because you never know what garbage the user has in their folder at the time of publish.
On a typical project with a build step, only a `dist` folder would published.
Not a fan of that one myself (it's far easier to tell what doesn't belong in a package vs. what does belong in a package) but that option does exist, so as a maintainer you really have no excuse, and as a user you have multiple MR/PRs that you can file to help them fix their cruft.
> On a typical project with a build step, only a `dist` folder would published.
Sort of, but always include your docs (readme, changelog, license, and whatever true docs dir you have, if you have one). No one should need a connection for those.
I don't know why, but clipboard libraries tend to be really poorly implemented, especially in scripting languages.
I just checked out clipboardy and all they do is dispatch binaries from the path and hope it's the right one (or if it's even there at all). I think I had a similar experience with Python and Lua scripts. There's an unfunny amount of poorly-written one-off clipboard scripts out there just waiting to be exploited.
I'm only glad that the go-to clipboard library in Rust (arboard) seems solid.
One of the things I like about node_modules is that it's not purely source code and it's not purely build artifacts.
You can read the code and you can usually read the actual README/docs/tests of the package instead of having to find it online. And you can usually edit library code for debugging purposes.
If node_modules is taking up a lot of space across a bunch of old projects, just write the `find` script that recursively deletes them all; You can always run `npm install` in the future when you need to work on that project again.
Yep, I wrote a script that starts at a root `node_modules` folder and iterates through to remove anything not required (dotfiles, Dockerfile, .md files, etc) - in one of our smaller apps this removes about 25Mb of fluff, some packages are up to 60-70mb of crap removed.
I don't see why it wouldn't be possible to hide behind a flag once Node.js supports zopfli natively.
In case of CI/CD, it's totally feasible to just add a --strong-compression flag. In that case, the user expects it to take its time.
TS releases a non-preview version every few months, so using 2.5 minutes for compression would work.
Think of the complexity involved, think of having to fix bugs because something went wrong.
Such effort would be better spent preparing npm/node for a future where packages with a lower-bound npm version constraint can be compressed with zstd or similar.
Every feature you add to a program is complexity, you need to really decide if you want it.
A few people have mentioned the environmental angle, but I'd care more about if/how much this slows down decompression on the client. Compressing React 20x slower once is one thing, but 50 million decompressions being even 1% slower is likely net more energy intensive, even accounting for the saved energy transmitting 4-5% fewer bits on the wire.
It's very likely zero or positive impact on the decompression side of things.
Starting with smaller data means everything ends up smaller. It's the same decompression algorithm in all cases, so it's not some special / unoptimized branch of code. It's yielding the same data in the end, so writes equal out plus or minus disk queue fullness and power cycles. It's _maybe_ better for RAM and CPU because more data fits in cache, so less memory is used and the compute is idle less often.
It's relatively easy to test decompression efficiency if you think CPU time is a good proxy for energy usage: go find something like React and test the decomp time of gzip -9 vs zopfli. Or even better, find something similar but much bigger so you can see the delta and it's not lost in rounding errors.
For formats like deflate, decompression time doesn't generally depend on compressed size. (zstd is similar, though memory use can depend on the compression level used).
This means an optimization like this is virtually guaranteed to be a net positive on the receiving end, since you always save a bit of time/energy when downloading a smaller compressed file.
I can speak to this - there is no meaningful decompression effect across an insane set of tested data at Google and elsewhere. Zopfli was invented prior to brotli
Zopfli is easiest to think of as something that just tries harder than gzip to find matches and better encodings.
Much harder.
decompression speed is linear either way.
It's easiest to think of decompression as a linear time vm executor[1], where the bytecoded instructions are basically
go back <distance> bytes, output the next <length> bytes you see, then output character <c>
(outputting literal data is the instruction <0,0,{character to output}>)
Assuming you did not output a file larger than the original uncompressed file (why would you bother?), you will, worst case, process N bytes during decompression, where N is the size of the original input file.
The practical decompression speed is driven by cache behavior, but it thrashes the cache no matter what.
In practice, reduction of size vs gzip occurs by either finding larger runs, or encodings that are smaller than the existing ones.
After all, if you want the compressed file to shrink, you need output less instructions somehow, or make more of the instructions identical (so they can be represented in less bits by later huffman coding).
In practice, this has almost exclusively positive effects on decompression speed - either the vm has less things to process (which is faster), or more of the things it does look the same (which has better cache behavior).
[1] this is one way archive formats will sometimes choose to deal with multiple compression method support - encode them all to the same kind of bytecode (usually some form of copy + literal instruction set), and then decoding is the same for all of them. ~all compression algorithms output some bytecode like the above on their own already, so it's not a lot of work.
This doesn't help you support other archive formats, but if you want to have a bunch of per-file compression options that you pick from based on what works best, this enables you to still only have to have one decoder.
This seems like a place where the more ambitious version that switches to ZSTD might have better tradeoffs. You would get similar or better compression, with faster decompression and recompression than zstd.It would lose backward compatibility though...
Brotli and lzo1b have good compression ratios and pretty fast decompression speeds. Compression speed should not matter that much, since you only do it once.
Not necessarily - could retain backward compat by publishing both gzip and zstd variants and having downloaders with newer npm’s prefer to download zstd. Over time, you could require packages only upload zstd going forward and either generate zstd versions of the backlog of unmaintained packages or at least those that see some amount of traffic over some time period if you’re willing to drop very old packages. The ability to install arbitrary versions of packages probably means you’re probably better off reprocessing the backlog although that may cost more than doing nothing.
The package lock checksum is probably a more solvable issue with some coordination.
The benefit of doing this though is less immediate - it will take a few years to show payoff and these kinds of payoffs are not typically made by the kind of committee decisions process described (for better or worse).
Thats a much higher hurdle to jump. I don’t blame the author for trying this first.
If accepted, it might have been a good stepping stone too. A chance to get to know everyone and their concerns and how they think.
So if you wanted to see how this works (proposal + in prod) and then come back later proposing something bigger by switching off zip that would make sense to me as a possible follow up.
But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do.
For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And I would have quit at the first roadblock.
What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line.
The problem is that this guy is treating open source as if it was a company where you need to direct a project to completion. Nobody in open source wants to be told what to do. Just release your work, if it is useful, the community will pick it up and everybody will benefit. You cannot force your improvement into the whole group, even if it is beneficial in your opinion.
> What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line.
Says who?
Says an engineer? Says a product person?
NPM is a company with 14 employees; with a system integrated into countless extremely niche and weird integrations they cannot control. Many of those integrations might make a professional engineer's hair catch fire - "it should never be done this way!" - but the real world is that the wrong way is the majority of the time. There's no guarantee that many of the downloads come from the official client, just as one example.
The last thing they need, or I want, or any of their customers want, or their 14 employees need, is something that might break backwards compatibility in an extremely niche case, anger a major customer, cause countless support tickets, all for a tiny optimization nobody cares about.
This is something I've learned here about HN that, for own mental health, I now dismiss: Engineers are obsessed with 2% optimizations here, 5% optimizations there; unchecked, it will literally turn into an OCD outlet, all for things nobody in the non-tech world even notices, let alone asks about. Just let it go.
Open source needs to operate differently than a company because people don’t have time/money/energy to deal with bullshit.
Hell. Even 15 employees larping as a corporation is going to be inefficient.
what you and NPM are telling us, is that they are happy to take free labor, but this is not an open source project.
> Engineers are obsessed with 2% optimizations here
Actually in large products these are incredible finds.
But ok. They should have the leadership to know which bandwidth tradeoffs they are committed to and tell him immediately it’s not what they want, rather than sending him to various gatekeepers.
Correct; NPM is not an "open source project" in the sense of a volunteer-first development model. Neither is Linux - over 80% of commits are corporate, and have been for a decade. Neither is Blender anymore - the Blender Development Fund raking in $3M a year calls the shots. Every successful "large" open source project has outgrown the volunteer community.
> Actually in large products these are incredible finds.
In large products, incredible finds may be true; but breaking compatibility with just 0.1% of your customers is also an incredible disaster.
But NPM has no proof their dashboard won't light up full of corporate customers panicking the moment it goes to production; because their hardcoded integration to have AWS download packages and decompress them with a Lambda and send them to an S3 bucket can no longer decompress fast enough while completing other build steps to avoid mandatory timeouts; just as one stupid example of something that could go wrong. IT is also demanding now that NPM fix it rather than modify the build pipeline which would take weeks to validate, so corporate's begging NPM to fix it by Tuesday's marketing blitz.
Just because it's safe in a lab provides no guarantee it's safe in production.
That’s an argument against making any change to the packaging system ever. “It might break something somewhere” isn’t an argument, it’s a paralysis against change. Improving the edge locality of delivery of npm packages could speed up npm installs. But speeding up npm installs might cause the CI system which is reliant on it for concurrency issues to have a race condition. Does that mean that npm can’t ever make it faster either?
This attitude is how in an age with gigabit fiber, 4GB/s hard drive write speed, 8x4 GHz cores with simd instructions it takes 30+ seconds to bundle a handful of files of JavaScript.
NPM is a webservice. They could package the top 10-15 enhancements call it V2. When 98% of traffic is V2 turn V1 off. Repeat every 10 years or so until they work their way into having a good protocol.
> Engineers are obsessed with 2% optimizations here, 5% optimizations there; unchecked, it will literally turn into an OCD outlet, all for things nobody in the non-tech world even notices, let alone asks about. Just let it go.
I absolutely disagree with you. If the world took more of those 5% optimisations here and there everything would be faster. I think more people should look at those 5% optimisations. In many cases they unlock knowledge that results in a 20% speed up later down the line. An example from my past - I was tasked with reducing the running speed of a one shot tool we were using at $JOB. It was taking about 15 minutes to run. I shaved off seconds here and there with some fine grained optimisations, and tens of seconds with some modernisation of some core libraries. Nothing earth shattering but improvements none the less. One day, I noticed a pattern was repeating and I was fixing an issue for the third time in a different place (searching a gigantic array of stuff for a specific entry). I took a step back and realised that if I replaced the mega list with a hash table it might fix every instance of this issue in our app. It was a massive change, touching pretty much every file. And all of a sudden our 15 minute runtime was under 30 seconds.
People used this tool every day, it was developed by a team of engineers wildly smarter than me. But it had grown and nobody really understood the impact of the growth. When it started that array was 30, 50 entries. On our project it was 300,000 and growing every day.
Not paying attention to these things causes decay and rot. Not every change should be taken, but more people should care.
I prevent cross-site scripting, I monitor for DDoS attacks, emergency database rollbacks, and faulty transaction handlings. The Internet heard of it? Transfers half a petabyte of data every minute. Do you have any idea how that happens? All those YouPorn ones and zeroes streaming directly to your shitty, little smart phone day after day? Every dipshit who shits his pants if he can't get the new dubstep Skrillex remix in under 12 seconds? It's not magic, it's talent and sweat. People like me, ensuring your packets get delivered, un-sniffed. So what do I do? I make sure that one bad config on one key component doesn't bankrupt the entire fucking company. That's what the fuck I do.
I think that the reason NPM responded this way is because it was a premature optimization.
If/when NPM has a problem - storage costs are too high, or transfer costs are too high, or user feedback indicates that users are unhappy with transfer sizes - then they will be ready to listen to this kind of proposal.
I think their response was completely rational, especially given a potentially huge impact on compute costs and/or publication latency.
I disagree with it being a premature optimisation. Treating everything that you haven’t already identified personally as a problem as a premature optimisation is cargo cutting in its own way. The attitude of not caring is why npm and various tools are so slow.
That said, I think NPM’s response was totally correct - explain the problem and the tradeoffs. And OP decided the tradeoffs weren’t worth it, which is totally fair.
While NPM is open source, it's in the awkward spot of also having... probably hundreds of thousands if not millions of professional applications depend on it; it should be run like a business, because millions depend on it.
...which makes it all the weirder that security isn't any better, as in, publishing a package can be done without a review step on the npm side, for example. I find it strange that they haven't doubled down on enterprise offerings, e.g. creating hosted versions (corporate proxies), reviewed / validated / LTS / certified versions of packages, etc.
Imagine being in the middle of nowhere, in winter, on Saturday night, on some farm, knee deep in a cow piss, servicing some 3rd party feed dispenser, only to discover that you have possible solution but it's in some obscure format instead of .tar.gz. Nearest internet 60 miles away. This is what I always imagine happening when some new obscure format come into play, imagine the poor fella, alone, cold, screaming. So incredibly close to his goal, but ultimately stopped by some artificial unnecessary made-up bullshit.
Years back I came to the conclusion that conda using bzip2 for compression was a big mistake.
Back then if you wanted to use a particular neural network it was meant for a certain version of Tensorflow which expected you to have a certain version of the CUDA libs.
If you had to work with multiple models the "normal" way to do things was use the developer unfriendly [1][2] installers from NVIDIA to install a single version of the libs at a time.
Turned out you could have many versions of CUDA installed as long as you kept them in different directories and set the library path accordingly, it made sense to pack them up for conda and install them together with everything else.
But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time.
If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression.
[1] click... click... click...
[2] like they're really going to do something useful with my email address
>But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time.
For Paper, I'm planning to cache both the wheel archives (so that they're available without recompressing on demand) and unpacked versions (installing into new environments will generally use hard links to the unpacked cache, where possible).
> If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression.
FWIW, in my testing LZMA is a big win (and I'm sure zstd would be as well, but LZMA has standard library support already). But there are serious roadblocks to adopting a change like that in the Python ecosystem. This sort of idea puts them several layers deep in meta-discussion - see for example https://discuss.python.org/t/pep-777-how-to-re-invent-the-wh... . In general, progress on Python packaging gets stuck in a double-bind: try to change too little and you won't get any buy-in that it's worthwhile, but try to change too much and everyone will freak out about backwards compatibility.
I designed a system which was a lot like uv but written in Python and when I looked at the politics I decided not to go forward with it. (My system also had the problem that it had to be isolated from other Pythons so it would not get its environment trashed, with the ability for software developers to trash their environment I wasn't sure it was a problem that could be 100% solved. uv solved it by not being written in Python. Genius!)
Yes, well - if I still had reason to care about the politics I'd be in much the same position, I'm sure. As is, I'm going to just make the thing, write about it, and see who likes it.
Thank you so much for posting this. The original logic was clear and it had me excited! I believe this is useful because compression is very common and although it might not fit perfectly in this scenario, it could very well be a breakthrough in another. If I come across a framework that could also benefit from this compression algorithm, I'll be sure to give you credit.
252 comments
[ 0.23 ms ] story [ 221 ms ] threadI don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes).
It would have been interesting to see what eventually played out if the author hadn't closed the RFC themselves. It could have been the sort of thing that eventually happens after 2 years, but then quietly makes everybody's lives better.
They soft-rejected by requiring more validation than was reasonable. I see this all the time. "But did you consider <extremely unlikely issue>? Please go and run more tests."
It's pretty clear that the people making the decision didn't actually care about the bandwidth savings, otherwise they would have put the work in themselves to do this, e.g. by requiring Zopfli for popular packages. I doubt Microsoft cares if it takes an extra 2 minutes to publish Typescript.
Kind of a wild decision considering NPM uses 4.5 PB of traffic per week. 5% of that is 225 TB/week, which according to my brief checks costs around $10k/week!
I guess this is a "not my money" problem fundamentally.
That's the market price though, for Microsoft its a tiny fraction of that.
I do agree that 10k/week is non-negligible. Perhaps that means the people responsible for the 10k weren't in the room?
And I can buy 225 TB of bandwidth for less than $2k, I assume Microsoft can get better than some HN idiot buying Linode.
Even so, $2k a week is at least one competent FTE.
Checksum matters aside, wouldn't that turn the 5% bandwidth savings into an almost double bandwidth increase though? IMHO, considering the complexity to even make it a build time option, the author made the right call.
I found it reasonable.
The 5% improvement was balanced against the cons of increased cli complexity, lack of native JS zopfli implementation, and slower compression .. and 5% just wasn't worth it at the moment - and I agree.
>or at least I think they warrant further discussion
I think that was the final statement.
This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm.
You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to make any changes, until you experience your very own "oh man, I really thought this change was perfectly safe and now my entire customer base is trashed" moment, and then suddenly things like "hey, we need to consider how this affects old signatures and the speed of decompression and just generally whether this is worth the non-zero risks for what are in the end not really that substantial benefits".
I do not say this as the wise Zen guru sitting cross-legged and meditating from a position of being above it all; I say it looking at my own battle scars from the Perfectly Safe things I've pushed out to my customer base, only to discover some tiny little nit caused me trouble. Fortunately I haven't caused any true catastrophes, but that's as much luck as skill.
Attaining the proper balance between moving forward even though it incurs risk and just not changing things that are working is the hardest part of being a software maintainer, because both extremes are definitely bad. Everyone tends to start out in the former situation, but then when they are inevitably bitten it is important not to overcorrect into terrified fear of ever changing anything.
But it is possible to do it more gradually, I.e. by sneaking it in with a new API that's used by new npm version or similar.
But it was his choice to make, and it's fine that he didn't feel enough value in pursuing such a tiny file size change
5% is 5% at any scale.
I could never get Docker to work on my ADSL when it was 2 Mbps (FTTN got it up to 20) though it was fine in the Montreal office which had gigabit.
If that moves the needle on your ability to purchase the car, you probably shouldn't be buying it.
5% is 5%.
I'm sure you can use your imagination to substitute "lunch" and "car" with other examples where the absolute change makes a difference despite the percent change being the same.
Even taking it literally... The 5% might not tip the scale of whether or not I can purchase the car, but I'll spend a few hours of my time comparing prices at different dealers to save $2500. Most people would consider it dumb if you didn't shop around when making a large purchase.
On the other hand, I'm not going to spend a few hours of my time at lunch so that I can save an extra $1 on a meal.
- Doing 1 hour of effort to save 5% on your $20 lunch is foolhardy for most people. $1/hr is well below US minimum wage. - Doing 1 hour of effort to save 5% on your $50k car is wise. $2500/hr is well above what most people are making at work.
It's not about whether the $2500 affects my ability to buy the car. It's about whether the time it takes me to save that 5% ends up being worthwhile to me given the actual amount saved.
The question is really "given the person-hours it takes to apply the savings, and the real value of the savings, is the savings worth the person-hours spent?"
Also, this is exactly what you company is doing, paying you to "read the book" so they don't have to.
Here the scale of time is larger and does make the 5$ significant, while it isn't at the scale of a few days.
Given his numbers, let's say he saves 100Tb of bandwidth over a year. At AWS egress pricing... that's $5,000 total saved.
And arguably - NPM is getting at least some of that savings by adding CPU costs to publishers at package time.
Feels like... not enough to warrant a risky ecosystem change to me.
Even that's addressable though if there's motivation, since something like transcoding server side during publication just for popular packages would probably get 80% of the benefit with no client-side increase in publication time.
NPM uses at least 5 petabytes per week. 5% of that is 250 terabytes.
So $15,000 a week, or $780,000 a year in savings could’ve been gained.
For reference, total bandwidth used by all 5000 packages is 4_752_397 GiB.
Packages >= 1GiB bandwidth/week - That turns out to be 437 packages (there's a header row, so it's rows 2-438) which uses 4_205_510 GiB.
So 88% of the top 5000 bandwidth is consumed by downloading the top 8.7% (437) packages.
5% is about 210 TiB.
Limiting to the top 100 packages by bandwidth results in 3_217_584 GiB, which is 68% of total bandwidth used by 2% of the total packages.
5% is about 161 TiB.
Less than 1% of top 5000 packages took 53% of the bandwidth.
5% would be about 127 TiB (rounded up).
The more bandwidth that Cloudflare needs, the more leverage they have at the peering table. As GitHub's largest repo (the @types / DefinitelyTyped repo owned by Microsoft) gets larger, the more experience the owner of GitHub (also Microsoft) gets in hosting the world's largest git repos.
I would say this qualifies as one of those cases, as npmjs is hosted on Azure. The more resources that NPM needs, the more Microsoft can build towards parity with AWS's footprint.
Everyone benefits, the only cost is a ultra microscopic time on the front end, and a tiny cost on the client end, and for a very significant number of users, time and money saved. The examples of compression here...
It needs a novel JS port of a C compresison library, which will be wired into a heavily-used and public-facing toolchain, and is something that will ruin a significant number of peoples' days if it breaks.
For me, that kind of ask needs a compelling use case from the start.
Another personal win is that you got a very thorough understanding of the people involved and how the outreach parts of the RFC process works. I've also had a few fail, but I've also had a few pass! Always easier to do the next time
Comparing to gzip isn't really worth it. Combine pigz (threaded) with zlib-ng (simd) and you get decent performance. pigz is used in `docker push`.
For example, gzipping llvm.tar (624MB) takes less than a second for me:
At the same time, zopfli compiled with -O3 -march=native takes 35 minutes. No wonder it's not popular.It is almost 2700x slower than the state of the art for just 6.8% bytes saved.
That is impressively slow.
In my opinion even the 28x decrease in performance mentioned would be a no-go. Sure the package saves a few bytes but I don't need my entire pc to grind to a halt every time I publish a package.
Besides, storage is cheap but CPU power draw is not. Imagine the additional CO2 that would have to be produced if this RFC was merged.
> 2 gigabytes of bandwidth per year across all installations
This must be a really rough estimate and I am curious how it was calculated. In any case 2 gigabytes over a year is absolutely nothing. Just my home network can produce a terabyte a day.
Because the authors mentioned package, Helmet[1], is 103KB uncompressed and has had 132 versions in 13 years. Meaning downloading every Helmet version uncompressed would result in 132*103KB = 13.7MB.
I feel like I must be missing something really obvious.
Edit: Oh it's 2GB/year across all installations.
[1]: https://www.npmjs.com/package/helmet?activeTab=versions
5% sounds like a good deal
Its not always obvious who has the most important use cases. In the case of NPM they are prioritizing the user experience of module authors. I totally see how this change would be great for module consumers, yet create potentially massive inconvenience for module authors.
Interesting write-up
Also, since this is a case of having something compressed once and decompressed potentially thousands of times, it seems like the perfect tool for the job.
This is changing every build in every CI system to make it slower.
The fix would then have to be some variation of:
a) Stop testing (so often)
b) Stop bundling before testing
c) Publish to a different package manager
- all of which would affect the overall quality and quantity npm modules.
But yes- you could bundle less, but that would be a disadvantage, particularly if a bundle suddenly fails and you don’t know which change caused it. But maybe that’s not a big deal for your use case.
Working approach, but in the end I abandoned the project - I doubt people care about such js size savings.
I'm sure it works great when you've got high-speed internet, but might break things unacceptably for users on mobile or satellite connections.
This part is obviously trivially solvable. I think the same basic idea is going to at some point make it but it’ll have to be through explicit annotations first and then there will be tooling to automatically do this for your code based upon historical visits where you get to tune the % of visitors that get additional fetches. Also, you could probably fetch the split off script in the background anyway as a prefetch + download everything rather than just 1 function at a time (or even downloading related groups of functions together)
The idea has lots of merit and you just have to execute it right.
Note that most of the unused code is located in libraries, not in the app code directly.
This would cause the bundler to inject a split point & know how to hide that + know what needs bundling and what doesn’t. GWT pioneered almost 20 years ago although not a fan of the syntax they invented to keep everything running within stock Java syntax: https://www.gwtproject.org/doc/latest/DevGuideCodeSplitting....
The tooling producing such annotaion based on historical visits whould constantly change the annotated set of functions, I suspect, whith app versions evolving.
Note, the inactive code parts are very often in 3rd party libraries.
Some are also in your own libraries shared between your different application. So the same shared library whould need one set of split-annotated functions for one app, and another set for another app. So these conflicting sets of annotations can not live in the library source code simultaneously.
As soon as the user executes a rarely used function, the information is about this fact is sent to the server and it includes this function into the active set to be send to future users.
In the video I manually initiate re-generation of the "active" and the "rest" scripts, but the most primitive MVP was supposed to schedule re-generation of the scripts when receiving info that some previously unseen functions are executed in browser.
Obviously, if the idea is developed further, the first user's experience may also be improved - a spinner shown. Pre-loading the inactive set of functions in background may also be considered (pros: it allows to avoid latency for users ho invoke rare functionality, cons: we lose the savings of the traffic and browser memory and cpu for compiling the likely unneded code).
(BTW, further development of the idea includes to splitting the code more granularity than just "active" / "inactive". E.g. active in the first 5 seconds after opening the page loaded immediately, likely to be active soon, active but rarely called - the later two these parts definitely need to be loaded in background)
50% is just a big O of the original size :)
If we consider overall Internet traffic, which is dominated by video and images, the size of javascript transmitted is negligible.
Savings of memory and cpu for end user's browser? Maybe, but our software at all layers is so bloated, without need, just carelessly, that I'm not sure working of such javascript savings is useful - any resources saved will be eaten by something else immediately.
For an application developer or operator, the 50% savings of javascript size are probably not worth dealing with some tool that dynamically prunes the app code, raising questions about privacy and security. I though through the security and privacy questions, but who would want to even spend attention on these considerations?
As I mention in the README at github, a Microsoft researcher investigated the same approach earlier, but Microsoft haven't taken it anywhere.
There was a commercial company offering this approach of javascript minification as a product, complete product. As well as other optimizations, like image size, etc. Their proxy embedded special "agent" code into the app which inspected the device and reported to server what image sizes are optimal for user, what js functions are invoked. And the server prepared optimized versions of app for various devices. Javascript was "streamed" in batches of only functions needed by the app.
Now the company is dissolved - that's why the website is unavailable. Wikipedia says they were bought out by Akamai - https://en.wikipedia.org/wiki/Instart. But I don't see any traces of this approach in the today's Akamai offerings.
I contacted several companies, in CDN business and others, trying to interest them in the idea and get very modest funding for a couple more months of my time to work on this (I was working on this in the end of a long break from payed work and was running out of savings). Didn't find anyone ready to take part.
This all may be signs that possibility of such an optimization is not valuable enough for users.
Then the info about the called functions is sent back to the server.
(Only the functions never seen to be called are instrumented, the known active functions are not instrumented).
I think people forget that, when you’re copying the same images to dozens and dozens of boxes, any improvement starts to add up to real numbers.
Reduced is the size of javascript loaded into the end user's browser.
https://jamiemagee.co.uk/blog/honey-i-shrunk-the-npm-package...
and the related HN discussion from that time:
https://news.ycombinator.com/item?id=37754489
> When it was finally my turn, I stammered.
> Watching it back, I cringe a bit. I was wordy, unclear, and unconvincing.
> You can watch my mumbling in the recording
I watched this, and the author was articulate and presented well. The author is too harsh!
Good job for trying to push the boundaries.
Maybe it would make sense to only apply this improvement in images that are a) either very big or b) get downloaded at least million times each year or so. That would cover most of the savings while leaving most packages and developers out of it.
with prod env: 700MB
without prod env: 900MB
sadly the bloat cannot be avoided that well :/
I also like Yarn pnp’s model of leaving node_modules as zip files. CPUs are way faster than storage, they can decompress on the fly. Less disk space at rest, less disk slack, less filesystem bookkeeping.
Every single filesystem is way faster at dealing with one file than dozens/hundreds. Now multiply that by the the hundreds if does, it add up.
They’ve done a lot of great work already.
Now if rxjs weren’t a dumpster fire…
(And it's also a little on the folks who install dependencies: if the cruft in a specific library bothers you, hit up the repo and file an issue (or even MR/PR) to get that .npmignore file filled out. I've helped folks reduce their packages by 50+MB in some cases, it's worth your own time as much as it is theirs)
On a typical project with a build step, only a `dist` folder would published.
> On a typical project with a build step, only a `dist` folder would published.
Sort of, but always include your docs (readme, changelog, license, and whatever true docs dir you have, if you have one). No one should need a connection for those.
[0] https://github.com/sindresorhus/clipboardy/tree/main/fallbac...
I just checked out clipboardy and all they do is dispatch binaries from the path and hope it's the right one (or if it's even there at all). I think I had a similar experience with Python and Lua scripts. There's an unfunny amount of poorly-written one-off clipboard scripts out there just waiting to be exploited.
I'm only glad that the go-to clipboard library in Rust (arboard) seems solid.
You can read the code and you can usually read the actual README/docs/tests of the package instead of having to find it online. And you can usually edit library code for debugging purposes.
If node_modules is taking up a lot of space across a bunch of old projects, just write the `find` script that recursively deletes them all; You can always run `npm install` in the future when you need to work on that project again.
TS releases a non-preview version every few months, so using 2.5 minutes for compression would work.
Such effort would be better spent preparing npm/node for a future where packages with a lower-bound npm version constraint can be compressed with zstd or similar.
Every feature you add to a program is complexity, you need to really decide if you want it.
Starting with smaller data means everything ends up smaller. It's the same decompression algorithm in all cases, so it's not some special / unoptimized branch of code. It's yielding the same data in the end, so writes equal out plus or minus disk queue fullness and power cycles. It's _maybe_ better for RAM and CPU because more data fits in cache, so less memory is used and the compute is idle less often.
It's relatively easy to test decompression efficiency if you think CPU time is a good proxy for energy usage: go find something like React and test the decomp time of gzip -9 vs zopfli. Or even better, find something similar but much bigger so you can see the delta and it's not lost in rounding errors.
This means an optimization like this is virtually guaranteed to be a net positive on the receiving end, since you always save a bit of time/energy when downloading a smaller compressed file.
Zopfli is easiest to think of as something that just tries harder than gzip to find matches and better encodings. Much harder.
decompression speed is linear either way.
It's easiest to think of decompression as a linear time vm executor[1], where the bytecoded instructions are basically
go back <distance> bytes, output the next <length> bytes you see, then output character <c>
(outputting literal data is the instruction <0,0,{character to output}>)
Assuming you did not output a file larger than the original uncompressed file (why would you bother?), you will, worst case, process N bytes during decompression, where N is the size of the original input file.
The practical decompression speed is driven by cache behavior, but it thrashes the cache no matter what.
In practice, reduction of size vs gzip occurs by either finding larger runs, or encodings that are smaller than the existing ones.
After all, if you want the compressed file to shrink, you need output less instructions somehow, or make more of the instructions identical (so they can be represented in less bits by later huffman coding).
In practice, this has almost exclusively positive effects on decompression speed - either the vm has less things to process (which is faster), or more of the things it does look the same (which has better cache behavior).
[1] this is one way archive formats will sometimes choose to deal with multiple compression method support - encode them all to the same kind of bytecode (usually some form of copy + literal instruction set), and then decoding is the same for all of them. ~all compression algorithms output some bytecode like the above on their own already, so it's not a lot of work. This doesn't help you support other archive formats, but if you want to have a bunch of per-file compression options that you pick from based on what works best, this enables you to still only have to have one decoder.
https://quixdb.github.io/squash-benchmark/
There even more obscure options:
https://www.mattmahoney.net/dc/text.html
The package lock checksum is probably a more solvable issue with some coordination.
The benefit of doing this though is less immediate - it will take a few years to show payoff and these kinds of payoffs are not typically made by the kind of committee decisions process described (for better or worse).
If accepted, it might have been a good stepping stone too. A chance to get to know everyone and their concerns and how they think.
So if you wanted to see how this works (proposal + in prod) and then come back later proposing something bigger by switching off zip that would make sense to me as a possible follow up.
But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do.
For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And I would have quit at the first roadblock.
What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line.
Do you want to get a change in, or not?
Is this a project working with the community or not?
> Just release your work
What motivation exists to optimize package formats if nobody uses that package format? There are no benefits unless it’s in mainline.
> Nobody in open source wants to be told what to do
He’s not telling anybody to do work. He is sharing an optimization with clear tradeoffs - not a new architecture.
> You cannot force your improvement into the whole group
Nope, but communication is key. “put up a PR and we will let you know whether it’s something we want to pursue”.
Instead they put him through several levels of gatekeepers where each one gave him incorrect feedback.
“Why do we want to optimize bandwidth” is a question they should have the answer to.
If this PR showed up on my project I would say “I’m worried about X,Y,Z” we will set up a test for X and Y and get back to you. Could you look into Z?
Says who?
Says an engineer? Says a product person?
NPM is a company with 14 employees; with a system integrated into countless extremely niche and weird integrations they cannot control. Many of those integrations might make a professional engineer's hair catch fire - "it should never be done this way!" - but the real world is that the wrong way is the majority of the time. There's no guarantee that many of the downloads come from the official client, just as one example.
The last thing they need, or I want, or any of their customers want, or their 14 employees need, is something that might break backwards compatibility in an extremely niche case, anger a major customer, cause countless support tickets, all for a tiny optimization nobody cares about.
This is something I've learned here about HN that, for own mental health, I now dismiss: Engineers are obsessed with 2% optimizations here, 5% optimizations there; unchecked, it will literally turn into an OCD outlet, all for things nobody in the non-tech world even notices, let alone asks about. Just let it go.
Hell. Even 15 employees larping as a corporation is going to be inefficient.
what you and NPM are telling us, is that they are happy to take free labor, but this is not an open source project.
> Engineers are obsessed with 2% optimizations here
Actually in large products these are incredible finds. But ok. They should have the leadership to know which bandwidth tradeoffs they are committed to and tell him immediately it’s not what they want, rather than sending him to various gatekeepers.
> Actually in large products these are incredible finds.
In large products, incredible finds may be true; but breaking compatibility with just 0.1% of your customers is also an incredible disaster.
Yes. But in this story nothing like that happened.
Just because it's safe in a lab provides no guarantee it's safe in production.
The conclusion of this line of reasoning is to never make any change.
If contributions are not welcome, don’t pretend they are and waste my time.
> can no longer decompress fast enough
Already discussed this in another thread. It’s not an issue.
"If it ain't broke, don't fix it."
I absolutely disagree with you. If the world took more of those 5% optimisations here and there everything would be faster. I think more people should look at those 5% optimisations. In many cases they unlock knowledge that results in a 20% speed up later down the line. An example from my past - I was tasked with reducing the running speed of a one shot tool we were using at $JOB. It was taking about 15 minutes to run. I shaved off seconds here and there with some fine grained optimisations, and tens of seconds with some modernisation of some core libraries. Nothing earth shattering but improvements none the less. One day, I noticed a pattern was repeating and I was fixing an issue for the third time in a different place (searching a gigantic array of stuff for a specific entry). I took a step back and realised that if I replaced the mega list with a hash table it might fix every instance of this issue in our app. It was a massive change, touching pretty much every file. And all of a sudden our 15 minute runtime was under 30 seconds.
People used this tool every day, it was developed by a team of engineers wildly smarter than me. But it had grown and nobody really understood the impact of the growth. When it started that array was 30, 50 entries. On our project it was 300,000 and growing every day.
Not paying attention to these things causes decay and rot. Not every change should be taken, but more people should care.
> Says an engineer?
I prevent cross-site scripting, I monitor for DDoS attacks, emergency database rollbacks, and faulty transaction handlings. The Internet heard of it? Transfers half a petabyte of data every minute. Do you have any idea how that happens? All those YouPorn ones and zeroes streaming directly to your shitty, little smart phone day after day? Every dipshit who shits his pants if he can't get the new dubstep Skrillex remix in under 12 seconds? It's not magic, it's talent and sweat. People like me, ensuring your packets get delivered, un-sniffed. So what do I do? I make sure that one bad config on one key component doesn't bankrupt the entire fucking company. That's what the fuck I do.
If/when NPM has a problem - storage costs are too high, or transfer costs are too high, or user feedback indicates that users are unhappy with transfer sizes - then they will be ready to listen to this kind of proposal.
I think their response was completely rational, especially given a potentially huge impact on compute costs and/or publication latency.
That said, I think NPM’s response was totally correct - explain the problem and the tradeoffs. And OP decided the tradeoffs weren’t worth it, which is totally fair.
...which makes it all the weirder that security isn't any better, as in, publishing a package can be done without a review step on the npm side, for example. I find it strange that they haven't doubled down on enterprise offerings, e.g. creating hosted versions (corporate proxies), reviewed / validated / LTS / certified versions of packages, etc.
Back then if you wanted to use a particular neural network it was meant for a certain version of Tensorflow which expected you to have a certain version of the CUDA libs.
If you had to work with multiple models the "normal" way to do things was use the developer unfriendly [1][2] installers from NVIDIA to install a single version of the libs at a time.
Turned out you could have many versions of CUDA installed as long as you kept them in different directories and set the library path accordingly, it made sense to pack them up for conda and install them together with everything else.
But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time.
If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression.
[1] click... click... click...
[2] like they're really going to do something useful with my email address
For Paper, I'm planning to cache both the wheel archives (so that they're available without recompressing on demand) and unpacked versions (installing into new environments will generally use hard links to the unpacked cache, where possible).
> If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression.
FWIW, in my testing LZMA is a big win (and I'm sure zstd would be as well, but LZMA has standard library support already). But there are serious roadblocks to adopting a change like that in the Python ecosystem. This sort of idea puts them several layers deep in meta-discussion - see for example https://discuss.python.org/t/pep-777-how-to-re-invent-the-wh... . In general, progress on Python packaging gets stuck in a double-bind: try to change too little and you won't get any buy-in that it's worthwhile, but try to change too much and everyone will freak out about backwards compatibility.