Nice write up. It still boggles the mind when coming from a Java background how things change but also get worse.
Having zip's (.jar/.war) which had the source and compiled artefacts available and not cluttering the OS with unzip items.
I've run into huge issues dealing with deployments to AWS Lambda nodejs, where the previous developer could not work out how to make a 'slip' deployment and just zipped the whole folder node_modules and all and shipped that, this included all testing tools including an entire browser exe etc. 150mb file expanded to 350mb with 300k in files. And 95%+ of it was boiler-pate to test the api endpoint logic...
I've still not come across the simplest way to have dependences on modules with testing and still be able to deploy a slim zip. There is more and more push to do container layers for the node_modules, but still many of these has 100'000 objects which are not needed causing lambda stutter(startup sleepyness).
yeah that's the biggest issue with node/npm. 90% of your code lives on the registry, and you probably not even sure what 99% of it actually does.
would be nice if you could somehow reverse the compilation of your build back into a source folder, instead of having node_modules - but i think that's probably not possible.
i think bundling generates your build from the source, but i mean generating a source from a build. so what i mean is more like reverse bundling.
so that you don't have to pull node_modules every time and install. i;m not totally sure but i dont think you can turn a minified bundled js files back into their original node_module source files...
That very much depends on the build -- if you've a source map, then it may very well be possible. And while I know not everyone publishes them, I strongly recommend doing so.
But honestly, ship your sources with your generated files. Please.
> just zipped the whole folder node_modules and all and shipped that
I mean, in a sense, something similar is what you should do if you're aiming for maximum reproducibility, of course, excluding the things that you don't need.
For a comparison, if you wanted to build a Docker container, it's going to include the node_modules folder in it, instead of installing dependencies on startup. So, in a word, somewhat similar to a fat .jar file (e.g. that has all dependencies, vs assuming that the app server will provide some).
My problem is that sometimes it's not awfully clear how to exclude all of the stuff that you don't need.
For example, my homepage is made with Ruby on Rails, for which I have a basic container image. But to actually bundle all of the resources, I do need Python, Node (with Yarn in my case) and also a bunch of gems, so I also make a separate dev container image.
I can install all of the dependencies I need for packaging and run them in a multi-stage image, carrying over the packaged files to something based on the basic image for just running it (e.g. without Python and Node), but I still need the gems to be present and there's no obvious way to carry over only the ones I need for running a Rails app.
Even without containers, this very same pain is present in many tech stacks, even if you wanted to run software directly on the servers to which you deploy it - many stacks out there aren't really big on fully self-contained executable packages (or even ones that depend on a particular version of a runtime, like JVM or .NET, but have everything else included).
Ergo, working with Ruby, Python, Node and many others can be a bit more troublesome than it should be. Personally, containers alleviate some of those problems to a degree, at the expense of exposing that complexity to you during build time.
Hmm, why don't you specify your development-time dependencies as devDependencies in your package.json? Then you can simply install only the production ones in your final step Docker image - and only use the devDependencies in your build step. I'm doing it, works like a charm. It'd be really weird and wasteful to ship my TypeScript and Eslint and whatever to cloud/Lambda/...
> Hmm, why don't you specify your development-time dependencies as devDependencies in your package.json? Then you can simply install only the production ones in your final step Docker image - and only use the devDependencies in your build step.
This is pretty close to the solution and would work if I ran just Node.js server side. Then I could easily have separate sets of dependencies for development and deployment/running, which is what most folks should do. Yet in certain tech stacks, things aren't as easy and there is reliance on system folders instead of local ones.
For example, my application that's running on the server should have absolutely nothing to do with Node, since it should run Ruby alone and use the bundled files, however it should have all of the Ruby Gems (basically packages) available. Of course, I also need Node and possibly something else during build time, for generating my assets.
For example, my intermediate container images might look like:
FROM ruby_with_node_and_python AS builder
# Copy only what's needed for installing dependencies and other garbage (e.g. changed code files won't invalidate Docker cache)
COPY ./src/.browserslistrc ./src/.ruby-version ./src/babel.config.js ./src/config.ru ./src/Gemfile ./src/Gemfile.lock ./src/yarn.lock ./src/package.json ./src/postcss.config.js ./src/Rakefile /app
# install front end dependencies
RUN yarn install
# add code and run asset creation
COPY ./src /app
RUN bundle exec rake assets:clobber RAILS_ENV=production
RUN bundle exec rake assets:precompile RAILS_ENV=production
# install back end dependencies
RUN bundle install
# run tests (local SQLite, for example)
RUN rails db:migrate RAILS_ENV=test
RUN rails test RAILS_ENV=test
# clean up files after build (so copying in later images doesn't drag garbage along; this could be a separate intermediate image too)
RUN rm -rf src/node_modules
# clean up files after test (so copying in later images doesn't drag garbage along; this could be a separate intermediate image too)
RUN rm -rf src/log/*.log src/tmp/*
And then my container images for deployment would look like this:
FROM ruby_with_nothing_else AS runner
# copy over application files
COPY --from=builder /app /app
Except that this doesn't work! Why? Because Ruby Gems aren't installed in the application directory, but instead sit in a variety of directories on the actual file system. So instead I'd need something like:
FROM ruby_with_package_manager AS runner
# we actually could not carry over the precompiled gems etc. because I'm stupid, so instead we now run the install again :(
# well it's not like you need front end files here, but this doesn't really mean much
COPY ./src/.browserslistrc ./src/.ruby-version ./src/babel.config.js ./src/config.ru ./src/Gemfile ./src/Gemfile.lock ./src/yarn.lock ./src/package.json ./src/postcss.config.js ./src/Rakefile /app
# install back end dependencies
RUN bundle install
# copy over application files
COPY --from=builder /app /app
So while npm/yarn allows you to avoid some of the issues with `npm install --production` to exclude the devDependencies, other tech stacks aren't quite there yet. In addition, it feels kind of crazy to ship package managers like npm/yarn/pip/composer/bundle/maven or anything else in containers that should be immutable, so in practice you'd end up with more stages, even if you knew how to carry over the packages from one image to another in the technology stacks where it's not entirely clear how to do that.
I actually tried something similar to this, but it sim...
For what? If you want to alter it, yes, otherwise package.json already has `bundledDependencies` to achieve the same without cluttering your repository.
So... set ourselves up for another Log4j situation where a vulnerability is found in a sub-sub-sub dependency of some dependency you didn't even know you were using, such that only good way of fixing it is to manually scrub the code from `.jar`s?
Wait a few days for what, exactly? For your sub-sub-sub dependency to patch it, then for your sub-sub dependency to notice this and update the version, then for your sub-dependency to notice it to update it, then for your dependency to finally notice and update it?
The solution that npm presents solves this problem without this insanity -- you can just create a `npm-shrinkwrap.json` file to manually update sub-dependencies. If one of your dependencies used bundling, this solution becomes impossible.
In NPM 8 you can override sub deps quite effectively, even using references.
Another option is package-patch. Very light, elegant and yet robust solution for fixing anything in any package you depend on yourself without waiting for maintainer to fix it.
You missed the point of Node’s dependency resolution.
If you want to lock the dependencies, just use a lockfile in your app, you don't need to ship the same code 10 times because your dependencies already decided to bundle their sub-dependencies.
A lot of libraries are published along with various types of bundles and non-bundles, including usually the source, so the user can pick which one is best for them. For example, you can import just one Lodash module at a time so you don't have to bloat your app with the whole library. This seems like the best of both worlds
We're converging on better norms for JS dialects/module systems, but we're not all the way there yet, so I don't see much purpose dropping support for multiple formats when some people might still need some of them
If you do want to go clean-slate and drop all the ceremony at once, I recommend Deno :)
It’s actually pretty annoying that some people are now going pure esm and actively removing cjs builds. There are certain libraries that don’t work with esm yet, so if you need one of those suddenly all the esm-only packages can’t be used.
I assume that you're talking about tools rather than libraries. Any simple library can import ESM asynchronously and any ESM packages can import CJS packages even with static imports.
Tools and Node’s strict ERR_REQUIRE_ESM have been the problem with the migration.
In my experience, you may encounter issues when you depends on pure esm package and cjs-only packages. IMHO, it is still useful to publish a cjs fallback.
You may encounter issues with mixed packages too, actually a lot more issues given that people expect `module.exports = ` and `export default` to be equal but are not.
Just save yourself some headache and publish either ESM or CJS, not both.
- dependency versions being fixed is a feature. I've had package installs fail because a dependency published a semver-compatible update that broke something. In a web context this would break the app until the dependency pulled the update or the dependent pushed an update disallowing that version.
bundling should be done as a step to build an application.
it's really not needed when you prepare a library.
> - dependency versions being fixed is a feature. I've had package installs fail because a dependency published a semver-compatible update that broke something.
If this is the case please open a bug to the dependency team. And then avoid ^ ~ and * in the dependencies version you are using.
This might make sense for performing a "final" build or for very small libraries, but IMO not really for anything beyond that or for any sort of application level codebase. Some of the links in OPs post mention projects and not necessarily libraries, where the amount of code included can increase pretty quickly and tsc is just so so so much slower then using a tool like vite/esbuild/tsx/vitest or similar swc derivatives. I think relying on your IDE for type checking and linting while periodically performing a full typecheck via tsc in CI could be the sweet spot. Not to mention other features like hot reloading, auto re-running tests, etc that these bundlers and tools that utilize them expose for you.
At least for me I think maintaining a tight feedback loop between "make a change" => "verify it works" is important to not be distracted by another task. esbuild taking 100 or so ms while tsc taking 30+ seconds makes a world of difference for that.
Huh, is bundling commonly found in library npm packages? Which packages do this? Asking so I can avoid this.
I already have a bundle intended for distribution to web browsers, I don't need these sorts of poor practices introducing unnecessary bundling code. Nested bundling sounds like insanity to me. So I very much agree with the author that this practice never made sense in the first place, and I'm baffled that it exists. It's very much an npm anti-pattern.
I think a lot of them use bundlers but they use them for transpiling, going from ts to js and creating esm and common js files not for bundling in dependencies. I’ve also seen bundlers be used to include css in js as to not require users to manually import it.
It’s not so bad, working with just tsc can be a little more cumbersome as you have to running it multiple times to get esm and common js and it’s slower than some alternatives. I see it more akin to using a makefile than an anti pattern.
As someone who regularly publishes npm packages [1], and have dealt with packages that do this, please don't.
At least not yet until ESM is fully supported by all parties in the ecosystem (node, V8, major libraries and frameworks). As of Sept 2022, we still have support issues with node (V8), TypeScript and jest. I have links to the issues in one of my GitHub repos[2].
Once you move past trivial hello world apps, it is literally impossible to run JavaScript in node in pure ESM because of various bugs and incompatibilities. The user of the library will hit "Cannot use import statement outside a module" error sooner or later.
Node 16+ and Typescript 4.7+ and recent versions of jest (I don't know offhand a version number) seem to have sorted most of the issues with using ESM modules at this point. I agree ESM was more trouble in 2021, but this deep in 2022 things are a lot rosier.
I accept that a library should not be bundled into a single file. However I use esbuild instead of tsc for trans piling my code. It is blazing fast and make a quick write-transpile-test feedback.
39 comments
[ 3.0 ms ] story [ 101 ms ] threadHaving zip's (.jar/.war) which had the source and compiled artefacts available and not cluttering the OS with unzip items.
I've run into huge issues dealing with deployments to AWS Lambda nodejs, where the previous developer could not work out how to make a 'slip' deployment and just zipped the whole folder node_modules and all and shipped that, this included all testing tools including an entire browser exe etc. 150mb file expanded to 350mb with 300k in files. And 95%+ of it was boiler-pate to test the api endpoint logic...
I've still not come across the simplest way to have dependences on modules with testing and still be able to deploy a slim zip. There is more and more push to do container layers for the node_modules, but still many of these has 100'000 objects which are not needed causing lambda stutter(startup sleepyness).
would be nice if you could somehow reverse the compilation of your build back into a source folder, instead of having node_modules - but i think that's probably not possible.
so that you don't have to pull node_modules every time and install. i;m not totally sure but i dont think you can turn a minified bundled js files back into their original node_module source files...
But honestly, ship your sources with your generated files. Please.
I mean, in a sense, something similar is what you should do if you're aiming for maximum reproducibility, of course, excluding the things that you don't need.
For a comparison, if you wanted to build a Docker container, it's going to include the node_modules folder in it, instead of installing dependencies on startup. So, in a word, somewhat similar to a fat .jar file (e.g. that has all dependencies, vs assuming that the app server will provide some).
My problem is that sometimes it's not awfully clear how to exclude all of the stuff that you don't need.
For example, my homepage is made with Ruby on Rails, for which I have a basic container image. But to actually bundle all of the resources, I do need Python, Node (with Yarn in my case) and also a bunch of gems, so I also make a separate dev container image.
I can install all of the dependencies I need for packaging and run them in a multi-stage image, carrying over the packaged files to something based on the basic image for just running it (e.g. without Python and Node), but I still need the gems to be present and there's no obvious way to carry over only the ones I need for running a Rails app.
Even without containers, this very same pain is present in many tech stacks, even if you wanted to run software directly on the servers to which you deploy it - many stacks out there aren't really big on fully self-contained executable packages (or even ones that depend on a particular version of a runtime, like JVM or .NET, but have everything else included).
Ergo, working with Ruby, Python, Node and many others can be a bit more troublesome than it should be. Personally, containers alleviate some of those problems to a degree, at the expense of exposing that complexity to you during build time.
This is pretty close to the solution and would work if I ran just Node.js server side. Then I could easily have separate sets of dependencies for development and deployment/running, which is what most folks should do. Yet in certain tech stacks, things aren't as easy and there is reliance on system folders instead of local ones.
For example, my application that's running on the server should have absolutely nothing to do with Node, since it should run Ruby alone and use the bundled files, however it should have all of the Ruby Gems (basically packages) available. Of course, I also need Node and possibly something else during build time, for generating my assets.
For example, my intermediate container images might look like:
And then my container images for deployment would look like this: Except that this doesn't work! Why? Because Ruby Gems aren't installed in the application directory, but instead sit in a variety of directories on the actual file system. So instead I'd need something like: So while npm/yarn allows you to avoid some of the issues with `npm install --production` to exclude the devDependencies, other tech stacks aren't quite there yet. In addition, it feels kind of crazy to ship package managers like npm/yarn/pip/composer/bundle/maven or anything else in containers that should be immutable, so in practice you'd end up with more stages, even if you knew how to carry over the packages from one image to another in the technology stacks where it's not entirely clear how to do that.I actually tried something similar to this, but it sim...
That's a reason for bundling.
The solution that npm presents solves this problem without this insanity -- you can just create a `npm-shrinkwrap.json` file to manually update sub-dependencies. If one of your dependencies used bundling, this solution becomes impossible.
Another option is package-patch. Very light, elegant and yet robust solution for fixing anything in any package you depend on yourself without waiting for maintainer to fix it.
If you want to lock the dependencies, just use a lockfile in your app, you don't need to ship the same code 10 times because your dependencies already decided to bundle their sub-dependencies.
Node package managers don't agree on lock file formats, and yarn doesn't read dependency lock files at all.
We're converging on better norms for JS dialects/module systems, but we're not all the way there yet, so I don't see much purpose dropping support for multiple formats when some people might still need some of them
If you do want to go clean-slate and drop all the ceremony at once, I recommend Deno :)
I assume that you're talking about tools rather than libraries. Any simple library can import ESM asynchronously and any ESM packages can import CJS packages even with static imports.
Tools and Node’s strict ERR_REQUIRE_ESM have been the problem with the migration.
Just save yourself some headache and publish either ESM or CJS, not both.
- dependency versions being fixed is a feature. I've had package installs fail because a dependency published a semver-compatible update that broke something. In a web context this would break the app until the dependency pulled the update or the dependent pushed an update disallowing that version.
bundling should be done as a step to build an application. it's really not needed when you prepare a library.
> - dependency versions being fixed is a feature. I've had package installs fail because a dependency published a semver-compatible update that broke something.
If this is the case please open a bug to the dependency team. And then avoid ^ ~ and * in the dependencies version you are using.
Pushing multiple versions of the same library is a bug. Semver ranges exist for a reason.
> a dependency published a semver-compatible update that broke something
The same can happen with your direct dependencies too. Just use a lockfile and only update it when the install is successful.
> In a web context this would break the app
Absolutely not. Use lockfiles. Nothing breaks until you merge that update PR.
At least for me I think maintaining a tight feedback loop between "make a change" => "verify it works" is important to not be distracted by another task. esbuild taking 100 or so ms while tsc taking 30+ seconds makes a world of difference for that.
I already have a bundle intended for distribution to web browsers, I don't need these sorts of poor practices introducing unnecessary bundling code. Nested bundling sounds like insanity to me. So I very much agree with the author that this practice never made sense in the first place, and I'm baffled that it exists. It's very much an npm anti-pattern.
It’s not so bad, working with just tsc can be a little more cumbersome as you have to running it multiple times to get esm and common js and it’s slower than some alternatives. I see it more akin to using a makefile than an anti pattern.
There are situations where a bundler is beneficial, but indeed you should not need it, as long as you just ship ESM and don't over-complicate it.
At least not yet until ESM is fully supported by all parties in the ecosystem (node, V8, major libraries and frameworks). As of Sept 2022, we still have support issues with node (V8), TypeScript and jest. I have links to the issues in one of my GitHub repos[2].
Once you move past trivial hello world apps, it is literally impossible to run JavaScript in node in pure ESM because of various bugs and incompatibilities. The user of the library will hit "Cannot use import statement outside a module" error sooner or later.
1: https://www.npmjs.com/~paradite
2: https://github.com/paradite/github-dramas#2021