Given NPM registries are presumably optimized to store text, I wonder how this scales and if other repositories that do expect binaries (eg. Maven, NuGet) compare.
NPM packages are uploaded / stored as `.tar.gz` files, so they can contain anything (alongside a package.json metadata file) - some public NPM packages contain binary files like images or compiled assets too. The registry doesn't really care about much beyond the package.json.
I've done something similar to the author at a previous company, distributing an internal development CLI written in Go, though in our case we published to the GitHub Packages NPM registry rather than private npmjs.com, so we could authenticate our developers via their GitHub accounts.
Including multiple binaries inside the published package, one for macOS and one for Linux (we had no Windows devs), did increase the package size, but the whole gzipped .tar.gz was around 20-30mb, so not terrible from a registry storage standpoint.
Publishing to an NPM registry, vs just hosting a binary for our devs to download, was convenient since we could use the registry API to check for updates and notify the user that an update was available. Users could then update via a subcommand, which just pulled the new package from the registry.
I haven't read the article yet, but I would assume they want versioning, history, checksumming and verification. All that stuff is worthwhile, it's just unfortunate npm is such a heavy dependency to achieve it.
The problem with .deb and .rpm is that their spec file format is horrendous. I've packaged multiple RPMs and every time I have to do it is a complete PITA, lots of quirks, macros, flags, etc.
The official docs are like a deep man page and do not have simple use cases, like, create a pkg for a binary
or create a pkg for a binary with systemd
or how to distribute a pkg of .so
IMO if the .deb and .rpm maintainers invested some time in modernizing their official pages with lots of quick starts and guides it would bring a lot of developers to their side.
(And in this specific case they needed more than Linux support)
"we ran into the issue of distributing the CLI tool to our team of engineers without the need of distributing a new set of tokens or making the binary public"
Seems a neat enough hack to make their newer shit available internally using existing tools/security.
I'm not going to expend any effort to circumvent medium's nonsense. The content there isn't of consistently high enough quality to justify jumping through any hoops. Any barriers they put up will just result in me not consuming content hosted there.
But that's exactly what to expect when people donate their writings to a platform they have no control over. There are always those who need to learn the hard way.
here's the summary: it opens with "We recently migrated our internal CLI tool from Node JS to Golang". It's just a way to keep using the hammer they already had, no matter if the problem is a nail or not.
could be titled "how to hack your private npm server to be a OS package manager, because we don't want to learn our OS packaging"
Same reason as you write javascript for the browser. Just because someone put it there at some point. Would it be better to use python or anything else in the browser? sure! why don't you do it? because many people have done it before and there's no adoption!
package manager and languages are just like curl or wget. try one, command not found? try the other. meh. ...until you find people that assume other people can't learn. Then they will create a third option because other-people-are-dumb and it will be worse than both options you had before.
Or since it is an engineering CLI tool, why not have engineers:
1. clone cli repo
2. run goreleaser
3. copy binary to desired path
sure, it's not 1 step (can be condensed with a build script), but it does teach the other engineers how to roll their own cli tool and contribute back if they fix a bug in it.
Or if the blog indicated that not all engineers understand golang and build tools, and there's too many teams - that'd be good for blog context. Hopefully the author didn't spend much time in figuring out or being too clever in making this "hack".
As the number of tools grow and the number of teams using/developing them grow this becomes more and more untenable. You not only have to maintain the tool itself, but also the build stacks associated with them and it becomes the task of the user to make sure they are running the most recent version. This leads to more complex support situations.
Basically, you need a deployment methodology that allows for:
* Ease of use by "novice" developers
* Only introduces minimal overhead for those developers
* Makes it easy to upgrade or remove a tool you installed 6 months ago
* Doesn't require a tool development team to come up with their own solution
This is a surprisingly tricky to do on a diverse environment with 1000+ internal users.
NPM could handle different OS, I would guess they run more than 1 os.
(It is also a tool which the team is used to handle, as they have been using node + npm)
I guess my comment appears rude, which wasn't intended, I just find npm a troublesome piece of software that is passable as a package manager for the javascript ecosystem, but not a tool one solves problems with if there are viable alternatives.
When you have one thing to nail and already own a hammer, why go and buy another one? Even if it's not a good one, it's gonna do the job, and be the least trouble.
The choice of package manage is much stickier than the choice of hammer. I can trivially change hammers between projects or even between nails. Changing package managers is a lot more involved, and gets more involved as it's used more---and they've just established a precedent for using npm to package go binaries.
Sorry I wasn't clear, I'm not making an argument for or against npm, only responding to the analogy.
But I wouldn't consider "we're already using X" to be sufficient reason to use X for something it wasn't designed around. Again, these choices are sticky and tend to be vulnerable to scope creep. I'd call it a textbook case of technical debt, and it's ok to point that out. They might have forestalled the criticism by acknowledging that, but they didn't.
Meh, depends. For their case, for an internal CLI tool that will get used interactively on development environments, it's fine, the cost of switching is not very heavy. At worst they're postponing the work of deploying on other package managers, and devs will have to run 2-3 commands to remove the npm package, configure whatever next place it gets deployed to and install the new executable.
Sure, should have s/much stickier/sticker/. I'm not going to comment on whether the tradeoffs are worth it, I just thought the hammer analogy doesn't work due to material differences in how you switch between hammers vs package managers.
I've had far more problems with yarn, which sometimes decides it and it alone not longer has internet access and none of the "fixes" published over the years have worked. Instead, you have to just set your timeout really, really high. That's been the only fix for a good while, and it frequently does not work.
Older NPM could be a pain but modern NPM is pretty smooth and works just fine.
I thought the same thing (Go binaries are so easy to distribute), but they do explain that their devs already have credentials for an internal NPM server, and they wanted to do this "without the need of distributing a new set of tokens or making the binary public". I don't think the trade-off is worth it, but at least it explains their motivation.
In their particular case, what's the trade-off that's so terrible though? They already all have npm, they had a binary to distribute, and they distributed it.
I was very tempted to reuse a Maven repository for publishing the handful of small Python utilities we had. It's a standardised format, we already had the server set up, we already had different kinds of artifacts in there and multiple languages, and frankly Python-on-Maven dependency management can't be worse than what standard Python dependency management looks like.
I kinda do that today. We’re primarily a Java shop, but we’re using Python for FaaS. So we package up our code and dependencies into a zip and use Maven to push the resulting archive into our common repo.
> As the project completed we ran into the issue of distributing the CLI tool to our team of engineers without the need of distributing a new set of tokens or making the binary public.
From the article itself, it seems like they wanted to re-use their existing process without disruptions.
At CERN we have an internal system to distribute software. It is mostly used to distribute software to production systems in the datacenters or to distribute nightlies and toolchains to developers. It replace standard package managers on most production systems.
It is a FUSE filesystem, where we can install software once and have it distributed to a practically unlimited number of clients thanks to a simple and performant cache system.
The filesystem is called CernVM-FileSystem, CVMFS for short.[1]
One of the best things about CVMFS is the idea of LCG views which are coherent installations of a large number of packages. This more or less eliminates the need for package management and guarantees that someone else can reproduce your environment.
It's like Stackage LTS releases, but everything's installed already.
There is obviously a need here that is not met by current tools.
I believe that Conda is the closest thing to an answer to this need.
It works on Windows, Mac and Linux. It supports packaging arbitrary binaries. It lets you specify dependencies that can be arbitrary libraries for arbitrary languages. And it has a relatively simple package format, build process, and repository format.
It's not the best piece of software when it comes to user experience, and its niche is still very much data science. But it is a glimpse of what should be possible, but is not currently available.
64 comments
[ 2.2 ms ] story [ 114 ms ] threadI've done something similar to the author at a previous company, distributing an internal development CLI written in Go, though in our case we published to the GitHub Packages NPM registry rather than private npmjs.com, so we could authenticate our developers via their GitHub accounts.
Including multiple binaries inside the published package, one for macOS and one for Linux (we had no Windows devs), did increase the package size, but the whole gzipped .tar.gz was around 20-30mb, so not terrible from a registry storage standpoint.
Publishing to an NPM registry, vs just hosting a binary for our devs to download, was convenient since we could use the registry API to check for updates and notify the user that an update was available. Users could then update via a subcommand, which just pulled the new package from the registry.
YUM is almost old enough to drink in Europe.
The official docs are like a deep man page and do not have simple use cases, like, create a pkg for a binary
or create a pkg for a binary with systemd
or how to distribute a pkg of .so
IMO if the .deb and .rpm maintainers invested some time in modernizing their official pages with lots of quick starts and guides it would bring a lot of developers to their side.
(And in this specific case they needed more than Linux support)
Seems a neat enough hack to make their newer shit available internally using existing tools/security.
Well, that's about the end of medium I think.
Only website that punishes being logged in.
could be titled "how to hack your private npm server to be a OS package manager, because we don't want to learn our OS packaging"
i call it "lazy opinionated adoption".
Why do you use apt in debian and yum on redhat?
Same reason as you write javascript for the browser. Just because someone put it there at some point. Would it be better to use python or anything else in the browser? sure! why don't you do it? because many people have done it before and there's no adoption!
package manager and languages are just like curl or wget. try one, command not found? try the other. meh. ...until you find people that assume other people can't learn. Then they will create a third option because other-people-are-dumb and it will be worse than both options you had before.
Why didn't they use the os package manager? Which OS? Their build clearly builds for osx, windows, and Linux.
1. clone cli repo 2. run goreleaser 3. copy binary to desired path
sure, it's not 1 step (can be condensed with a build script), but it does teach the other engineers how to roll their own cli tool and contribute back if they fix a bug in it.
Or if the blog indicated that not all engineers understand golang and build tools, and there's too many teams - that'd be good for blog context. Hopefully the author didn't spend much time in figuring out or being too clever in making this "hack".
Basically, you need a deployment methodology that allows for: * Ease of use by "novice" developers * Only introduces minimal overhead for those developers * Makes it easy to upgrade or remove a tool you installed 6 months ago * Doesn't require a tool development team to come up with their own solution
This is a surprisingly tricky to do on a diverse environment with 1000+ internal users.
I bet some day we'll have a Linux distro with npm as the system package manager here on HN.
But I wouldn't consider "we're already using X" to be sufficient reason to use X for something it wasn't designed around. Again, these choices are sticky and tend to be vulnerable to scope creep. I'd call it a textbook case of technical debt, and it's ok to point that out. They might have forestalled the criticism by acknowledging that, but they didn't.
Not a big deal at all.
Older NPM could be a pain but modern NPM is pretty smooth and works just fine.
From the article itself, it seems like they wanted to re-use their existing process without disruptions.
[0] Abraham H. Maslow (1966). The Psychology of Science.
It is a FUSE filesystem, where we can install software once and have it distributed to a practically unlimited number of clients thanks to a simple and performant cache system.
The filesystem is called CernVM-FileSystem, CVMFS for short.[1]
I maintain a public instance of it under https://packages.redbeardlab.com
It contains "common" software, like standard unix tools, compilers, webservers, etc...
I wrote a bit about it:
https://redbeardlab.com/2020/12/20/announcing-packages-redbe...
https://redbeardlab.com/2020/12/21/bootstrapping-packages-re...
https://redbeardlab.com/2020/12/21/packages-redbeardlab-com-...
https://redbeardlab.com/2020/12/23/packages-redbeardlab-com-...
[1]: https://cernvm.cern.ch/fs/
It's like Stackage LTS releases, but everything's installed already.
Fortunately Nix helped in creating the same coherent installation of packages.
Then `source /cvmfs/packages.redbeardbal.com/setup.sh` give you the same feel of an LCG view, but with more widespread packages.
I believe that Conda is the closest thing to an answer to this need.
It works on Windows, Mac and Linux. It supports packaging arbitrary binaries. It lets you specify dependencies that can be arbitrary libraries for arbitrary languages. And it has a relatively simple package format, build process, and repository format.
It's not the best piece of software when it comes to user experience, and its niche is still very much data science. But it is a glimpse of what should be possible, but is not currently available.
Why? Out of things to do? Wanted to look productive?