I agree with this. But dont just use `npm ci`on prod builds since that would typically include all the dev dependencies as well in your production builds, which is not usually desirable. It might be possible to add the `--only=production` flag to npm ci? But otherwise, as pointed out, pinned versions are needed for all dependencies.
Last time I know during node 16, npm install updates your package-lock.json, but for patch version number (unsure about minor version) only and if the package.json is using relative version number. Definitely use npm ci or yarn if you want to adhere to lock files.
> if the package.json is using relative version number
What do you mean by this? You can specify dependency versions using "^" which means "I accept new minor and patch versions" or "~" which says "I accept new patch versions", or a bare version to say I want an exact version; but none of those have anything to do with a bare "npm install" command. They are relevant when you install a new package, with "npm install <package>". That is a totally different operation, and it absolutely can update versions when needed.
However, when you just do "npm install" it does not update your package-lock.json to the latest version that matches your semver requirements in package.json. That will only happen if you install a new package that needs it or you use `npm update` or some other command.
This is easy to test but it's also intuitively correct- you don't expect to run `npm install` and have an uncommitted change to `package-lock.json` appear, and that does not happen in normal usage.
It is kind of unfortunate that "npm install" and "npm install <package>" use the same verb of "install" for two different things- other package managers don't make this mistake and it avoids the sort of misconception we're running into here.
Exactly what I said, I have experience with ~ but not with ^ and have never used it. Let's say that you have a package version ~1.2.11 and have 1.2.11 as installed version in package lock. Then if let's say 1.2.13 is out there, npm install will update package lock to that one. npm ci won't change that. package.json file will kept unchanged though.
No, "npm install" doesn't actually do that. In the situation you are talking about, "npm install" and "npm ci" behave the same.
It makes sense that they behave the same here right? What is the point of a package lock if just installing packages on a new copy of a codebase updates the dependencies?
Package locks aren't just about deploying: As a developer, I need to be assured that the code I'm running is the same as my coworker.
Yes I think you are right when NPM first introduced it, which was in NPM 5 [1]. At that time package locks were new.
Node 16 shipped with NPM 8, and by then this was not the behavior any more. I don't remember exactly when, but I believe it might've been in NPM 7 when this changed. Node 15 would have been the first release to ship with at least NPM 7 [2]
Honestly no guarantees it worked as intended right at that release, but the intention is clear.
EDIT 4: I realize that the following may present retort but it matches my mental model perfectly:
> If you manually edit your package.json to have different ranges and run npm i and those ranges aren't compatible with your package-lock.json then the latter will be updated with version that are compatible with your package.json. Further runs of npm i will be as with 2 above.
This is what I've mentioned elsewhere: You should not update package.json's versions and deploy it and expect that versions won't change when NPM install runs. In fact, npm ci will just fail in this situation, which is a Good Thing (TM), and is documented as such.
LAST EDIT SRSLY:
It is conceivable, especially given OPs other red flags, that perhaps they were on an older Node/NPM like Node 14.
That's another practice that I regularly do, pin NPM itself in deploy scripts. While I was using Node 14, I was using NPM 8, because you can just "npm i npm@8 -g" to upgrade your NPM regardless of Node version.
To be fair, JavaScript is harder that most people think, the problem is that everyone thinks they are an expert, so we end up with this kind of problems.
There is a lack of engineering discipline and too much hacking!
Javascript is way easier than most people think. The problems come when they try to force their preconceived notions on it, and/or they don't keep it simple. Sure, complex things can be done with Javascript but as you said, it takes discipline.
tldr: A dependency wasn't pinned, the dependency changed, and stuff broke. Author thinks Javascript sucks.
I will say this though: it is infuriating that package managers (every single one I've used, not just NPM) don't default to pinning exact dependency versions, and instead let it "automatically" update minor versions.
I used to maintain some open source libraries and absolutely no change we would release would break anything and definitely not over a minor version. We even did a major rewrite that completely changed the API and we broke nothing, but it did mean your project got a lot of deprecation errors.
And our docs would mention both the old and new way so if you were searching the Internet for help, you wouldn't get confused when you saw wildly completely examples.
>The only change that isn't "breaking" is one that only adds functionality.
Ha! Think again. On of the more annoying problems I had to analyze was when a new PHP version added a function that already existed in my application (this was before namespaces). The runtime did not like that.
Patch or major is always a judgement call. Making all changes a major jump is a bad idea, because it means that dependent packages are less likely to adopt important functionality or security fixes.
NPM already gives you a way to control when version changes happen, it's called package-lock.json, and it's used for all NPM installs (both in development and when deploying). Just don't listen to the bootcampers on StackOverflow telling you to delete it whenever anything looks weird.
If you delete package-lock.json, assume every dependency version has changed and regress accordingly. The same is true of all reasonable version-locking package managers.
That’s why for building your JS bundle you’d use „npm ci“ instead of „npm install“ to avoid random upgrades of your packages. It uses your vc’ed „package-lock.json“ file containing your established dependency tree to build your JS bundle.
However, if you need to integrate a new dependency it might require a later version of a package you already installed. NPM is clunky at times but most of the times it works quite well — despite all the micropackaging madness.
NPM uses your package-lock.json for normal installs as well. "npm ci" behaves nearly identical to npm install except for a few things, detailed here: https://docs.npmjs.com/cli/v8/commands/npm-ci
Note that actually using package-lock.json to resolve dependency versions is not actually a difference between npm install and npm ci
The real benefit is that npm ci will refuse to work if package.json and package-lock.json disagree, which protects against changing package.json without running NPM install during development.
This is not the default for NPM, Yarn, or PNPM. The "package-lock.json" is what does the pinning, and it's exact version.
If you listen to very bad advice though, half of the Q&A answers on the Internet suggest all problems can be solved by deleting package-lock.json, to be fair.
I didn’t write the first quote, but a comparison to an opinionated strongly typed language where types are a first class citizen, and a typing sandbox with nonsense constructs enabled by antipatterns such as unions (‘string|number’ anyone?)… these are not the same things.
What I get on the issue is that library a using library b which is different version. In this case yes it will solve the issue during tsc or build, only if skipLibCheck flag is turned off.
Maybe? The discipline of using typecript and carefully distinct commits might not have solved this particular problem, but I suspect a team that does this would not be facing this issue as a Friday night 10 hour debugging session.
TypeScript types work only on compile time. If library changes without changing its public TypeScript signatures it won't help at all.
Also, after coming from real compiled languages I find TypeScript ecosystem not to be robust at all. Change in one file sometimes does not result in compile time error in another file where changed type is used. Yes, it's not an issue with TypeScript itself but with its ecosystem, transpilers and compilers, but that's not something I experienced ever with C# or Java.
The types should be shipped with the code which is nearly ubiquitous for NPM packages these days (at least the ones you should actually be using). If they were, you would catch the issue as soon as you built the app, which should be happening in a CI or deploy process (never commit your compiled code).
> Change in one file sometimes does not result in compile time error in another file where changed type is used.
Sounds to me like you are missing types or the types you have are incomplete.
When you do things correctly, this is not an issue. Part of doing things correctly is preferring (and in some cases requiring) that your dependencies either be written in Typescript or at least ship the types directly in the NPM package, per best practice.
Granted, I'm a novice in TS/JS world after decades in strongly typed environments so it's possible that I'm doing something wrong.
I'm not talking about npm, I experienced these issues in my own code. It's mostly related to ts transpilers. They're fast, they're used in build systems which should be latest and greatest (e.g. Vite), but they're not robust, they don't always catch type changes in dependent files. And now I can't just change a type and rely on compiler to check ALL occurrences in entire code base. Because it's not really a compiler but a transpiler and it misses some things. Also, tsc doesn't seem to handle jsx/tsx, is there a way to perform real type script compilation with full type checks on tsx files?
Sounds like you are talking about frontend development. In that case you are trading off speed for accuracy. Some dev server tooling has this issue.
Most systems built on Vite though do check across files, so likely that's just a bug in the build system you were using. A bug that is likely fixed.
One thing you can be assured of is that when you actually build your app (hopefully using CI), it will find those problems, because you are not relying on hot reloading to do it.
On the backend side it's far less common to use a build server model, and personally not very warranted -- building a frontend app can take a lot of time and the feedback cycle needs to be short for a dev to be effective. On the backend, a small amount of time rebuilding the app before it is rerun is not nearly as costly. If your Typescript codebase is so large that the feedback cycle of rebuilding is too long, well you have to take the tradeoff, but it's a rare problem.
Understand that the only way a "dev server" style backend build process is saving you time is by not checking the types during recompilation. It wouldn't be surprising that it would miss typing changes.
Yes it's for the frontend, but it's not really related to hot reload thing. Is there a way to really compile tsx? I couldn't find any, tsc doesn't support it so you have to use some of the transpilers, and they don't perform full type check across files. So there's no way to perform real "full build" when tsx files are used, even if you don't care about build duration. If there is, I would love to hear about it.
Yes it very likely would have. Most libraries have their types shipped alongside the JS, and if that's where the change was, it would have been caught during build.
I'd argue that even if the types for the changed package were from DefinitelyTyped, it probably still would have solved the issue if the developer had just blindly updated some package versions-- they likely just upgraded everything, ignoring semver.
I didn't quite understand why they had to debug on the spot, thus working like 36 hours uninterrupted or so.
A sane working environment would file a priority 0 bug and address it during working hours.
I'm not throwing a rock but oftentimes this sort of heroic behavior is self-inflected.
Also why is heroic debugging required on something that is not production? The article mentions it was a "quick demo". Then who gives a shit? There's time to investigate. And this would/should NEVER happen to production. The whole point of testing and staging/UAT environment is to catch this sort of problems. Again, a normal day at work in software engineering and not something to kill yourself living on amphetamine like a commando behind enemy lines in WW2.
Jup, and if your employer doesn't stop you or even expects such behavior that is called a bad work culture.
Sure, I would also do long programming sessions on my own things every now and then, but those happen because I am in the flow and it feels like more work to come back into it the other day. But those are my own projects. I don't owe this to my employer. If they want that they could offer me double my current salary and even then I would insist on only doing it when it makes sense to me.
In my case my superior comes by in my office or sees I made a commit and tells me I should stop working (I tend to forget to look at the time if I am in a flow).
But I also live in Europe and my superiors want me to come in with a fresh mind, instead of burning myself for things that should be normal work.
When you are not experienced on the platform you are building on, all debugging is heroic.
It's really strange that they would invest all this time as if they are keen to understand the cause of why it happened, but then be satisfied when jiggling the handle solved it. The real cause ultimately is a process one (developer discipline) or a pipeline one (bad build/test/deployment processes). They got close to realizing it and then blamed the language instead of what appears to be bad practices happening on their dev team.
While I agree with everything you have said, and it's been my motto since I read, "I'm sure you're joking, Mr. Feynman." The point of the post was to point out (maybe I did it wrongly) that those things you have described are pretty impossible given the ecosystem we have nowadays in Javascript (as a whole!). Of course, you can try to theorize and try to get a hypothesis on things and then try to prove those, but it's just too hard to keep track of everything.
In addition, we are using lock files and good practices but it's just too much to handle for our small team (small startup) so we will plan to migrate onto a more stable platform.
It's really funny the heated arguments this post generated but I have been using Javascript for 12 years to these days, still this looks as a nightmare to me.
Tsdoing has a good video trying to install React from scratch, that's exactly how I felt during this debugging process
Another random story from this week: we have been using `nanoid` to generate IDs internally for an internal tool. Apparently, they did a breaking change release, and now you cannot use it anymore on commonJS env. See my point? where's the hypothesis we can make? It's hours and hours spent tinkering with code and other people's dependencies, and if, as you said, put a theory that theory won't match if you think how real systems work (the networking section on the blog post)
Well your reply still doesn't explain the administrative (not technical) reason why you had to spend some 36 hours uninterrupted hunting for this bug. Particularly since it was 'a demo' and what broke was 'PDF export', while other exports still worked. Hardly the end of the world if you ask me.
Technically though, I have hardly met this dependency update problem with C++ simply because dependencies are rarely if ever updated. Android on the other hand was a nightmare. I had colleagues quitting in a rage and swearing to never again touch Android in their life. Probably half the effort was spent fixing crap that was broken by updated dependencies. Luckily I'm back to C++ now and hope never again will I have to touch such a crazy system. The whole 'update for the sake of update' seems to be a grave problem of the Android / Kotlin / Javascript universe. The mentality is like 'bread won't be sliced anymore, eggs won't hatch, sun won't rise if we don't absolutely have the latest bleeding edge alpha version of all libraries'.
> The point of the post was to point out (maybe I did it wrongly) that those things you have described are pretty impossible given the ecosystem we have nowadays in Javascript (as a whole!).
That's the thing, nothing you have asserted as impossible actually is as you'll see throughout this thread. I have been writing JavaScript/Typescript professionally since for about the same amount of time and I am not intimidated or challenged by my projects in the way you lament about.
That being said, if you've been at it 12 years and this is still your opinion, fair enough, try something else then.
You don't have a mechanism of trust in Javascript. Typescript up to this point is just sugar syntax. npm is a dumpsterfire (see npm everything package)
If you are not intimidated by the complexities of this Babel tower, it's up to you. For me, this was just a personal catharsis in my personal blog that I wrote mostly for me (and I didn't share it here)
I believe there's no point on discussing this with particularly you since you have been commenting all around trying to just confirm your view instead of just comprehend that this is not sustainable for new developers (try to follow the conversations on packaging, lol)
It's simply too much burden just to maintain someone else mess. We will pay a high bill in the future for all of this.
> I believe there's no point on discussing this with particularly you since you have been commenting all around trying to just confirm your view instead of just comprehend that this is not sustainable for new developers
I don't need to confirm my view at all. Advocating for one's view doesn't imply a lack of conviction, rather the opposite.
No need for a personal attack here. It's definitely sustainable for new developers who learn the stack. It has a learning curve but so does everything else. I've built applications in just about every stack there is (except for the more esoteric stuff like OCaml, Erlang, etc) and over those years complaints like those you raised are easily levied against any language or stack that is practically used in the industry. The common thread is that these sorts of complaints arise when there are more unknowns for a developer than knowns-- and as developers we are in the business of eliminating unknowns. When you encounter unknowns and your response is "this is why this tool isn't fit for use" instead of "let me turn unknowns into knowns", this is often the result.
> (try to follow the conversations on packaging, lol)
There isn't a packaging system out there without complications, but my posts regarding NPM's behavior points to how it actually already works to reduce or eliminate the version pinning failures you appear to have hit. Now if you are using very old versions of the tooling or if you do things like remove or ignore the version pinning tooling and how it should work, then yes the kind of problems you hit can happen. But that does not mean this problem isn't already solved, or at least the problem becomes an education one; which is why I'm out here trying to solve the education problem.
> It's simply too much burden just to maintain someone else mess. We will pay a high bill in the future for all of this.
That's, like, your opinion man. I've yet to find a development stack without it's costs, and once you are fluent in how to work in the ecosystem, Javascript's are not significantly higher than any of the others. And personally the cost/benefit of the Typescript ecosystem for me is quite rosy compared to the limitations of other stacks. I'm clearly not alone in that calculus.
In a statically compiled language, when APIs change, you get compile errors that show up at build time. It's usually pretty obvious. With dynamically typed errors, you get weird run-time failures that can be well hidden until you trigger a use case that actually needs the bit that changed. That combined with poor test coverage means a lot of potential for things to slip through build and QA processes.
I use Kotlin, both on the JVM and in the browser (via kotlin-js). Dependencies are equally complex as in javascript but it's much easier to find out if stuff breaks or not. I update often to minimize the amount of changes I have to deal with. This may sound counter intuitive but integration testing small deltas is exponentially easier and less risky than big deltas.
My attitude to having big updates lurking is that they represent technical debt. Just because you don't know you have a problem because you haven't updated yet doesn't mean you don't have a problem. By not updating, you just increase your ignorance of technical debt. It might be piling up without you realizing. And you might be making things worse by working against obsolete APIs.
Upgrading libraries often means I typically only have to deal with a handful of minor updates for libraries that are mostly unproblematic. And if stuff breaks, it's easier to pin point the source of the breakage. And I find out as early as possible about problems I need to deal with. If I don't know about it, I can't deal with it either. Sometimes downgrading a library temporarily is needed. But then you document why and stay on top of issue trackers and upstream development to see when you can upgrade or you start looking for alternatives.
Occasionally, some API breaking changes happen and then I fix it (because the compiler tells me stuff is not compiling). If I don't update for a year, I probably have a lot of breakage on my hands and a lot of work fixing things. Much harder and much more time consuming. And risky. All I know then is that I probably have a lot of problems (i.e. technical debt) on my hands that I don't know about.
That's why I use kotlin-js. It's like typescript with the training wheels removed. I always joke that typescript is a bit of a gateway drug for web developers that know javascript to actually get into other languages. Kotlin-js is a bit of a niche thing of course and soon going to be obsoleted by Kotlin's wasm compiler. But kotlin-js actually works very nicely. Just like typescript, it's a transpiler. Unlike typescript, it does not really have a non strict mode (except for the dynamic keyword so you can deal with some js libraries and its weird type system).
Fun, all the power to ya on that. I've done a bunch of Kotlin for Android and although I initially didn't like a lot of how it does syntax, it has grown on me a lot. It is definitely a language that is focused on correctness and convenience.
That being said, there isn't too much about Kotlin I miss when I write Typescript in strict mode.
> Unlike typescript, it does not really have a non strict mode (except for the dynamic keyword so you can deal with some js libraries and its weird type system).
Why is the existence of non-strict mode a concern for you if you are using strict mode? After all, whether you are using Typescript or kotlin-js, you are likely to have JS dependencies, and those might not be written with Typescript strict mode. I guess you could just opt not to use any "native" (Javascript) dependencies and just do it all in Kotlin JS, but beyond a certain point that's not very practical.
Strictness should not be optional. A lot of typescript projects don't use all the features. I've been on projects where people were just working around those features to do all the crazy hacks javascript is famous for; or to be able to use libraries that use those hacks. It's a constant fight to push back that madness. With Kotlin, that's just much less of a concern. Having a language that just says no to all of that is valuable. IMHO typescript could just remove the option to turn off the strict mode and always be strict. It would be a better language. It's impractical of course because a lot of stuff would break. That's the problem with it. Don't get me wrong, I've actually used typescript and don't mind it. But I prefer kotlin.
I do indeed do a lot in Kotlin-js rather than using javascript libraries. We make a few exceptions for libraries like map-libre for which we have a very thin hand crafted Kotlin facade. Things like react we simply don't use at all (we use a pure kotlin framework called fritz2). We actually use a lot of Kotlin multi platform libraries as well. And even some wasm libraries (e.g. sqlite). Most of the foreign code dependencies in our codebase are not javascript libaries.
Well, strictness actually is optional in Kotlin too, since you can lateinit things and !! your way through nullability like in any language. And of course if you are consuming Java stuff this is more likely, since Kotlin exposes nullable types for everything in Java, so sometimes the right call is to non-null assert since the interface you are consuming can't represent non-nullability.
>With dynamically typed errors, you get weird run-time failures that can be well hidden until you trigger a use case that actually needs the bit that changed.
Maybe you've never heard of "writing tests"? If you update a dependency, and the dependency has a breaking change, your tests will fail. It's really that simple, and this should be the same in any language, not just Javascript.
This is the second time I've ended up reading this post, and I confess I still don't exactly understand what was going on here. There's an issue with PDFs missing their contents, but also timeouts and a missing method exception? The problem seems to have come from upgrading some dependencies at some point, but the solution is also an upgrade, as far as I can tell? And then the cache issues at the end sound very much like a red flag in terms of managing JS dependencies - as if they're trying to cache the node_modules directory in some way that isn't properly configured.
That said, the main sense I get is that the team aren't so experienced with Javascript, and therefore working with it is full of surprises because they just aren't used to it. I found the line about bugs in C code being "your fault" particularly amusing: having occasionally had to touch C and C++ code, I would have said the complete opposite! Javascript, for me, is very predictable - if something goes wrong, then it's usually because I made a clear mistake. But with C, I end up trying to debug chains of Makefile and CMake problems that have no rhyme or reason to them.
In other words: as someone inexperienced in C, it is a chaotic, magical language and ecosystem to me, in exactly the same way that the author describes Javascript being to them.
More practically, it sounds like the team need some regression tests for this case that they can run next time they try and upgrade their dependencies. They also need to make sure that caching is working properly for their Javascript code - rerunning the CI multiple times on the same commit should always produce exactly the same dependency tree every time. Probably that means caching the right directory and using NPM CI to clear and rebuild node_modules from scratch every time.
More generally, the author complains about a lack of type checks. In this case, it sounds like the missing method was somewhere in a dependency, so this may not have helped, but: Typescript! It exists, it adds types, it makes a lot of things easier. Easy mode Typescript is to just start writing `.ts` files and use the `tsx` package to automatically compile and execute everything - there are other approaches that they could explore, but this is probably the simplest.
Stuff that gets pulled in when you do 'npm install' is scary for a more traditional server-side developer. Python for example comes with a lot of built-in libs maintained by the core python team. These could get you pretty far before you start pulling in libs from random authors.
But that's exactly the sort of thing I'm talking about. We find scary what we're unfamiliar with.
To be clear, I agree that the Javascript ecosystem isn't perfect, particularly when it comes to supply chain issues. But in a lot of ways it's more secure than Python's typical dependency setup. By default, NPM uses a lock file, which enforces that specific versions of existing packages are used unless explicitly updated. This means that you always know which versions of which dependencies you're currently using, it allows you to upgrade different parts of your dependency tree independently, it ensures that when you remove a dependency, all of its transitive dependencies also get removed, and it allows for different dependencies for production and development environments, and for different portions of a given project.
Doing all - or even part - of that in Python is difficult. There's no built-in way, and so people tend to adopt third-party tools or roll their own scripts. The roll-your-own process, in my experience, almost inevitably leads to some sort of failure, usually a certain updating unexpectedly and breaking everything, and then not being able to roll back to a previous version of that dependency. The third party tools (Poetry, Pip-Tools, etc) work better, but are often difficult to integrate more widely into the Python ecosystem.
I don't want to make this about Python vs Javascript - both are tools that make different tradeoffs and can be valuable in different ways. The point here is that, as software developers, we often end up with what are essentially superstitions about other programming languages - we describe things we don't know or don't understand as "magical" or "scary", because that's easier than admitting we're just not that well informed. And while it's okay not to be informed about everything, in situations like this, it can lead to bad outcomes.
In the article, the author ends up blaming the situation on the magic and weirdness of Javascript, but it sounds like the issue had more to do with them not understanding how best to do package management in Javascript.
> I deleted all cache on my local environment, all node_modules references, all lock files. I ran an installation process once again and exported again… it didn’t work.
> The next message I sent: “We found it!”
May be I misunderstand something, but from the post it sounds that they don't even commit their package-lock.json. They're using node like its 2012.
This was very funny to read because as someone who is actually familiar with these JS builds I immediately guessed that a dependency had changed. That's literally the first thing I'd check if an app is behaving differently in a remote environment than it is locally. I'd recommend the OP learn some JS best practices and apply them rather than complaining that JS isn't C.
I'm honestly split on this. It is obvious that they are not experienced with js ecosystem, but is it fair to frame it as they obviously haven't been burned enough by the ecosystem to internalise the common issues?
Dependency silent upgrades and dependency hell and bloat are not unique to javascript, but it feels like the js community has a perverse fetish for fragile build systems and dependencies.
Ofcourse, the community is not one organised entity and this is merely the result of multiple individual decisions taken independently, but still. I can not help but empathisewith the author despite seeing that they aren't locking deps, not doing reproducible and rollback-capable deployments via docker or artifact tarballs, don't seem to have robust ci, or good work life balance..
Is it elitist to remark "haha, don't you know you should smerp the blergh?"
Is it not atleast somewhat crazy that we normalise rube-goldberg-esque contraptions?
I don't even know what I'm getting at, at this point honestly. Js gives me very mixed feelings. I love the accessibility amd universally available execution runtime (browser), and understand why it hassome of the warts it does, but it is also honestly a scary ecosystem with hyper fragile tooling. I've personally burnt multiple full days on webpack config tweaks and it all feels so unnecessary and pointless.
I think the problem is, JS is actually quite a minimal language as far as standard library is concerned. Node is a good runtime (as is deno), but imho, they really missed the boat on building a standard library and making it a real ecosystem. .NET, Java, and Python, arguably the top most used languages besides JS, have an insanely more extensive standard library than JS does.
Packages built by 3rd parties are the Wild West, and unless they’re a full-on company of dedicated, experienced engineers, they’re gonna break semver best practices. They’ll also design bad APIs for interacting with their packages sometimes, and there will be a total lack of consistency across the ecosystem, which is what we have currently.
If I had more free time, building a stdlib for JS would be high on the list of “help the world” activities.
The problem philosophy that a few HTTP requests need to be wrapped in a library that includes hundreds of other libraries is not a JavaScript problem. It's a social problem.
Agreed, when they popped out this "such and such isn't a function" I lost the thread. You would know about that before you even started if you just read some logs. Unless they don't have logs, in which case a choice of language isn't going to solve their problems.
Handling uncaughtException and unhandledPromiseRejection is a common mistake. You should not do it unless you have a good reason. Your program should die for science, and you should fix the place where you didn't handle that exception or rejection.
Unless of course your handler results in dying for science, then it's fine (ie for sending to an observability service first)
Are these direct deps? Because just a cursory look raises all kinds of red flags. There's a disclaimer for the former warning about it being in developer preview, and the rest of the README has a bunch of notes about edge cases.
I get that the aws-sdk uses them, but if it works for other people it should be fine. Having these as direct deps and also calling in complex libs like the aws-sdk honestly seems like a recipe for disaster.
The dependency update (i.e package.json lock file change) as I understand it was unrelated to their feature, and should have gone in on its own, and it should have broken CI if it was a serious piece of functionality that warranted a late night debugging session.
It sounds to me they rolled in the dependency update with their features, then had a massive surface area to cover when it flaked out.
Exactly, and since this is AWS SDK for S3 we're taking about, and I depend on that same SDK in my own projects and they aren't broken (including using some of the more advanced streaming parts of it), I'm guessing there was a major version update applied somewhere in the dependency tree when it shouldn't have been.
This sounds like a team ignoring a huge amount of best practices for JavaScript development. Some combination of the following were not observed:
- Use Typescript.
- Practice version locking on your dependencies and make sure it works correctly throughout your processes
- Do not commit vendored dependencies
- Do not commit compiled JavaScript
- Use a CI process for deployment which runs your tests
- Have tests
- Prefer (or require) types for your dependencies, or dependencies written in Typescript
- Prefer dependencies that ship the types in package (and not via DefinitelyTyped)
Some additional thoughts, they act like running out of memory is something they need to actively handle in the codebase-- it's not in this case. It shows a lack of familiarity with how JavaScript runtimes work which hints at a team which is very inexperienced with the platform. JavaScript runtimes enter GC lockup and ultimately crash if there is no memory, so this is a logging problem, not an error handling problem. There is no mechanism for the runtime to inform the app that an allocation failed.
The real question the OP should be asking is not which language to use, but how this change managed to pass unit and e2e tests and be deployed to master in the first place. Fair enough javascript has its sharp bits, but those should show up in proper commercial level of unit and integration testing.
100 comments
[ 6.8 ms ] story [ 173 ms ] threadFeels more like an oversight from the developers rather than a JavaScript problem.
When you want to upgrade, only then do you run all regression tests and do so.
However, "npm install" also adheres to your package-lock.json, and does not update things on you.
npm ci is just npm install with a few tweaks that make it better to use for CI and builds, see here: https://docs.npmjs.com/cli/v8/commands/npm-ci
What do you mean by this? You can specify dependency versions using "^" which means "I accept new minor and patch versions" or "~" which says "I accept new patch versions", or a bare version to say I want an exact version; but none of those have anything to do with a bare "npm install" command. They are relevant when you install a new package, with "npm install <package>". That is a totally different operation, and it absolutely can update versions when needed.
However, when you just do "npm install" it does not update your package-lock.json to the latest version that matches your semver requirements in package.json. That will only happen if you install a new package that needs it or you use `npm update` or some other command.
This is easy to test but it's also intuitively correct- you don't expect to run `npm install` and have an uncommitted change to `package-lock.json` appear, and that does not happen in normal usage.
It is kind of unfortunate that "npm install" and "npm install <package>" use the same verb of "install" for two different things- other package managers don't make this mistake and it avoids the sort of misconception we're running into here.
It makes sense that they behave the same here right? What is the point of a package lock if just installing packages on a new copy of a codebase updates the dependencies?
Package locks aren't just about deploying: As a developer, I need to be assured that the code I'm running is the same as my coworker.
Node 16 shipped with NPM 8, and by then this was not the behavior any more. I don't remember exactly when, but I believe it might've been in NPM 7 when this changed. Node 15 would have been the first release to ship with at least NPM 7 [2]
EDIT: Node 16.0.0 shipped with NPM 7.8.0
https://github.com/nodejs/node/blob/main/doc/changelogs/CHAN...
EDIT 2: A StackOverflow confirming NPM 5 behaved the way you remember: https://stackoverflow.com/questions/45022048/why-does-npm-in...
EDIT 3: And, as that SO documents, 5.4.2 changed the behavior to prevent this, here's the GitHub issue: https://github.com/npm/npm/issues/17979#issuecomment-3327012...
Honestly no guarantees it worked as intended right at that release, but the intention is clear.
EDIT 4: I realize that the following may present retort but it matches my mental model perfectly:
> If you manually edit your package.json to have different ranges and run npm i and those ranges aren't compatible with your package-lock.json then the latter will be updated with version that are compatible with your package.json. Further runs of npm i will be as with 2 above.
This is what I've mentioned elsewhere: You should not update package.json's versions and deploy it and expect that versions won't change when NPM install runs. In fact, npm ci will just fail in this situation, which is a Good Thing (TM), and is documented as such.
LAST EDIT SRSLY:
It is conceivable, especially given OPs other red flags, that perhaps they were on an older Node/NPM like Node 14.
That's another practice that I regularly do, pin NPM itself in deploy scripts. While I was using Node 14, I was using NPM 8, because you can just "npm i npm@8 -g" to upgrade your NPM regardless of Node version.
[1] https://blog.npmjs.org/post/171556855892/introducing-npm-ci-...
[2] https://nodejs.org/en/about/previous-releases
>> ground suddenly became air out of nowhere
> Lol, skill issue!
Not sure why I can't downvote articles, ah well.
I will say this though: it is infuriating that package managers (every single one I've used, not just NPM) don't default to pinning exact dependency versions, and instead let it "automatically" update minor versions.
Otherwise, it changes behavour, and has the potential to break your software.
In practice, almost all changes should be major version bumps.
Anyway that's besides the point. Let me control when version changes happen!
And our docs would mention both the old and new way so if you were searching the Internet for help, you wouldn't get confused when you saw wildly completely examples.
It's all about craftmanship.
Ha! Think again. On of the more annoying problems I had to analyze was when a new PHP version added a function that already existed in my application (this was before namespaces). The runtime did not like that.
NPM already gives you a way to control when version changes happen, it's called package-lock.json, and it's used for all NPM installs (both in development and when deploying). Just don't listen to the bootcampers on StackOverflow telling you to delete it whenever anything looks weird.
If you delete package-lock.json, assume every dependency version has changed and regress accordingly. The same is true of all reasonable version-locking package managers.
However, if you need to integrate a new dependency it might require a later version of a package you already installed. NPM is clunky at times but most of the times it works quite well — despite all the micropackaging madness.
Note that actually using package-lock.json to resolve dependency versions is not actually a difference between npm install and npm ci
The real benefit is that npm ci will refuse to work if package.json and package-lock.json disagree, which protects against changing package.json without running NPM install during development.
If you listen to very bad advice though, half of the Q&A answers on the Internet suggest all problems can be solved by deleting package-lock.json, to be fair.
Well, here is your problem.
Yes?
The point still stands.
> getting huge “silver bullet” sentiment from your replies
???
Also, after coming from real compiled languages I find TypeScript ecosystem not to be robust at all. Change in one file sometimes does not result in compile time error in another file where changed type is used. Yes, it's not an issue with TypeScript itself but with its ecosystem, transpilers and compilers, but that's not something I experienced ever with C# or Java.
> Change in one file sometimes does not result in compile time error in another file where changed type is used.
Sounds to me like you are missing types or the types you have are incomplete.
When you do things correctly, this is not an issue. Part of doing things correctly is preferring (and in some cases requiring) that your dependencies either be written in Typescript or at least ship the types directly in the NPM package, per best practice.
I'm not talking about npm, I experienced these issues in my own code. It's mostly related to ts transpilers. They're fast, they're used in build systems which should be latest and greatest (e.g. Vite), but they're not robust, they don't always catch type changes in dependent files. And now I can't just change a type and rely on compiler to check ALL occurrences in entire code base. Because it's not really a compiler but a transpiler and it misses some things. Also, tsc doesn't seem to handle jsx/tsx, is there a way to perform real type script compilation with full type checks on tsx files?
Most systems built on Vite though do check across files, so likely that's just a bug in the build system you were using. A bug that is likely fixed.
One thing you can be assured of is that when you actually build your app (hopefully using CI), it will find those problems, because you are not relying on hot reloading to do it.
On the backend side it's far less common to use a build server model, and personally not very warranted -- building a frontend app can take a lot of time and the feedback cycle needs to be short for a dev to be effective. On the backend, a small amount of time rebuilding the app before it is rerun is not nearly as costly. If your Typescript codebase is so large that the feedback cycle of rebuilding is too long, well you have to take the tradeoff, but it's a rare problem.
Understand that the only way a "dev server" style backend build process is saving you time is by not checking the types during recompilation. It wouldn't be surprising that it would miss typing changes.
So all the dependencies would need typescript?
The article is not very concrete in what was the actual bug
I'm not sure where that quote comes from, but these days in terms of TS vs JS it's 100% "not using untyped JavaScript for any serious product".
A sane working environment would file a priority 0 bug and address it during working hours.
I'm not throwing a rock but oftentimes this sort of heroic behavior is self-inflected.
Also why is heroic debugging required on something that is not production? The article mentions it was a "quick demo". Then who gives a shit? There's time to investigate. And this would/should NEVER happen to production. The whole point of testing and staging/UAT environment is to catch this sort of problems. Again, a normal day at work in software engineering and not something to kill yourself living on amphetamine like a commando behind enemy lines in WW2.
Sure, I would also do long programming sessions on my own things every now and then, but those happen because I am in the flow and it feels like more work to come back into it the other day. But those are my own projects. I don't owe this to my employer. If they want that they could offer me double my current salary and even then I would insist on only doing it when it makes sense to me.
But I also live in Europe and my superiors want me to come in with a fresh mind, instead of burning myself for things that should be normal work.
It's really strange that they would invest all this time as if they are keen to understand the cause of why it happened, but then be satisfied when jiggling the handle solved it. The real cause ultimately is a process one (developer discipline) or a pipeline one (bad build/test/deployment processes). They got close to realizing it and then blamed the language instead of what appears to be bad practices happening on their dev team.
While I agree with everything you have said, and it's been my motto since I read, "I'm sure you're joking, Mr. Feynman." The point of the post was to point out (maybe I did it wrongly) that those things you have described are pretty impossible given the ecosystem we have nowadays in Javascript (as a whole!). Of course, you can try to theorize and try to get a hypothesis on things and then try to prove those, but it's just too hard to keep track of everything.
In addition, we are using lock files and good practices but it's just too much to handle for our small team (small startup) so we will plan to migrate onto a more stable platform.
It's really funny the heated arguments this post generated but I have been using Javascript for 12 years to these days, still this looks as a nightmare to me.
Tsdoing has a good video trying to install React from scratch, that's exactly how I felt during this debugging process
Another random story from this week: we have been using `nanoid` to generate IDs internally for an internal tool. Apparently, they did a breaking change release, and now you cannot use it anymore on commonJS env. See my point? where's the hypothesis we can make? It's hours and hours spent tinkering with code and other people's dependencies, and if, as you said, put a theory that theory won't match if you think how real systems work (the networking section on the blog post)
Technically though, I have hardly met this dependency update problem with C++ simply because dependencies are rarely if ever updated. Android on the other hand was a nightmare. I had colleagues quitting in a rage and swearing to never again touch Android in their life. Probably half the effort was spent fixing crap that was broken by updated dependencies. Luckily I'm back to C++ now and hope never again will I have to touch such a crazy system. The whole 'update for the sake of update' seems to be a grave problem of the Android / Kotlin / Javascript universe. The mentality is like 'bread won't be sliced anymore, eggs won't hatch, sun won't rise if we don't absolutely have the latest bleeding edge alpha version of all libraries'.
That's the thing, nothing you have asserted as impossible actually is as you'll see throughout this thread. I have been writing JavaScript/Typescript professionally since for about the same amount of time and I am not intimidated or challenged by my projects in the way you lament about.
That being said, if you've been at it 12 years and this is still your opinion, fair enough, try something else then.
If you are not intimidated by the complexities of this Babel tower, it's up to you. For me, this was just a personal catharsis in my personal blog that I wrote mostly for me (and I didn't share it here)
I believe there's no point on discussing this with particularly you since you have been commenting all around trying to just confirm your view instead of just comprehend that this is not sustainable for new developers (try to follow the conversations on packaging, lol)
It's simply too much burden just to maintain someone else mess. We will pay a high bill in the future for all of this.
I don't need to confirm my view at all. Advocating for one's view doesn't imply a lack of conviction, rather the opposite.
No need for a personal attack here. It's definitely sustainable for new developers who learn the stack. It has a learning curve but so does everything else. I've built applications in just about every stack there is (except for the more esoteric stuff like OCaml, Erlang, etc) and over those years complaints like those you raised are easily levied against any language or stack that is practically used in the industry. The common thread is that these sorts of complaints arise when there are more unknowns for a developer than knowns-- and as developers we are in the business of eliminating unknowns. When you encounter unknowns and your response is "this is why this tool isn't fit for use" instead of "let me turn unknowns into knowns", this is often the result.
> (try to follow the conversations on packaging, lol)
There isn't a packaging system out there without complications, but my posts regarding NPM's behavior points to how it actually already works to reduce or eliminate the version pinning failures you appear to have hit. Now if you are using very old versions of the tooling or if you do things like remove or ignore the version pinning tooling and how it should work, then yes the kind of problems you hit can happen. But that does not mean this problem isn't already solved, or at least the problem becomes an education one; which is why I'm out here trying to solve the education problem.
> It's simply too much burden just to maintain someone else mess. We will pay a high bill in the future for all of this.
That's, like, your opinion man. I've yet to find a development stack without it's costs, and once you are fluent in how to work in the ecosystem, Javascript's are not significantly higher than any of the others. And personally the cost/benefit of the Typescript ecosystem for me is quite rosy compared to the limitations of other stacks. I'm clearly not alone in that calculus.
This is true in many other languages, not just Javascript. This isn't a "Javascript nightmare", this is a dependency nightmare.
I use Kotlin, both on the JVM and in the browser (via kotlin-js). Dependencies are equally complex as in javascript but it's much easier to find out if stuff breaks or not. I update often to minimize the amount of changes I have to deal with. This may sound counter intuitive but integration testing small deltas is exponentially easier and less risky than big deltas.
My attitude to having big updates lurking is that they represent technical debt. Just because you don't know you have a problem because you haven't updated yet doesn't mean you don't have a problem. By not updating, you just increase your ignorance of technical debt. It might be piling up without you realizing. And you might be making things worse by working against obsolete APIs.
Upgrading libraries often means I typically only have to deal with a handful of minor updates for libraries that are mostly unproblematic. And if stuff breaks, it's easier to pin point the source of the breakage. And I find out as early as possible about problems I need to deal with. If I don't know about it, I can't deal with it either. Sometimes downgrading a library temporarily is needed. But then you document why and stay on top of issue trackers and upstream development to see when you can upgrade or you start looking for alternatives.
Occasionally, some API breaking changes happen and then I fix it (because the compiler tells me stuff is not compiling). If I don't update for a year, I probably have a lot of breakage on my hands and a lot of work fixing things. Much harder and much more time consuming. And risky. All I know then is that I probably have a lot of problems (i.e. technical debt) on my hands that I don't know about.
That being said, there isn't too much about Kotlin I miss when I write Typescript in strict mode.
> Unlike typescript, it does not really have a non strict mode (except for the dynamic keyword so you can deal with some js libraries and its weird type system).
Why is the existence of non-strict mode a concern for you if you are using strict mode? After all, whether you are using Typescript or kotlin-js, you are likely to have JS dependencies, and those might not be written with Typescript strict mode. I guess you could just opt not to use any "native" (Javascript) dependencies and just do it all in Kotlin JS, but beyond a certain point that's not very practical.
I do indeed do a lot in Kotlin-js rather than using javascript libraries. We make a few exceptions for libraries like map-libre for which we have a very thin hand crafted Kotlin facade. Things like react we simply don't use at all (we use a pure kotlin framework called fritz2). We actually use a lot of Kotlin multi platform libraries as well. And even some wasm libraries (e.g. sqlite). Most of the foreign code dependencies in our codebase are not javascript libaries.
Maybe you've never heard of "writing tests"? If you update a dependency, and the dependency has a breaking change, your tests will fail. It's really that simple, and this should be the same in any language, not just Javascript.
That said, the main sense I get is that the team aren't so experienced with Javascript, and therefore working with it is full of surprises because they just aren't used to it. I found the line about bugs in C code being "your fault" particularly amusing: having occasionally had to touch C and C++ code, I would have said the complete opposite! Javascript, for me, is very predictable - if something goes wrong, then it's usually because I made a clear mistake. But with C, I end up trying to debug chains of Makefile and CMake problems that have no rhyme or reason to them.
In other words: as someone inexperienced in C, it is a chaotic, magical language and ecosystem to me, in exactly the same way that the author describes Javascript being to them.
More practically, it sounds like the team need some regression tests for this case that they can run next time they try and upgrade their dependencies. They also need to make sure that caching is working properly for their Javascript code - rerunning the CI multiple times on the same commit should always produce exactly the same dependency tree every time. Probably that means caching the right directory and using NPM CI to clear and rebuild node_modules from scratch every time.
More generally, the author complains about a lack of type checks. In this case, it sounds like the missing method was somewhere in a dependency, so this may not have helped, but: Typescript! It exists, it adds types, it makes a lot of things easier. Easy mode Typescript is to just start writing `.ts` files and use the `tsx` package to automatically compile and execute everything - there are other approaches that they could explore, but this is probably the simplest.
And if you want reproducible dependencies, use npm ci not npm install, or use yarn. Best combined with replicated self hosted npm repository.
To be clear, I agree that the Javascript ecosystem isn't perfect, particularly when it comes to supply chain issues. But in a lot of ways it's more secure than Python's typical dependency setup. By default, NPM uses a lock file, which enforces that specific versions of existing packages are used unless explicitly updated. This means that you always know which versions of which dependencies you're currently using, it allows you to upgrade different parts of your dependency tree independently, it ensures that when you remove a dependency, all of its transitive dependencies also get removed, and it allows for different dependencies for production and development environments, and for different portions of a given project.
Doing all - or even part - of that in Python is difficult. There's no built-in way, and so people tend to adopt third-party tools or roll their own scripts. The roll-your-own process, in my experience, almost inevitably leads to some sort of failure, usually a certain updating unexpectedly and breaking everything, and then not being able to roll back to a previous version of that dependency. The third party tools (Poetry, Pip-Tools, etc) work better, but are often difficult to integrate more widely into the Python ecosystem.
I don't want to make this about Python vs Javascript - both are tools that make different tradeoffs and can be valuable in different ways. The point here is that, as software developers, we often end up with what are essentially superstitions about other programming languages - we describe things we don't know or don't understand as "magical" or "scary", because that's easier than admitting we're just not that well informed. And while it's okay not to be informed about everything, in situations like this, it can lead to bad outcomes.
In the article, the author ends up blaming the situation on the magic and weirdness of Javascript, but it sounds like the issue had more to do with them not understanding how best to do package management in Javascript.
> The next message I sent: “We found it!”
May be I misunderstand something, but from the post it sounds that they don't even commit their package-lock.json. They're using node like its 2012.
Dependency silent upgrades and dependency hell and bloat are not unique to javascript, but it feels like the js community has a perverse fetish for fragile build systems and dependencies.
Ofcourse, the community is not one organised entity and this is merely the result of multiple individual decisions taken independently, but still. I can not help but empathisewith the author despite seeing that they aren't locking deps, not doing reproducible and rollback-capable deployments via docker or artifact tarballs, don't seem to have robust ci, or good work life balance..
Is it elitist to remark "haha, don't you know you should smerp the blergh?" Is it not atleast somewhat crazy that we normalise rube-goldberg-esque contraptions?
I don't even know what I'm getting at, at this point honestly. Js gives me very mixed feelings. I love the accessibility amd universally available execution runtime (browser), and understand why it hassome of the warts it does, but it is also honestly a scary ecosystem with hyper fragile tooling. I've personally burnt multiple full days on webpack config tweaks and it all feels so unnecessary and pointless.
JS almost gives me existential crisis at times.
Packages built by 3rd parties are the Wild West, and unless they’re a full-on company of dedicated, experienced engineers, they’re gonna break semver best practices. They’ll also design bad APIs for interacting with their packages sometimes, and there will be a total lack of consistency across the ecosystem, which is what we have currently.
If I had more free time, building a stdlib for JS would be high on the list of “help the world” activities.
And then to prevent this, you just lock the dependencies via a lockfile -- a well documented solution.
This is just another amateur "javascript bad" post.
Unless of course your handler results in dying for science, then it's fine (ie for sending to an observability service first)
> @smithy/is-array-buffer@2.1.1
Are these direct deps? Because just a cursory look raises all kinds of red flags. There's a disclaimer for the former warning about it being in developer preview, and the rest of the README has a bunch of notes about edge cases.
I get that the aws-sdk uses them, but if it works for other people it should be fine. Having these as direct deps and also calling in complex libs like the aws-sdk honestly seems like a recipe for disaster.
It sounds to me they rolled in the dependency update with their features, then had a massive surface area to cover when it flaked out.
- Use Typescript.
- Practice version locking on your dependencies and make sure it works correctly throughout your processes
- Do not commit vendored dependencies
- Do not commit compiled JavaScript
- Use a CI process for deployment which runs your tests
- Have tests
- Prefer (or require) types for your dependencies, or dependencies written in Typescript
- Prefer dependencies that ship the types in package (and not via DefinitelyTyped)
Some additional thoughts, they act like running out of memory is something they need to actively handle in the codebase-- it's not in this case. It shows a lack of familiarity with how JavaScript runtimes work which hints at a team which is very inexperienced with the platform. JavaScript runtimes enter GC lockup and ultimately crash if there is no memory, so this is a logging problem, not an error handling problem. There is no mechanism for the runtime to inform the app that an allocation failed.