This is off by default due to security concerns but can be turned on with a flag. It's mainly in the community feedback phase so if there is anything you want to tell Node (be it "I want this!" or "This is a bad idea" - please do)
Longtime node.js developer... I don't think we need this, and I don't think it's a good idea to add.
The feature is _strange._ Which we know because it's behind a flag, i.e., it will have low utilization. It defies conventions—because the convention is to use a registry for packages. And it presents significant security concerns. (Which I don't really have to explain, as they're already cited as the reason to put it behind a flag.)
Use of this feature seems like something you might see once in a career. (Indeed, I've seen a similar thing in another language employed only once... AND THERE WERE OTHER WAYS for that developer to have designed that system that would not have been so vulnerable.) When you see it, do you recognize what it is, understand and evaluate the risks it poses? Do you validate that the library sitting on the other side hasn't been taken over by a malicious actor? Do you validate that the library on the other side has up-to-date dependencies? If so, on what cadence do you do this, realistically?
All the security tooling progress we've made (npm audit, etc.) seems shot in the foot by this. And I get that these concerns aren't dispositive on their own. I just don't see the argument on the other side that clearly. Is there some class of application that people really want to write in node that can't be written for lacking this? Do we really think this is a pattern that would be employed in well-made, secure codebases? I don't see the argument. Like I said, I've seen a feature like this used in another language. But I haven't seen it needed for any real application.
I love me some good node ugly hacks. Still, the language features should be nudging us to write good software.
Central authority != Good software. You are essentially advocating to provide a single point of failure that can and will be abused by nation-state hackers who just have to find one popular package and compromise the developer.
The argument is less about central authority than standard process. You don't have to use the official NPM registry. The point is that having a pipeline for modules with established conventions and automated means of audits and visibility on the dependency is a net good. Yes, there is such a thing as a supply chain attack. But even when that happens, the community puts out the call to patch immediately and points a finger at the offending package. Your HTTPS import from a random URL could fail silently. While NPM is a big target for supply chain attack, they know they are, so there are a lot of eyes on it. Not so with your HTTPS import.
I recall when I was a completely PHP n00b (babbys first programming language), I might encounter an issue and search for the error on expertsexchange. Some of the answers would be really complicated, but there'd be that one answer: Just edit your php.ini and change `dangerous_option_you_will_die=true` - wow, such an easy fix.
I'd know better later, but I'd still find horrendous options like these enabled in real software I would come accross in my professional career.
I have been doing nodejs backend developpement for the past 10 years. I'm not sure why we would need that honestly. Looks like it could create some mess if packages on npm all start importing files from random servers. Suddenly your CI relies on many servers to be up. And I don't know what's the problem with declaring things in package.json.
All of the things that make this enticing as a feature are large security risks, and the less risky version that people are pitching me doesn't have any of the enticing features.
What people are pitching me on is the idea that these wouldn't be dynamic imports. They're saying they wouldn't be fetched during runtime, they'd be fetched before. They're saying they would be pinned.
Well, you're describing a package system like npm. And for those purposes, as a way of defining static packages that get installed before runtime and can't be dynamically swapped out by the application -- npm is better than the system being proposed.
The only way this would actually be interesting is if they were actually dynamic imports, but having them be dynamic imports would be very insecure. And for declarative imports that can be statically analyzed, this is inferior syntax to what we already have and encourages worse developer habits.
Some problems:
1. Updating dependencies is harder, because imports across multiple files have to be synced. If a junior developer imports `https://lodash.com/v4` in two files and later someone changes that to be `http://lodash.com/v5` in one file, now I have dependency mismatches. The existing system doesn't have this problem because my dependencies are in one place.
2. Finding out which dependencies I'm using is harder. Maybe there's another tool that lists out all of the imports, that would be handy. But also, maybe the dependencies are all in one file with the requested version numbers right next to them. That seems a lot more convenient.
3. This forces me to actually run the Node program to install the dependencies, and it forces me to actually import them in a file before they'll be fetched. This is cumbersome, it would be good to be able to list out dependencies before they ever get imported, say for boilerplate imports that are used in lots of projects.
4. People have pointed out the lack of pinning.
All of this is inferior to having all of the imports in one file that you can specify, maintain, and update without running the program or using an analyzer. I don't understand what the benefit is or why I should care about an import system that is just as inflexible as the current npm/Yarn package manager setup, but that also has what seems to be worse syntax and makes it harder for me to know what packages are being imported.
I wish I understood more what problem this proposed syntax was actually trying to solve. I've read through the entire thread and I still don't understand why people want this. People in the Github thread are reassuring readers that this could work exactly the same way as node_modules. That's great, but if that's the case, what is the problem with node_modules that is prompting us to build a second system that is less organized and less explicit about what gets imported?
We can already import URLs with both npm and Yarn. That's not a new feature, the only new feature is that now the URLs are spread across the entire codebase.
If anything, this shows how ahead Deno is and how features like this in Node.js are too little too late.
Deno's standard library is just better than Node's, IMO, and it's going to be hard for Node to play catchup, if that's even practical. In summary, it doesn't contain the cruft of the pre-ES2015 era that is still present in Node, even though Node has tried working around it. The global objects in Deno better replicate the browser, so it takes a lot less effort to do essentially the same things you would in a browser but in a DOMless environment. Deno is slightly more performant in Node in some areas, although this is usually negligible, and I don't have the JSPerf test on hand.
I don't think Node is bad, by any means. If people are concerned about security... then don't use so many dependencies, or consider just not updating them until they've been demonstrated not to contain exploits; you don't actually need much code to essentially serve webpages.
But I think Deno, over time, will at least match Node in popularity, maybe even more, because it has "modern" approaches right out of the box and doesn't need to move as slowly as Node in order to maintain a legacy. Although Deno misses some things here and there, I've seen features get added very quickly over the last couple years; at this point, it would take Node much longer to do the same things for fear of breaking something and because it's simply much older code.
Then again, maybe I'll be wrong. Perhaps Node gets a second "big bang" and makes Deno obsolete. And I wouldn't care, either. All I want is a JavaScript language runtime with fewer concepts to learn that can also move quickly when something needs to be improved. If that ends up being Node, I'd be just as glad.
Definitely not, deno does typescript out of the box without pulling in a bunch of npm dependencies and tools. It also is taking more of a go or python/batteries included view of building out a proper standard library vs. relying on the community to provide it. And of course deno has a much stricter execution sandbox model where you have to opt in to explicit permissions like file system access. IMHO we'll probably see node get more and more deno-like over time.
If node shares import syntax with browsers, it’s easier to write code that runs on both platforms. As far as leftpad, this isn’t that much different from using a package manager. You either cache your dependencies, or rely on a third party (npm repo/cdn).
When leftpad was in NPM there was an expectation that it wouldn't disappear suddenly. Package registries are thought to be a constant, reliable source of packages (even when they not quite that, as leftpad demonstrated). There would be no such expectation for a module that you include directly from an https URL.
The main benefit I can see is that you can include code from your own "registry" very easily in unmanaged scripts. You can already do that because you can add a dependency to package.json using an https link already, but with this new feature you can include those dependencies in node scripts you aren't managing with a package file.
You can do that. I'm not suggesting you should. I won't.
On one hand do what you want, on the other hand you can already add https imports via package.json.
Why in God's name would you not have a package.json.
I think this would be awesome for hacking out prototypes, but if I saw this in production code I'd ether work to get it fixed or find a new job. Being reckless is fine in your spare time. It's not ok at work.
My fear is this is going to push junior devs into sloppy practices.
I have the distinct impression that whether you put the imports in package.json or in the source file itself, ~100% of the industry is engaging in sloppy practices now. I am terrified when trying to write secure es6.
AIUI, when you add a module in go, the specific and exact version of the module is cryptographically locked in your project metadata. npm/yarn store the hashes in the lockfile but the package.json still uses those (usually) fuzzy matchers, and I'm not sure whether "npm i"/"yarn install" uses the lockfile exclusively by default. "npm help install" suggests that it uses package.json (which does not cryptographically secure the deps so that they match what you originally added) by default.
(Additionally, tons of people install and deploy docker images by tag name alone, which is not cryptographically secure - different environment, same problem.)
My mixed feelings about it are that we are slowly blurring the line between local and remote software, and pretty soon it will be nearly impossible to use most modern software in an offline environment (which is a hard requirement for certain security circumstances).
Basically every single js developer in the world has granted Microsoft (owners of GitHub and NPM domains) the ability to remotely execute code on their machine at will. It's a lot of tooling and infrastructure to update and change in the event that the threat landscape changes significantly (e.g. war, national security mandates, etc).
Manually auditing an update of potentially hundreds of packages when package-lock.json gets regenerated is so challenging that many projects won't do it — and for those projects, the cryptographic hashes offer limited value.
To make industry-wide improvements, we need to share resources for auditing of packages. There are many such endeavors underway at various stages of maturity.
But this feature doesn't move that ball forwards. From the comments:
> It is insecure because there is no signature of the original file. Anybody that can take control of a domain name can take control of a server.
Downloaded arbitrary code can't be added to a pool of pre-validated packages, except insofar as you control the server and only make pre-validated packages available on it exclusively to your own services.
NPM, as far I can tell was not designed to have you run npm install every single time you deploy. People decided to do it anyway, I still remember the day NPM's SSL certs went down ( 2013 or 2014). It's like the internet broke overnight.
It's a real catch 22. When you look for a magic npm package to save some time, you have no idea what's really in it unless you inspect all of it's code, and it's entire dependency tree.
I guess you could build an solution to check for any vulnerabilities, but then you just move the problem elsewhere. Said ML tool would never be perfect
Its also possible that this enables you to deploy all your assets prebuilt and optimized to a CDN?
For Node apps running on services like fly.io this may be useful, since they're regionally distributed and their claim of running closest to the customers using it is at least partially true. This makes your deployed images (and therefore disk size needed) much smaller
I agree that not having SRI as a requirement is a miss, however I see legitimate use cases here.
As for why, it's simpler than adding a package.json and meticulously curating it. If you only have a few small dependencies this might make sense. You can also pull in stuff that isn't hosted on npm or properly packaged--for example just pull down a js file from someone's github repo.
If you start getting a lot of dependencies then it might turn into a new nightmare to maintain. It's an experiment--people don't really know if it's good or bad until using it a bit.
Sounds like it. These days, HTML is trusting JavaScript that's loaded from all over the place already. A "secure" database hosting service loads assets from 20 different domains on their login page.
Loading code from remote domains can be abused and mis-used in any language. In this case, I think it's considered a "feature" because code was already being loaded remotely with NPM, who in turn got the code from the original author.
Now you can load the code directly from the author. So instead of trusting both the author and NPM, you only need to trust the author. This could be considered a reduction in your attack surface area, and it's definitely helping with the NPM registry being a single-point-of-failure in the Node.js ecosystem.
I don’t know where you got that idea. NPM doesn’t have any say in CJS (or ESM) module resolution other than that they conform to relevant aspects of it. Node and other implementers of those module systems absolutely do allow dynamic require/import.
PHP got ridiculed for years for doing all the things all the other popular frameworks are coming around to support now. Server side rendering is particularly hilarious because it's like: oh, you're compiling .jsx files to .html now? Cool....can't imagine anyone ever did that before...
Is it though? I remember injecting <php> tags in the middle of JS scripts to fill out variable values. I'm sure someone somewhere was using PHP to inject conditionally dependent JavaScript onto their page. Is that much different from Next rendering a React component on the backend, injecting it into HTML, and then injecting its corresponding component state into the JS bundle?
...Yes? For one thing, PHP code runs inside the process of rendering HTML, but JS SSR runs under the process of a React JS app, running in Node. That alone is an absolutely massive difference. Not just on a technical level; React JSX also doesn't have nearly as much hazards surrounding HTML sanitization, for example, because JSX is a DSL that protects inputs with proper sanitization by default, among other practical differences. Another difference is that SSR is more flexible than for runtime environment; it is very often used as a static compilation technique, where it is run as a compilation step for a static site, but either way, it is not married to any specific runtime model, whereas PHP pretty much is (for now.)
A closer analogue to what React does would be comparing XHP[1] to JSX, but even that misses the isomorphic aspect of a SPA with SSR, which is almost the entire value proposition, because it allows you to only need to write your component code once while reusing the same render cycle at browser runtime.
It should be pretty evident that PHP-style CGI scripting is not ideal given that even early on many PHP apps moved to using template systems like Smarty and never actually embedding HTML directly into PHP code.
The reason SSR/SSG of JSX (or Svelte, Vue, Marko, whatever) is appealing isn’t simply that you can run it on server/build. The appeal is that you can use the same code client-side. And of course each such framework/library has its own value propositions that are useful in both contexts.
An additional of JSX in particular is that it’s render-target agnostic. Yeah you can render JS/DOM, and HTML… and CLI, and native UI APIs, and smart TVs, and WebGL, and…
(I know there are similar solutions for some of these in the others, but JSX was specifically designed to accommodate this.)
My impression is this is really just to have some sort of comparable feature to deno. You may think this somehow is more of a vulnerability, but the reality is the npm registry is already fairly insecure, we've seen npm packages get hijacked regularly.
This also adds less dependence on npm, and maybe with all the packages that get hijacked there, this could be a reasonable move.
I'd really argue "regularly" is a bit much. It happens rarely and moving to URIs is only shifting the problem.
I'd also argue the claim npm is insecure. How would you improve its security over what's there now?
Lastly, there isn't really a dependence on npm at all. You were always able to 1) put URIs into package.json, and 2) make your own package manager. Node does not require npm to work. It just looks in node_modules by default, which npm uses to manage packages for you. They are completely decoupled.
URI imports, on the other hand, have a lot of security problems and surface area and will probably face a load of criticism going forward.
Controlled builds based on public repos prevents a malicious person (original author or not) from invisibly pushing a packages that doesn’t correspond to public sources.
Signing packages prevents account takeovers from publishing bad packages.
The remaining security problem is the author themselves coupled with too few eyes on public repos.
Yeah, a bit of hyberbole, but I think my point is it's been proven to be relatively simple, or at least simple in theory.
I think npm could improve security with two steps: 1) all packages must be user-or-organization/package-name, and 2) all packages need to be signed with a key. Have namespaced-by-default packages at least will give a path for developers to trust an author or source. Signed packages would give developers the means to trust a key. All systems that allow public packages to be published are going to have inherit security flaws, and the best security I think we can give is trust. It's easy to invalidate a key, and it's easy to flag a namespace as untrustworthy.
While there is no "dependence" on npm, most package managers ship node with npm, and the typical flow for a node app, according to most tutorials and community documentation, is to use npm. Even most of the tools a developer would use with node are npm-centric (documentation almost never recommends cloning the repository, just npm install). So I'd agree, decoupled in reality, but is perceived as reality among tutorials and the community.
What about TypeScript here? I think it does not yet understand such imports. However, especially for types (i.e., evaluation at compile-time) imports from a remote source would be good.
Instead of keeping up-to-date via npm I could just write `import type { FooType } from 'https://remote-types-registry.com/foopackage@2'` and wouldn't have to worry about typing updates. (surely this scenario is only valid when types are not shipped with the dependency)
In my limited experience with Deno, it seems direct URL import just leads to a crappy, nonstandard, hard-to-upgrade version of a package.json in the form of, say, a deps.ts, once you get past a few dependencies. (And it’s very easy to get past a few dependencies in Deno since std modules need to be imported as URLs like everything else.)
That is my experience as well. I was really excited about this feature in Deno when I first heard about it. Then I started to get in to Deno, and it just seemed like it made everything harder. I thought it was going to be something like "here is a better way to do dependency management". But in practice it felt like "we got rid of dependency management. Figure it out on your own".
I feel the same and import-maps do not solve the issue since they are not transitively loaded. In other word, if one of your dependency uses an import-map, then you have to copy-paste it in your own project to get its import-mapping.
The people in the Github thread advocating for this feature seem to be pushing really hard on the point that this isn't any more insecure than NPM already is. But that's myopic. That's like saying that the lock is broken on the second floor window of your house, so you may as well unlock all the other windows. If a junior developer working for me proposed something like this I would pull them away from the computer and have a long coffee chat with them about basic architecture principles.
Yes, we know there are problems with NPM. But why broaden the possible attack surface area by introducing another kind-of-sort-of-equivalent means of remote module loading? Now you're just going to have to spend time hardening two systems instead of one.
At the end of the day this just feels like a misguided attempt to try and recapture some sense of dynamism from Deno. But if I wanted dynamism, I'd use Deno, with all of its pubescent awkwardness. I'm using Node.js stable for a reason. I spend a lot of cycles trying to defend Node from its many (often reasonable) detractors, and things like this make me feel like I'm wasting my breath...
Lul what? First of all, this tone is completely against the HN guidelines, so take your flag.
Second of all, centralization is not often a part of a security model. Usually in such cases where availability is a concern you have a mirror set up, which is ridiculously easy to do with CouchDB and completely nullifies your arrogant point.
My comment wasn't really about the merits of central authorities vs multiple authorities (although I would fall on the side of preferring a single package repository than having to worry about many of them). It was about having two equivalent mechanisms for solving the remote-code-loading problem, regardless of where you're actually loading it from.
Hi, I'm one of those people in the PR advocating for this change :) I think you may have missed the point of comparing the security aspects to NPM. By saying "it's no less secure" it's not saying "well it's just as bad, so what's the point?", but rather "given that it doesn't introduce any more vulnerabilities, what does this model unlock for us?"
HN comments probably aren't the best place to go in-depth on this topic, but just off the top of my head, HTTPS imports allow a greater detachment from a centralized registry, something we're seeing used a lot in the Go ecosystem, and we're moving server-side JavaScript closer to how browser JavaScript works, which is a good thing for anyone using the language, as it reduces friction across the board. This might not be something obvious for people working mostly/only on the backend, in which case those frictions aren't as apparent, but they are there nonetheless.
> HTTPS imports allow a greater detachment from a centralized registry,
I know that the people advocating for this are smart, I know you're being thoughtful about this, so forgive the simplistic question, but every package manager I'm aware of for NodeJS already supports this.
Can't people already import npm dependencies from arbitrary URLs without signing or attaching anything to the npm repository? And the other thought that jumps into my mind is, if people aren't doing that with npm that even though it's well-supported, that might be a signal that there are problems with the model of using arbitrary URLs for dependencies that make it less attractive overall to developers -- and I'm not sure how ES Imports using URLs would change or fix any of that.
> given that it doesn't introduce any more vulnerabilities
I think this is where we are in disagreement. You're adding code to Node core in order to support this, which opens the door to new vulnerabilities (bugs) by definition, because people make mistakes. Now you will have two code systems to scrutinize and maintain that are doing essentially the same thing. I don't think the arguably-slightly-better ergonomics of HTTPS imports justifies the increased risk.
It would be vastly superior. I'd go so far as to require it without an additional --unsafe-import type flag.
However, I think they did not do this because the import of dependencies then comes into question. IMO this feature should not have been added until developers could guarantee the expected integrity hash matches.
We've seen time and again what will happen now is hundreds of blog posts talking about this new feature, none of which will mention the integrity hashing functionality (because it's not there). When it does eventually get added, those posts will be the most popular anyone can find, and thus new users will end up using those eventually-wrong posts and implementing things in a less-secure way.
It's rubygems all over again. I think only Nix and Go are doing dependencies right (cryptographically verified) these days.
One upside is that it's https-only, which does include authentication, provided that you trust the holder of the certificate for my_local_repo.tld (in your example) to serve you the right files.
Exactly, this kind if import without a hash validation is a big no for security reasons (unless you 100% trust your import source). This feature exists on the browser side with the script element: https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
Came here to say the same thing. In fairness, this isn't specifically Node's fault. I feel that a mistake was made early in the SPA/mobile craze a decade ago, where the web should have used subresource integrity (SRI) everywhere:
These operate at different levels of abstraction. SRI is mathematically strong, whereas CORS requires a bunch of hoops and mostly just gets in the way. I could understand an argument for SRI+CORS, but using CORS alone sets off alarm bells in my head.
CORS and SRI are just completely different. CORS exists to allow requests that would be banned under the Same Origin Policy, most of which have dynamic responses and could not integrate with SRI at all.
I agree - I'm not really sure what the parent poster is on about. CORS and SRI are orthogonal, and designed to solve entirely different problems.
CORS is primarily focused on removing csrf (cross site request forgery) vulnerabilities. It doesn't really care at all about what resources you pass around (or their integrity) it only cares about ensuring that the request is coming from a known and allowed origin.
This just isn't true. I don't consider request forgery to just be side-effect based (the classic example of a bank form post). I'm including any request that leaks data to a site based on cookies present only on the client (forged authenticated request).
SOP/CORS was the foundation that allowed CSRF tokens to even work. (No SOP/CORS? Just request the form with the csrf token first, parse with js, append to next request)
I guess you could say SOP is really the protecting force here, but CORS only exists because strict SOP broke a LOT of stuff. Enter trust zones in IE, and then CORS.
SameSite now mostly tackles this by fixing the root issue with the cookie jar.
cors never protected against csrf because you could just have a form that gets submitted via js. that worked for ALL post requests and it works in modern browsers as well and most csrf attackes used forms.
Just for anyone curious, I know that SRI and CORS are different. SRI negates the need for CORS in some situations because we're saying that we only care about the file content, not where it came from.
For the web, we could have been sourcing all of our javascript files from a central CDN and used the hashes to prove that they weren't tampered with this whole time. Instead, we went down the rabbit hole of build systems and endless local hosting.
For this Node.js use case, they could have allowed remote includes, with the understanding that it's only secure when the hash is specified. Verifying only the location is like CORS, it's security theater based on trust instead of a mathematical level of confidence about the contents.
Good point about dynamic responses not being compatible with SRI though. I hadn't thought about that use case, but you might be onto something about its lack of adoption. It may be a red herring however, because to make SRI dynamic, we'd need some kind of trust process that tells us that a new version's hash is safe, more like a BitTorrent tracker, but the file would still be requested through ordinary SRI. Also it doesn't apply to what we're talking about here, because package versioning means that the files are always static anyway.
But what would be the point? Why would you load code over HTTPS if you're going to make it static anyway? If the code can't change dynamically, then you might as well put it next to the JS file on the filesystem.
I think this is a terrible idea that can only end in misery but I can understand why they don't add integrity validation by default.
It’s a feature that Deno has and that works on the web (ESM imports from URLs). People have use cases for it and they want it. That’s not to say it’s secure by default of course but the demand is there.
HTTPS doesn't magically protect you from the content of the package changing.
They are different security mechanisms designed to defend against different threats.
One answers the question: Was this the correct server I got this package from?
The other answers the question: Does the package's hash match what I expect in my code?
As to why you would want this, I've worked on several projects that take advantage of some things that pulling dependencies only from a local repository can give you. Take the recent Log4J vuln as an example. Immediately on seeing that vulnerability, I disable the JAR file, causing builds to fail wherever that dependency exists until the updated one is added as a dependency and used. From experience having to do dependency analysis on an old toothy codebase, it's sometimes simpler and more accurate to just let everything break and fix the breaking spots.
I understand that HTTPS doesn't protect you, and that's the reason why you'd normally add a hash.
If your goal is to make the build fail for vulnerable applications, you can run a local repository with upstream proxying functionality like Nexus Repository and kill the package there. This won't let you kill production (but why would you want to kill it this way?) but it'll give you the same benefit without risking remote code execution.
If you understand HTTPS doesn't protect you, I'm unclear on your initial point. Any chance you can elaborate on precisely what you meant when you said:
But what would be the point? Why would you load code over HTTPS if you're going to make it static anyway? If the code can't change dynamically, then you might as well put it next to the JS file on the filesystem.
I think this is a terrible idea that can only end in misery but I can understand why they don't add integrity validation by default.
I provided an example of why you would want to still load it remotely (specific situations where it's beneficial) rather than simply copying the code into the source repo, which you have also echoed, though seemingly suggesting that the example means you can do away with integrity checks?
If my read on that is correct, then point then is to worry about the repository - wherever it lives - being poisoned, but still wanting some of the benefits of having a remote repository (such as centralized dependency management and control).
> I provided an example of why you would want to still load it remotely (specific situations where it's beneficial) rather than simply copying the code into the source repo
Doesn't Node cache imports, though? I think this approach would protect you if you were running a script on a timer (thus triggering a fresh load at a regular interval) but not if you were running an Express server (which would only verify the hash of the remote library at application startup).
The code can update dynamically. You simply have to update the version number and verification hash on your end (same as you would in a package manager). This is still vastly easier than copy-pasting entire repositories into your codebase every time.
Right, but if you don't want the code to update unless you specifically update the verification hash and version number, why wouldn't you download and install it locally?
There's two separate questions:
A) is it ever good to have dependencies that can change underneath you? (arguably no)
B) if you don't want dependencies that change underneath you, why would you ever want to import directly from a URL at runtime?
----
> The code can update dynamically. You simply have to update the version number and verification hash on your end (same as you would in a package manager).
Code that I import through a package manager can update dynamically. I just need to update the version number and re-run `npm install`. Why would someone want to use a URL if the update process is the same as using a package manager?
I'm having a somewhat difficult time imagining a use-case for fetching dependencies at runtime that isn't related to not knowing until runtime what the dependencies are or what version is being fetched. Maybe for making installation smoother? But anything that requires compilation or is system-dependent isn't something you're going to want to re-fetch at runtime anyway. I suspect those dependencies will be slow to install.
And we're not dealing with a web situation where you might want to import from a CDN or another location that has better uptime. It sometimes makes sense not to serve assets yourself on the web just because it's expensive/difficult to serve them quickly. Node dependencies don't have that problem, pulling the dependency in as an install step will mean runtime performance is faster, not slower.
I get the argument that you shouldn't do dynamic imports that don't have resource integrity checks, but that seems like an argument against this feature in general -- you shouldn't really be doing runtime imports at all, because... why would you need them except for unpinned/unchecked dependencies?
The advantage of this approach is that you don't need a separate package management ecosystem, no central hosted registry, no submissions. A bit of code is hosted at a URL, and that's it. It uses the same ES standards as importing any another JS script from your filesystem.
Secondly, this doesn't mean all this code has to be dynamically loaded at runtime. You can still have equivalents of npm install which cache the linked resources locally on your server, perform integrity checks etc (this is exactly how Deno does it).
> The advantage of this approach is that you don't need a separate package management ecosystem, no central hosted registry, no submissions.
But npm has this already. You can already install from urls, you can even install from remote git urls. It's not even complicated, it's just `npm install git+ssh://user@hostname:git_repo.git#commit-ish`.
I get that npm isn't Node, but this feels like a lot of duplicated effort for something that is much more safely handled with the package manager that ships with every single default Node installation. It's not exclusive to npm either, Yarn supports this stuff too.
Are there really people building Node projects who are using a package manager that won't let them import from urls?
----
> Secondly, this doesn't mean all this code has to be dynamically loaded at runtime. You can still have equivalents of npm install which cache the linked resources locally on your server, perform integrity checks etc (this is exactly how Deno does it).
Unless Deno is crawling all of the source files in the project and doing this before the code is even run, aren't you still going to run into situations where you run your project and it hits an import it didn't see before, and it then gets dynamically imported at runtime?
On the web, ES imports are designed to be kind of inflexible and static so that you can't have those dynamic systems, it's all designed to make it harder to run into unexpected imports and easier to do static analysis of dependencies. And maybe Deno is the same, but if so then I don't see what the advantage is over doing that instead of putting all of your dependencies in a package.json and fetching them as part of an explicit step. The web made this stuff super static and inflexible to try and make dependencies more explicit, but an install step is kind of the end-goal of those restrictions, it's the most explicit way possible of importing dependencies.
And remember that npm lets you get your dependencies from anywhere, and they all get packaged in a predictable way, transparently on disk. You don't lose any flexibility in regards to where you get your packages from. At best, this system does the same thing, in which case why are we duplicating something that already exists? At worst, it caches resources only for future runs after it fetches them the first time, and it fetches them the first time at runtime, in which case I just don't understand why that would that be desirable.
Maybe conditional installs of packages so you can only fetch a subset of the potential dependencies you might need based on what is actually used during program usage? But this seems like a really error-prone, potentially dangerous way to achieve that.
> But npm has this already. You can already install from urls, you can even install from remote git urls
Yes, but npm can only install npm packages. This is a step towards reducing/removing the dependency on npm itself from the Node ecosystem. It is duplicated effort, sure, but that effort is spent in undoing extra complexity and getting to a much easier "hello, world" state for both app and library developers.
> Unless Deno is crawling all of the source files in the project and doing this before the code is even run
I don't understand what you mean by this. What is an npm package?
Node already manages how `node_modules` get imported from disk, and it handles the package.json format; that stuff is already part of Node. The only thing that npm does is fetch the packages/dependencies, stick them on disk, and then do integrity checking with package-lock.json. And all of that is already somewhat optional. Even when you're using npm, there's no certification you need to go through, you don't need to use a proprietary format, you don't need to get anyone's permission to distribute your package, you don't need to sign anything. You need to write a bit of JSON and then zip your source into a tarball, and then anyone can use any Node-compatible package manager to import it and its dependencies.
npm isn't even a requirement for fetching packages or pinning resources, you can use something like Yarn as well (and arguably should, because Yarn has much better dependency pinning). You can even hand-write your own compressed packages that work with npm.
> It is duplicated effort
It is duplicated effort of functionality in NodeJS. And it's duplicated effort of good functionality, Node's `node_modules` system is decently good design. To the extent that we are getting rid of being able to use 3rd-party tools to fetch packages, we shouldn't do that. npm/Yarn have already proven that having multiple ways to import packages is beneficial, and being able to dictate one program that's in charge of how dependencies are checked for resource integrity and where they get stored on the disk is also good. We only got package-lock.json in npm after Yarn implemented it.
But I guess that's arguable and maybe someone thinks that we shouldn't have a package manager at all. I disagree, but sure, I'll concede that it would be reasonable for someone to think that.
> That's exactly what it is doing.
So for whatever reason, Node wants to handle fetching resources itself. Fine, we'll say there's a good reason for that. Having said that, when we look at the proposed syntax, why would you want your dependency declarations spread throughout your code rather than having all of your dependencies defined in one place? This sounds like a strict downgrade in terms of auditability, code complexity, etc.
You're saying that we're reimplementing things so we can undo extra complexity, but how is parsing out an import tree from source files simpler than reading a list of imports out of a package file? If the goal is to make a simpler system, I kind of like having a system where I can audit dependencies by hand rather than looking through source files and making a list of all of the imports.
Not to mention that this system also isn't flexible enough to handle dev-dependencies or global dependencies, so we're going to end up adding a bunch of complexity and extra syntax for that I suppose if we really want to get rid of npm/Yarn.
----
To play devil's advocate, I'll grant that now you can kind of imitate ES Imports without changing syntax. If I remember correctly, that was part of Deno's reasoning for using URL imports in code.
But again, why? ES Imports are a compromise due to the structure of the web and the fact that fetching packages at runtime on a website matters. Why hobble a native system with an import strategy that realistically wouldn't have been chosen for the web if any other option had been available?
If I'm understanding you correctly, this has built an install step that works the exact same way as a normal install step for languages with normal package managers, except it doesn't have strong verification guarantees, and it's harder to audit by hand, and (assuming I'm understanding the docs[0] correctly) no longer goes through an explicit install step and requires you to actually boot up the app once before...
>But npm has this already. You can already install from urls
One use case is if you have a JS module that depends on another JS module, and you want to be able to run it without transpilation in both Node applications and in browser environments. NPM-installed modules wouldn't be available to the browser engine, regardless of how NPM installed them.
I’m also concerned about resource integrity, and that was my immediate concern initially when I heard of Deno. But it’s important to also recognize that it’s quite a bit more complex than a hash for the requested module. Because of course that hash has nothing to say about the integrity of transitive imports.
I’m willing to give Node the benefit of the doubt that this will remain experimental with appropriate warnings until resource integrity is addressed to some reasonable level of confidence. I’m also glad to see it being a first party concern, especially given that it’s historically been/still is a much higher risk factor as both:
- delegated to their ESM loader API, which is incredibly difficult to get right, despite the API itself being deceptively simple
- a hilariously insecure example in their docs for same
It's unfortunate I think that the docs don't mention integrity checks or caching, but those things are mentioned in the PR thread. Integrity via policies which seems to be a new feature as well[1], perhaps designed to essentially obsolete things like package-lock.json et al? Caching is punted till later, presumably to not hold up progress on this particular feature.
Seems to me these are actually steps in the right direction, paving some of the cow paths already created by npm and yarn. I for one would love to see package-lock.json etc make way for policies. Security is a concern for sure if there's no integrity checking on the packages loaded via https, but the same holds true for packages regardless of delivery method so I don't see how this is much worse security wise than just declaring dependencies in package.json. Policies does seem to be an effort to solve the integrity issue regardless of how the package is installed, which seems like a good idea to me.
If the caching and integrity issues are resolved well then this would essentially mean package managers become optional, no? Seems pretty good to me!
A lot of the complaints here claim that this is bad because it’s insecure. However, allow me to posit a question: what if I made a function that just fetches the script and evals it? Node.JS already involves running unsandboxed, practically untrusted arbitrary code, that nobody audits. I’d have more sympathy if not for the fact that you can pretty much emulate this with the permission level that Node.JS code runs today.
Also, no… this is not like PHP. PHP include got lambasted because it compounded other PHP security issues. If you use ESM imports, the obvious way, there’s never going to be an accident where user input can control the URL, because the grammar makes it impossible. You can use the dynamic import statement, but at that point you’re off into the abyss.
While this is true, if this is going to be the future, it may as well be provided as a feature with the proper precautions. While this doesn’t fully sandbox code, it does at least seem to provide some decent baseline security features. If I needed to import code from a web address for some reason, this is certainly the way to go, and whether ideal or not, sometimes that is a need that crops up.
That said, proper sandboxing and/or at least optional integrity checking would certainly be nice.
Evaluating these things at runtime is absurd. It was absurd when deno wanted to do it and it’s absurd now. People keep equating the threat model of npm and url imports, but with npm the untrusted code gets downloaded at build time, so tools like snyk can scan it and flag problems. Is snyk supposed to intercept the live download and patch files now?
Are the policy integrity checks they mention in the PR allow or block lists? For example, if my malicious trojan module does a dynamic import, can it bypass a saved integrity check by adding a cache bust url parameter?
Honestly, I don’t think the idea of URL imports is inherently bad. Go manages with them pretty reasonably. However, go only processes them at build time. My docker image from node might run today and break tomorrow because it’s allowed to change the remote code it runs day to day.
If you actually read the (admittedly long) thread, it's pretty clear that those imports are not resolved at runtime, and Deno has never evaluated imports at runtime either. Dependencies get evaluated and downloaded prior to startup, very similar to Go in fact.
I don't see that listed anywhere in the thread, so I'll ask here instead:
Does prior to startup mean there's a separate build step, or are you saying it happens during the loading of the script. I'm pretty confident that the latter is what deno was doing, although I'm happy to be proven wrong.
Are dynamic imports simply disallowed? Since it is impossible to download in advance a script to which you don't know the URL without running the program, they'd have to be, in order for your assertion to be true.
The whole conversation is a bit hard to follow, there was an issue initially, then this PR and then another Discussion that's linked somewhere in the PR. I'll let you go read those, I don't recall exactly what decisions were taken and what's the roadmap for this to become stable.
Dynamic imports are... wild, to say the least. Unfortunately they will always be inherently harder to secure, and will most likely always be treated differently than static import statements. At least that was the consensus in Deno.
ES imports on the web have a lot of restrictions around where they can be placed in code. Trying to pass a variable in for the path results in a syntax error. There are basically a lot of syntax requirements that make it basically impossible to do imports that aren't statically analyzable without the need to run the code.
However, given that ES imports are basically trying as hard as possible to be explicitly defined static imports that can be analyzed ahead of time before code is run -- it prompts the question, why would you ever choose to use that system in a native environment where you could have have an explicit install step?
ES Imports are trying very hard to imitate what Node already has, and I can't figure out what the advantage is of building an import system that imitates dynamic imports, but isn't dynamic. My understanding was that Deno went that direction just to make webpack bundling easier and to decrease the amount of new syntax. I felt like that was not great reasoning at the time, but I sort of get it.
But note that when I look at Deno's docs[0], the first paragraph I see says:
> One convention places all these dependent URLs into a local deps.ts file. Functionality is then exported out of deps.ts for use by local modules.
> Continuing this convention, dev only dependencies can be kept in a dev_deps.ts file.
It's almost like Deno is advising developers to put their dependencies in (just trying to come up with a random descriptive term off the top of my head) a package file. Next, wouldn't be it be great if that list of dependencies could also specify other things like a set of common scripts, or instructions on how to run the program if it's imported? There's potentially a really good concept embedded in those docs.
Deno was made by smart people, and it was made by smart people who also created Node and are intimately familiar with it. So I don't want to be dismissive because I'm sure they had some reasons for this, but I don't understand why any of this exists or who it benefits, or why the syntax is exciting given that Deno itself seems to be immediately advising people not to use it directly.
> ES imports on the web have a lot of restrictions around where they can be placed in code. Trying to pass a variable in for the path results in a syntax error.
That being said, yes, if there are plans to support dynamic imports, then literally all of the reassurances people have been giving me about pinning and fetching dependencies ahead of time and having tools to find out-of-date dependencies -- none of that works at all if dynamic imports are supported, you're completely correct.
Deno does support dynamic imports, but I have always understood that they use a separate system and are handled/installed differently? I might be wrong about that though, I haven't worked with Deno closely enough to be really confident about its internals.
At this point they just try to make the ecosystem even worse and open as many attack vectors as possible, aren't they? Has it become some "funny meme" in the maintainer circle or something? Why there is no SRI for example?
Won't this effectively prevent an application to start if it cannot reach the internet? I haven't done much development with Node, but I've seen similar patterns with Java applications that attempt to pull in an XSD at start up, only to fail to boot completely, because the application server doesn't have direct internet access.
It's a bit of a weird feature, which should only be used for a very limited number of project, in a niche which I'm not completely able to identify.
First thoughts, I generally think this is bad practice. I mean, for one thing this guarantees you need an Internet connection to run your Node script, which seems pretty problematic.
I'm also reading some really questionable security justifications from people on the issue tracker about why dynamic imports are actually fine and just as secure as importing packages from NPM. There are security improvements that could be made to NPM, particularly around package search, and NPM has suffered from people replacing existing package versions (although I think they're clamping down on this more now).
But there is a really fundamental difference between taking that risk once during installation and taking that risk multiple times at runtime. Additionally, in regards to some of the comments saying that installing code directly from the source ("express.com" vs "npm.com") ignores the fact that this is already kind of possible with npm dependencies and promoting that method wouldn't have any of the other problems.
----
HOWEVER:
You read further and there are actually some really interesting security risk mitigations being proposed that don't just boil down to saying "all imports are dangerous, why not do them here too." "Scopes" (which I was not aware was a concept that Node had any support for) are the idea that packages in a scope or a domain can't access any dependencies outside of them.
Additionally, people are talking about mirroring Deno's system where these get cached locally and not re-fetched later, which is kind of similar to npm's system and reduces some risk. So you can load a module from HTTPS and in theory it only gets fetched once and it can't access the file system, cool.
Additionally, `node_modules` dependencies can't use them, which is cool, means that it's less likely a random dependency starts doing this behind your back.
----
HOWEVER:
Prototype pollution is still a thing, passing objects around across closures is still a thing, and as far as I know this is a big part of why it's difficult to sandbox Javascript modules. Scopes are really interesting, but I can't find a lot of information about them online, and their name conflicts with npm scopes, which are a different thing. And I can't find any info suggesting they've solved the bigger problems of sandboxing modules.
And there's still the elephant in the room, which is that all of those security measures are stuff that could be done in a more general way that applies to normal modules? I'm not sure why I would only want scopes for HTTPS imports, assuming they even work securely for HTTPS imports in the first place, which I wish I could find more information about. It's cool that a module gets fetched once at runtime and not later, but that still means I need an Internet connection the first time the Node script runs. And it seems like "tell me what dependencies you need and I'll fetch them" is a good candidate for an install step, which Node has.
I'm not totally against the idea, I can in theory think of situations where you might want to install something at runtime without knowing ahead of time what it is. But that's also really dangerous to do, and doesn't really fit with the caching model that is being proposed.
----
So I still feel pretty doubtful that this is a good feature to add, and it seems really dangerous at first glance, but maybe there are other use-cases that I don't understand or maybe there's something to scopes that I don't get. I will say that if I actually trusted the security of scopes as a way of sandboxing individual modules, then at that point I feel a lot better about dangerous imports in general. But I can't find any info on scopes, and as far as I know, systems like Deno gave up on this because it was really hard, and I also vaguely feel like effective module-level sandboxing in Node would be much
1. This follows closely behind the recent addition of fetch support. I expected this would leverage that, but it does not. So I assume this was developed in greater isolation than I would have hoped. I don’t have time to look deeper now, but given overlapping responsibilities I would think fetch would be a better fit than the older http(s) APIs.
2. Again following close behind landing fetch, Node is gaining more and more feature overlap with Deno. It’s not clear yet, to me anyway, whether that’s a direct motivation… or a somewhat convenient coincidence of similar goals. If it is a direct motivation, I feel like API compatibility, while valuable, may not be the most valuable priority. At least speaking for myself, my primary attraction to Deno is its fast and build-/hassle-free TypeScript support. Everything else is icing on the cake.
118 comments
[ 3.2 ms ] story [ 177 ms ] threadThis let's you import code via a URL, seems quite similar?
In this case the flag has to be manually set and the code manually added as a dependency.
Basically every dependency manager out there will allow you to load "bad code" from a URL.
The feature is _strange._ Which we know because it's behind a flag, i.e., it will have low utilization. It defies conventions—because the convention is to use a registry for packages. And it presents significant security concerns. (Which I don't really have to explain, as they're already cited as the reason to put it behind a flag.)
Use of this feature seems like something you might see once in a career. (Indeed, I've seen a similar thing in another language employed only once... AND THERE WERE OTHER WAYS for that developer to have designed that system that would not have been so vulnerable.) When you see it, do you recognize what it is, understand and evaluate the risks it poses? Do you validate that the library sitting on the other side hasn't been taken over by a malicious actor? Do you validate that the library on the other side has up-to-date dependencies? If so, on what cadence do you do this, realistically?
All the security tooling progress we've made (npm audit, etc.) seems shot in the foot by this. And I get that these concerns aren't dispositive on their own. I just don't see the argument on the other side that clearly. Is there some class of application that people really want to write in node that can't be written for lacking this? Do we really think this is a pattern that would be employed in well-made, secure codebases? I don't see the argument. Like I said, I've seen a feature like this used in another language. But I haven't seen it needed for any real application.
I love me some good node ugly hacks. Still, the language features should be nudging us to write good software.
You can already pull your dependencies from arbitrary URLs, they just have to be tarred up first and go through npm: https://docs.npmjs.com/cli/v8/configuring-npm/package-json#u...
I'd know better later, but I'd still find horrendous options like these enabled in real software I would come accross in my professional career.
What people are pitching me on is the idea that these wouldn't be dynamic imports. They're saying they wouldn't be fetched during runtime, they'd be fetched before. They're saying they would be pinned.
Well, you're describing a package system like npm. And for those purposes, as a way of defining static packages that get installed before runtime and can't be dynamically swapped out by the application -- npm is better than the system being proposed.
The only way this would actually be interesting is if they were actually dynamic imports, but having them be dynamic imports would be very insecure. And for declarative imports that can be statically analyzed, this is inferior syntax to what we already have and encourages worse developer habits.
Some problems:
1. Updating dependencies is harder, because imports across multiple files have to be synced. If a junior developer imports `https://lodash.com/v4` in two files and later someone changes that to be `http://lodash.com/v5` in one file, now I have dependency mismatches. The existing system doesn't have this problem because my dependencies are in one place.
2. Finding out which dependencies I'm using is harder. Maybe there's another tool that lists out all of the imports, that would be handy. But also, maybe the dependencies are all in one file with the requested version numbers right next to them. That seems a lot more convenient.
3. This forces me to actually run the Node program to install the dependencies, and it forces me to actually import them in a file before they'll be fetched. This is cumbersome, it would be good to be able to list out dependencies before they ever get imported, say for boilerplate imports that are used in lots of projects.
4. People have pointed out the lack of pinning.
All of this is inferior to having all of the imports in one file that you can specify, maintain, and update without running the program or using an analyzer. I don't understand what the benefit is or why I should care about an import system that is just as inflexible as the current npm/Yarn package manager setup, but that also has what seems to be worse syntax and makes it harder for me to know what packages are being imported.
I wish I understood more what problem this proposed syntax was actually trying to solve. I've read through the entire thread and I still don't understand why people want this. People in the Github thread are reassuring readers that this could work exactly the same way as node_modules. That's great, but if that's the case, what is the problem with node_modules that is prompting us to build a second system that is less organized and less explicit about what gets imported?
We can already import URLs with both npm and Yarn. That's not a new feature, the only new feature is that now the URLs are spread across the entire codebase.
that’s a pro in my book.
If anything, this shows how ahead Deno is and how features like this in Node.js are too little too late.
Deno's standard library is just better than Node's, IMO, and it's going to be hard for Node to play catchup, if that's even practical. In summary, it doesn't contain the cruft of the pre-ES2015 era that is still present in Node, even though Node has tried working around it. The global objects in Deno better replicate the browser, so it takes a lot less effort to do essentially the same things you would in a browser but in a DOMless environment. Deno is slightly more performant in Node in some areas, although this is usually negligible, and I don't have the JSPerf test on hand.
I don't think Node is bad, by any means. If people are concerned about security... then don't use so many dependencies, or consider just not updating them until they've been demonstrated not to contain exploits; you don't actually need much code to essentially serve webpages.
But I think Deno, over time, will at least match Node in popularity, maybe even more, because it has "modern" approaches right out of the box and doesn't need to move as slowly as Node in order to maintain a legacy. Although Deno misses some things here and there, I've seen features get added very quickly over the last couple years; at this point, it would take Node much longer to do the same things for fear of breaking something and because it's simply much older code.
Then again, maybe I'll be wrong. Perhaps Node gets a second "big bang" and makes Deno obsolete. And I wouldn't care, either. All I want is a JavaScript language runtime with fewer concepts to learn that can also move quickly when something needs to be improved. If that ends up being Node, I'd be just as glad.
The main benefit I can see is that you can include code from your own "registry" very easily in unmanaged scripts. You can already do that because you can add a dependency to package.json using an https link already, but with this new feature you can include those dependencies in node scripts you aren't managing with a package file.
You can do that. I'm not suggesting you should. I won't.
On one hand do what you want, on the other hand you can already add https imports via package.json.
Why in God's name would you not have a package.json.
I think this would be awesome for hacking out prototypes, but if I saw this in production code I'd ether work to get it fixed or find a new job. Being reckless is fine in your spare time. It's not ok at work.
My fear is this is going to push junior devs into sloppy practices.
AIUI, when you add a module in go, the specific and exact version of the module is cryptographically locked in your project metadata. npm/yarn store the hashes in the lockfile but the package.json still uses those (usually) fuzzy matchers, and I'm not sure whether "npm i"/"yarn install" uses the lockfile exclusively by default. "npm help install" suggests that it uses package.json (which does not cryptographically secure the deps so that they match what you originally added) by default.
(Additionally, tons of people install and deploy docker images by tag name alone, which is not cryptographically secure - different environment, same problem.)
My mixed feelings about it are that we are slowly blurring the line between local and remote software, and pretty soon it will be nearly impossible to use most modern software in an offline environment (which is a hard requirement for certain security circumstances).
Basically every single js developer in the world has granted Microsoft (owners of GitHub and NPM domains) the ability to remotely execute code on their machine at will. It's a lot of tooling and infrastructure to update and change in the event that the threat landscape changes significantly (e.g. war, national security mandates, etc).
To make industry-wide improvements, we need to share resources for auditing of packages. There are many such endeavors underway at various stages of maturity.
But this feature doesn't move that ball forwards. From the comments:
https://github.com/nodejs/node/pull/36328#issuecomment-73603...
> It is insecure because there is no signature of the original file. Anybody that can take control of a domain name can take control of a server.
Downloaded arbitrary code can't be added to a pool of pre-validated packages, except insofar as you control the server and only make pre-validated packages available on it exclusively to your own services.
Setting up a enterprise mirror isn't hard.
NPM, as far I can tell was not designed to have you run npm install every single time you deploy. People decided to do it anyway, I still remember the day NPM's SSL certs went down ( 2013 or 2014). It's like the internet broke overnight.
It's a real catch 22. When you look for a magic npm package to save some time, you have no idea what's really in it unless you inspect all of it's code, and it's entire dependency tree.
I guess you could build an solution to check for any vulnerabilities, but then you just move the problem elsewhere. Said ML tool would never be perfect
For Node apps running on services like fly.io this may be useful, since they're regionally distributed and their claim of running closest to the customers using it is at least partially true. This makes your deployed images (and therefore disk size needed) much smaller
I agree that not having SRI as a requirement is a miss, however I see legitimate use cases here.
As for why, it's simpler than adding a package.json and meticulously curating it. If you only have a few small dependencies this might make sense. You can also pull in stuff that isn't hosted on npm or properly packaged--for example just pull down a js file from someone's github repo.
If you start getting a lot of dependencies then it might turn into a new nightmare to maintain. It's an experiment--people don't really know if it's good or bad until using it a bit.
<?php include("https://otherdomain.com/somephpcode.txt"); ?>
PHP got ridiculed for years for having this sort of stuff available (even if off by default eventually).
Loading code from remote domains can be abused and mis-used in any language. In this case, I think it's considered a "feature" because code was already being loaded remotely with NPM, who in turn got the code from the original author.
Now you can load the code directly from the author. So instead of trusting both the author and NPM, you only need to trust the author. This could be considered a reduction in your attack surface area, and it's definitely helping with the NPM registry being a single-point-of-failure in the Node.js ecosystem.
myRemoteLib = require(userGeneratedVariable)
A closer analogue to what React does would be comparing XHP[1] to JSX, but even that misses the isomorphic aspect of a SPA with SSR, which is almost the entire value proposition, because it allows you to only need to write your component code once while reusing the same render cycle at browser runtime.
It should be pretty evident that PHP-style CGI scripting is not ideal given that even early on many PHP apps moved to using template systems like Smarty and never actually embedding HTML directly into PHP code.
[1]: https://github.com/phplang/xhp
An additional of JSX in particular is that it’s render-target agnostic. Yeah you can render JS/DOM, and HTML… and CLI, and native UI APIs, and smart TVs, and WebGL, and…
(I know there are similar solutions for some of these in the others, but JSX was specifically designed to accommodate this.)
This also adds less dependence on npm, and maybe with all the packages that get hijacked there, this could be a reasonable move.
I'd also argue the claim npm is insecure. How would you improve its security over what's there now?
Lastly, there isn't really a dependence on npm at all. You were always able to 1) put URIs into package.json, and 2) make your own package manager. Node does not require npm to work. It just looks in node_modules by default, which npm uses to manage packages for you. They are completely decoupled.
URI imports, on the other hand, have a lot of security problems and surface area and will probably face a load of criticism going forward.
Automated builds on NPM servers based on code in public repos, like Docker has (had?). The least they could do is require packages to be signed.
Signing packages prevents account takeovers from publishing bad packages.
The remaining security problem is the author themselves coupled with too few eyes on public repos.
Most malicious package activity I've seen has been at the hands of the original maintainers to begin with.
The last thing can't be solved technologically.
I think npm could improve security with two steps: 1) all packages must be user-or-organization/package-name, and 2) all packages need to be signed with a key. Have namespaced-by-default packages at least will give a path for developers to trust an author or source. Signed packages would give developers the means to trust a key. All systems that allow public packages to be published are going to have inherit security flaws, and the best security I think we can give is trust. It's easy to invalidate a key, and it's easy to flag a namespace as untrustworthy.
While there is no "dependence" on npm, most package managers ship node with npm, and the typical flow for a node app, according to most tutorials and community documentation, is to use npm. Even most of the tools a developer would use with node are npm-centric (documentation almost never recommends cloning the repository, just npm install). So I'd agree, decoupled in reality, but is perceived as reality among tutorials and the community.
Instead of keeping up-to-date via npm I could just write `import type { FooType } from 'https://remote-types-registry.com/foopackage@2'` and wouldn't have to worry about typing updates. (surely this scenario is only valid when types are not shipped with the dependency)
Yes, we know there are problems with NPM. But why broaden the possible attack surface area by introducing another kind-of-sort-of-equivalent means of remote module loading? Now you're just going to have to spend time hardening two systems instead of one.
At the end of the day this just feels like a misguided attempt to try and recapture some sense of dynamism from Deno. But if I wanted dynamism, I'd use Deno, with all of its pubescent awkwardness. I'm using Node.js stable for a reason. I spend a lot of cycles trying to defend Node from its many (often reasonable) detractors, and things like this make me feel like I'm wasting my breath...
Second of all, centralization is not often a part of a security model. Usually in such cases where availability is a concern you have a mirror set up, which is ridiculously easy to do with CouchDB and completely nullifies your arrogant point.
Get off your high horse.
HN comments probably aren't the best place to go in-depth on this topic, but just off the top of my head, HTTPS imports allow a greater detachment from a centralized registry, something we're seeing used a lot in the Go ecosystem, and we're moving server-side JavaScript closer to how browser JavaScript works, which is a good thing for anyone using the language, as it reduces friction across the board. This might not be something obvious for people working mostly/only on the backend, in which case those frictions aren't as apparent, but they are there nonetheless.
I know that the people advocating for this are smart, I know you're being thoughtful about this, so forgive the simplistic question, but every package manager I'm aware of for NodeJS already supports this.
Can't people already import npm dependencies from arbitrary URLs without signing or attaching anything to the npm repository? And the other thought that jumps into my mind is, if people aren't doing that with npm that even though it's well-supported, that might be a signal that there are problems with the model of using arbitrary URLs for dependencies that make it less attractive overall to developers -- and I'm not sure how ES Imports using URLs would change or fix any of that.
I think this is where we are in disagreement. You're adding code to Node core in order to support this, which opens the door to new vulnerabilities (bugs) by definition, because people make mistakes. Now you will have two code systems to scrutinize and maintain that are doing essentially the same thing. I don't think the arguably-slightly-better ergonomics of HTTPS imports justifies the increased risk.
Idea for the new toolset start-up: - A cyberware to validate source code and its modules against https imports
If this had been implemented as something akin to the integrity HTML attribute, e.g.:
It would be vastly superior. I'd go so far as to require it without an additional --unsafe-import type flag.However, I think they did not do this because the import of dependencies then comes into question. IMO this feature should not have been added until developers could guarantee the expected integrity hash matches. We've seen time and again what will happen now is hundreds of blog posts talking about this new feature, none of which will mention the integrity hashing functionality (because it's not there). When it does eventually get added, those posts will be the most popular anyone can find, and thus new users will end up using those eventually-wrong posts and implementing things in a less-secure way.
Huge disservice to the developer community.
One upside is that it's https-only, which does include authentication, provided that you trust the holder of the certificate for my_local_repo.tld (in your example) to serve you the right files.
This is a perfect circumstance for the adage "trust but verify".
https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
But instead went with less-secure Cross-Origin Resource Sharing (CORS):
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
These operate at different levels of abstraction. SRI is mathematically strong, whereas CORS requires a bunch of hoops and mostly just gets in the way. I could understand an argument for SRI+CORS, but using CORS alone sets off alarm bells in my head.
CORS is primarily focused on removing csrf (cross site request forgery) vulnerabilities. It doesn't really care at all about what resources you pass around (or their integrity) it only cares about ensuring that the request is coming from a known and allowed origin.
basically CORS gives you zero defense against csrf. (ok, not zero, but also not much)
SOP/CORS was the foundation that allowed CSRF tokens to even work. (No SOP/CORS? Just request the form with the csrf token first, parse with js, append to next request)
I guess you could say SOP is really the protecting force here, but CORS only exists because strict SOP broke a LOT of stuff. Enter trust zones in IE, and then CORS.
SameSite now mostly tackles this by fixing the root issue with the cookie jar.
For the web, we could have been sourcing all of our javascript files from a central CDN and used the hashes to prove that they weren't tampered with this whole time. Instead, we went down the rabbit hole of build systems and endless local hosting.
For this Node.js use case, they could have allowed remote includes, with the understanding that it's only secure when the hash is specified. Verifying only the location is like CORS, it's security theater based on trust instead of a mathematical level of confidence about the contents.
Good point about dynamic responses not being compatible with SRI though. I hadn't thought about that use case, but you might be onto something about its lack of adoption. It may be a red herring however, because to make SRI dynamic, we'd need some kind of trust process that tells us that a new version's hash is safe, more like a BitTorrent tracker, but the file would still be requested through ordinary SRI. Also it doesn't apply to what we're talking about here, because package versioning means that the files are always static anyway.
I think this is a terrible idea that can only end in misery but I can understand why they don't add integrity validation by default.
They are different security mechanisms designed to defend against different threats.
One answers the question: Was this the correct server I got this package from?
The other answers the question: Does the package's hash match what I expect in my code?
As to why you would want this, I've worked on several projects that take advantage of some things that pulling dependencies only from a local repository can give you. Take the recent Log4J vuln as an example. Immediately on seeing that vulnerability, I disable the JAR file, causing builds to fail wherever that dependency exists until the updated one is added as a dependency and used. From experience having to do dependency analysis on an old toothy codebase, it's sometimes simpler and more accurate to just let everything break and fix the breaking spots.
If your goal is to make the build fail for vulnerable applications, you can run a local repository with upstream proxying functionality like Nexus Repository and kill the package there. This won't let you kill production (but why would you want to kill it this way?) but it'll give you the same benefit without risking remote code execution.
If my read on that is correct, then point then is to worry about the repository - wherever it lives - being poisoned, but still wanting some of the benefits of having a remote repository (such as centralized dependency management and control).
Doesn't Node cache imports, though? I think this approach would protect you if you were running a script on a timer (thus triggering a fresh load at a regular interval) but not if you were running an Express server (which would only verify the hash of the remote library at application startup).
Yeah, you might be on to something with that!
There's two separate questions:
A) is it ever good to have dependencies that can change underneath you? (arguably no)
B) if you don't want dependencies that change underneath you, why would you ever want to import directly from a URL at runtime?
----
> The code can update dynamically. You simply have to update the version number and verification hash on your end (same as you would in a package manager).
Code that I import through a package manager can update dynamically. I just need to update the version number and re-run `npm install`. Why would someone want to use a URL if the update process is the same as using a package manager?
I'm having a somewhat difficult time imagining a use-case for fetching dependencies at runtime that isn't related to not knowing until runtime what the dependencies are or what version is being fetched. Maybe for making installation smoother? But anything that requires compilation or is system-dependent isn't something you're going to want to re-fetch at runtime anyway. I suspect those dependencies will be slow to install.
And we're not dealing with a web situation where you might want to import from a CDN or another location that has better uptime. It sometimes makes sense not to serve assets yourself on the web just because it's expensive/difficult to serve them quickly. Node dependencies don't have that problem, pulling the dependency in as an install step will mean runtime performance is faster, not slower.
I get the argument that you shouldn't do dynamic imports that don't have resource integrity checks, but that seems like an argument against this feature in general -- you shouldn't really be doing runtime imports at all, because... why would you need them except for unpinned/unchecked dependencies?
Secondly, this doesn't mean all this code has to be dynamically loaded at runtime. You can still have equivalents of npm install which cache the linked resources locally on your server, perform integrity checks etc (this is exactly how Deno does it).
But npm has this already. You can already install from urls, you can even install from remote git urls. It's not even complicated, it's just `npm install git+ssh://user@hostname:git_repo.git#commit-ish`.
I get that npm isn't Node, but this feels like a lot of duplicated effort for something that is much more safely handled with the package manager that ships with every single default Node installation. It's not exclusive to npm either, Yarn supports this stuff too.
Are there really people building Node projects who are using a package manager that won't let them import from urls?
----
> Secondly, this doesn't mean all this code has to be dynamically loaded at runtime. You can still have equivalents of npm install which cache the linked resources locally on your server, perform integrity checks etc (this is exactly how Deno does it).
Unless Deno is crawling all of the source files in the project and doing this before the code is even run, aren't you still going to run into situations where you run your project and it hits an import it didn't see before, and it then gets dynamically imported at runtime?
On the web, ES imports are designed to be kind of inflexible and static so that you can't have those dynamic systems, it's all designed to make it harder to run into unexpected imports and easier to do static analysis of dependencies. And maybe Deno is the same, but if so then I don't see what the advantage is over doing that instead of putting all of your dependencies in a package.json and fetching them as part of an explicit step. The web made this stuff super static and inflexible to try and make dependencies more explicit, but an install step is kind of the end-goal of those restrictions, it's the most explicit way possible of importing dependencies.
And remember that npm lets you get your dependencies from anywhere, and they all get packaged in a predictable way, transparently on disk. You don't lose any flexibility in regards to where you get your packages from. At best, this system does the same thing, in which case why are we duplicating something that already exists? At worst, it caches resources only for future runs after it fetches them the first time, and it fetches them the first time at runtime, in which case I just don't understand why that would that be desirable.
Maybe conditional installs of packages so you can only fetch a subset of the potential dependencies you might need based on what is actually used during program usage? But this seems like a really error-prone, potentially dangerous way to achieve that.
Yes, but npm can only install npm packages. This is a step towards reducing/removing the dependency on npm itself from the Node ecosystem. It is duplicated effort, sure, but that effort is spent in undoing extra complexity and getting to a much easier "hello, world" state for both app and library developers.
> Unless Deno is crawling all of the source files in the project and doing this before the code is even run
That's exactly what it is doing.
I don't understand what you mean by this. What is an npm package?
Node already manages how `node_modules` get imported from disk, and it handles the package.json format; that stuff is already part of Node. The only thing that npm does is fetch the packages/dependencies, stick them on disk, and then do integrity checking with package-lock.json. And all of that is already somewhat optional. Even when you're using npm, there's no certification you need to go through, you don't need to use a proprietary format, you don't need to get anyone's permission to distribute your package, you don't need to sign anything. You need to write a bit of JSON and then zip your source into a tarball, and then anyone can use any Node-compatible package manager to import it and its dependencies.
npm isn't even a requirement for fetching packages or pinning resources, you can use something like Yarn as well (and arguably should, because Yarn has much better dependency pinning). You can even hand-write your own compressed packages that work with npm.
> It is duplicated effort
It is duplicated effort of functionality in NodeJS. And it's duplicated effort of good functionality, Node's `node_modules` system is decently good design. To the extent that we are getting rid of being able to use 3rd-party tools to fetch packages, we shouldn't do that. npm/Yarn have already proven that having multiple ways to import packages is beneficial, and being able to dictate one program that's in charge of how dependencies are checked for resource integrity and where they get stored on the disk is also good. We only got package-lock.json in npm after Yarn implemented it.
But I guess that's arguable and maybe someone thinks that we shouldn't have a package manager at all. I disagree, but sure, I'll concede that it would be reasonable for someone to think that.
> That's exactly what it is doing.
So for whatever reason, Node wants to handle fetching resources itself. Fine, we'll say there's a good reason for that. Having said that, when we look at the proposed syntax, why would you want your dependency declarations spread throughout your code rather than having all of your dependencies defined in one place? This sounds like a strict downgrade in terms of auditability, code complexity, etc.
You're saying that we're reimplementing things so we can undo extra complexity, but how is parsing out an import tree from source files simpler than reading a list of imports out of a package file? If the goal is to make a simpler system, I kind of like having a system where I can audit dependencies by hand rather than looking through source files and making a list of all of the imports.
Not to mention that this system also isn't flexible enough to handle dev-dependencies or global dependencies, so we're going to end up adding a bunch of complexity and extra syntax for that I suppose if we really want to get rid of npm/Yarn.
----
To play devil's advocate, I'll grant that now you can kind of imitate ES Imports without changing syntax. If I remember correctly, that was part of Deno's reasoning for using URL imports in code.
But again, why? ES Imports are a compromise due to the structure of the web and the fact that fetching packages at runtime on a website matters. Why hobble a native system with an import strategy that realistically wouldn't have been chosen for the web if any other option had been available?
If I'm understanding you correctly, this has built an install step that works the exact same way as a normal install step for languages with normal package managers, except it doesn't have strong verification guarantees, and it's harder to audit by hand, and (assuming I'm understanding the docs[0] correctly) no longer goes through an explicit install step and requires you to actually boot up the app once before...
One use case is if you have a JS module that depends on another JS module, and you want to be able to run it without transpilation in both Node applications and in browser environments. NPM-installed modules wouldn't be available to the browser engine, regardless of how NPM installed them.
I’m willing to give Node the benefit of the doubt that this will remain experimental with appropriate warnings until resource integrity is addressed to some reasonable level of confidence. I’m also glad to see it being a first party concern, especially given that it’s historically been/still is a much higher risk factor as both:
- delegated to their ESM loader API, which is incredibly difficult to get right, despite the API itself being deceptively simple
- a hilariously insecure example in their docs for same
It's unfortunate I think that the docs don't mention integrity checks or caching, but those things are mentioned in the PR thread. Integrity via policies which seems to be a new feature as well[1], perhaps designed to essentially obsolete things like package-lock.json et al? Caching is punted till later, presumably to not hold up progress on this particular feature.
Seems to me these are actually steps in the right direction, paving some of the cow paths already created by npm and yarn. I for one would love to see package-lock.json etc make way for policies. Security is a concern for sure if there's no integrity checking on the packages loaded via https, but the same holds true for packages regardless of delivery method so I don't see how this is much worse security wise than just declaring dependencies in package.json. Policies does seem to be an effort to solve the integrity issue regardless of how the package is installed, which seems like a good idea to me.
If the caching and integrity issues are resolved well then this would essentially mean package managers become optional, no? Seems pretty good to me!
[1]: https://nodejs.org/dist/latest/docs/api/policy.html
Also, no… this is not like PHP. PHP include got lambasted because it compounded other PHP security issues. If you use ESM imports, the obvious way, there’s never going to be an accident where user input can control the URL, because the grammar makes it impossible. You can use the dynamic import statement, but at that point you’re off into the abyss.
That said, proper sandboxing and/or at least optional integrity checking would certainly be nice.
Are the policy integrity checks they mention in the PR allow or block lists? For example, if my malicious trojan module does a dynamic import, can it bypass a saved integrity check by adding a cache bust url parameter?
Honestly, I don’t think the idea of URL imports is inherently bad. Go manages with them pretty reasonably. However, go only processes them at build time. My docker image from node might run today and break tomorrow because it’s allowed to change the remote code it runs day to day.
Does prior to startup mean there's a separate build step, or are you saying it happens during the loading of the script. I'm pretty confident that the latter is what deno was doing, although I'm happy to be proven wrong.
Are dynamic imports simply disallowed? Since it is impossible to download in advance a script to which you don't know the URL without running the program, they'd have to be, in order for your assertion to be true.
Dynamic imports are... wild, to say the least. Unfortunately they will always be inherently harder to secure, and will most likely always be treated differently than static import statements. At least that was the consensus in Deno.
However, given that ES imports are basically trying as hard as possible to be explicitly defined static imports that can be analyzed ahead of time before code is run -- it prompts the question, why would you ever choose to use that system in a native environment where you could have have an explicit install step?
ES Imports are trying very hard to imitate what Node already has, and I can't figure out what the advantage is of building an import system that imitates dynamic imports, but isn't dynamic. My understanding was that Deno went that direction just to make webpack bundling easier and to decrease the amount of new syntax. I felt like that was not great reasoning at the time, but I sort of get it.
But note that when I look at Deno's docs[0], the first paragraph I see says:
> One convention places all these dependent URLs into a local deps.ts file. Functionality is then exported out of deps.ts for use by local modules.
> Continuing this convention, dev only dependencies can be kept in a dev_deps.ts file.
It's almost like Deno is advising developers to put their dependencies in (just trying to come up with a random descriptive term off the top of my head) a package file. Next, wouldn't be it be great if that list of dependencies could also specify other things like a set of common scripts, or instructions on how to run the program if it's imported? There's potentially a really good concept embedded in those docs.
Deno was made by smart people, and it was made by smart people who also created Node and are intimately familiar with it. So I don't want to be dismissive because I'm sure they had some reasons for this, but I don't understand why any of this exists or who it benefits, or why the syntax is exciting given that Deno itself seems to be immediately advising people not to use it directly.
----
[0]: https://deno.land/manual/examples/manage_dependencies
I'm referring to dynamic imports, which do not have any compunctions around dynamic strings being used to import modules. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
----
That being said, yes, if there are plans to support dynamic imports, then literally all of the reassurances people have been giving me about pinning and fetching dependencies ahead of time and having tools to find out-of-date dependencies -- none of that works at all if dynamic imports are supported, you're completely correct.
Deno does support dynamic imports, but I have always understood that they use a separate system and are handled/installed differently? I might be wrong about that though, I haven't worked with Deno closely enough to be really confident about its internals.
It's a bit of a weird feature, which should only be used for a very limited number of project, in a niche which I'm not completely able to identify.
With npm I can easily get metrics for how stale dependencies are or quickly check for security issues.
The closest I found for deno was maybe trex? And that felt incredibly anemic.
First thoughts, I generally think this is bad practice. I mean, for one thing this guarantees you need an Internet connection to run your Node script, which seems pretty problematic.
I'm also reading some really questionable security justifications from people on the issue tracker about why dynamic imports are actually fine and just as secure as importing packages from NPM. There are security improvements that could be made to NPM, particularly around package search, and NPM has suffered from people replacing existing package versions (although I think they're clamping down on this more now).
But there is a really fundamental difference between taking that risk once during installation and taking that risk multiple times at runtime. Additionally, in regards to some of the comments saying that installing code directly from the source ("express.com" vs "npm.com") ignores the fact that this is already kind of possible with npm dependencies and promoting that method wouldn't have any of the other problems.
----
HOWEVER:
You read further and there are actually some really interesting security risk mitigations being proposed that don't just boil down to saying "all imports are dangerous, why not do them here too." "Scopes" (which I was not aware was a concept that Node had any support for) are the idea that packages in a scope or a domain can't access any dependencies outside of them.
Additionally, people are talking about mirroring Deno's system where these get cached locally and not re-fetched later, which is kind of similar to npm's system and reduces some risk. So you can load a module from HTTPS and in theory it only gets fetched once and it can't access the file system, cool.
Additionally, `node_modules` dependencies can't use them, which is cool, means that it's less likely a random dependency starts doing this behind your back.
----
HOWEVER:
Prototype pollution is still a thing, passing objects around across closures is still a thing, and as far as I know this is a big part of why it's difficult to sandbox Javascript modules. Scopes are really interesting, but I can't find a lot of information about them online, and their name conflicts with npm scopes, which are a different thing. And I can't find any info suggesting they've solved the bigger problems of sandboxing modules.
And there's still the elephant in the room, which is that all of those security measures are stuff that could be done in a more general way that applies to normal modules? I'm not sure why I would only want scopes for HTTPS imports, assuming they even work securely for HTTPS imports in the first place, which I wish I could find more information about. It's cool that a module gets fetched once at runtime and not later, but that still means I need an Internet connection the first time the Node script runs. And it seems like "tell me what dependencies you need and I'll fetch them" is a good candidate for an install step, which Node has.
I'm not totally against the idea, I can in theory think of situations where you might want to install something at runtime without knowing ahead of time what it is. But that's also really dangerous to do, and doesn't really fit with the caching model that is being proposed.
----
So I still feel pretty doubtful that this is a good feature to add, and it seems really dangerous at first glance, but maybe there are other use-cases that I don't understand or maybe there's something to scopes that I don't get. I will say that if I actually trusted the security of scopes as a way of sandboxing individual modules, then at that point I feel a lot better about dangerous imports in general. But I can't find any info on scopes, and as far as I know, systems like Deno gave up on this because it was really hard, and I also vaguely feel like effective module-level sandboxing in Node would be much
1. This follows closely behind the recent addition of fetch support. I expected this would leverage that, but it does not. So I assume this was developed in greater isolation than I would have hoped. I don’t have time to look deeper now, but given overlapping responsibilities I would think fetch would be a better fit than the older http(s) APIs.
2. Again following close behind landing fetch, Node is gaining more and more feature overlap with Deno. It’s not clear yet, to me anyway, whether that’s a direct motivation… or a somewhat convenient coincidence of similar goals. If it is a direct motivation, I feel like API compatibility, while valuable, may not be the most valuable priority. At least speaking for myself, my primary attraction to Deno is its fast and build-/hassle-free TypeScript support. Everything else is icing on the cake.