I'm fascinated in learning more about how you run a web server with this - the article says "WebContainers include a virtualized TCP network stack that's mapped to your browser's ServiceWorker API" but I'd love to understand a bit more about how that works.
Since this is all done with WebAssembly are you planning on targeting other stacks such as Python? The amount of time I lose helping other people getting their Python development environments to work remains horrifying.
Yes definitely! Firefox compatibility is close to done and will ship soon. Waiting on Safari to ship some additional features. They don’t share timelines so it’s unclear at this point.
Can you link to the blog post where you actually discuss literally any of the technical stack in any depth further than one sentence of wow more pizzaz much fast.
Could you point us to an introduction to how operating system concepts get mapped to browser concepts?
It sounds like each native binary gets compiled to a WebAssembly binary, but how do they communicate? How are the network and file system implemented? Are parts of the file system persistent? How does a system call work?
First, congratulations! This is an astonishing advancement.
I found this on the GH repo:
"Is this open source? Today no, but the API will be open source to developers at GA."
I'm confused by what "the API" is, exactly. Is WebContainers a technology you plan to make available for others to use node, in the browser, in their own apps?
> If they make it Open Source, Microsoft will add it to VScode and eat their lunch.
Put the code under the GPL so no one can use it to build proprietary software. Visual Studio Code contains proprietary components, so this would prevent that.
Hi, CTO of LeaningTech here. Congratulations for the highly polished demo. I think you might find interesting this one we did months ago: https://repl.leaningtech.com/?nodejs
With all the security and design problems of Node.js, when there are actually secure and reliable ways (harder ways, yes. Security and reliability are hard) to do all of this. Why?
> Faster than your local environment. Builds complete up to 20% faster and package installs complete >= 5x faster than yarn/npm.
Above is a quote from the post. I feel like this is a stupid question, but how can running yarn/npm in a browser on my machine be faster than running yarn/npm on my machine? Particularly when each page load runs a fresh npm/yarn install?
Nonetheless, this is a incredible piece of software that i'll be following closely
I've also asked myself this question.
I think the comparison is not with a local environment but rather with running node/yarn/npm on a remote virtual machine/ci. This is my hypothesis, I have no proof that's what they meant.
I'm wondering if there is more happening in-memory, even if the project files and modules themselves are on-disk (accessed via FileSystemAPI) the vscode/node/other bits including temporary files may be held in RAM once initially loaded and never swapped out.
If that is the reason, then if your machine becomes memory constrained, the performance will drop through the floor quicker than with full-local installs.
It could be a mix of this and your suggestion (lower network latency related bottlenecks than experienced with non-local deployment).
You mention being able to use the debugger. How do I do this? The docs are very poor.
I dont see any way to set breakpoints. When i use a debugger statement and open inspector it breaks on transpiled/packaged code (its different to the actual file). This is a deal breaker for me at present. Am I doing something wrong?
This pretty cool stuff.
I am wondering about another use case for this type of thing.
We have a CLI tool built with node.js, could we build one of these containers with with the CLI tool setup and expose just a terminal with access to the local file system?
You are right. In Firefox I get "StackBlitz v2 Beta currently works in Chrome and Chromium-based browsers. We’re hoping to add support for more browsers as they implement the necessary Web Platform features."
Oh no! The user experience must surely not be enriched! Damn those smart ass web developers who aren't like the rest of us that care about code quality and engineering principles!
I'm trying to get my head around how the networking of it works. Is local.webcontainer.io being overridden somehow on the browser like a hosts file might do? Trying to figure out how I'm hitting that URL from another tab
If by "works" you mean "serves up an error page". It's a standard domain, not real localhost, and it needs the service worker from the editor page running to work properly.
There is a service worker registered on the domain name, for example https://nextjs-agaw8i--3000.local.webcontainer.io/, then when you visit this URL in another tab it basically shares the service worker from the stackblitz tab.
For now. It works in Firefox today (minor issues keeping it feature flagged for now) and Safari is close to shipping WASM Threads, so this will likely work on all major browsers by EOY.
Can you please go into more detail as to what exactly is being compiled to wasm? The post does not describe it. In particular there seems to be a lot of confusion about whether the JS engine itself is running in wasm or not.
The largest wasm file I see downloaded on the website is 440K. Disassembling it, it seems to contain a bunch of libc-like filesystem code, and was built using the rust toolchain. So it doesn't look like v8 has been compiled to wasm here.
Rather, I assume they compiled a library or two, but otherwise they use the JS VM in the browser, and they've ported the Node.js runtime scripts to that environment.
That doesn't take anything away from the achievement here - it's really impressive! It's better to use the VM in the browser, if you can get those runtime scripts portable enough, which from other comments it seems like the answer is (or will be) yes.
Alright, so you can create an IDE in the browser, but, what else? I'm just not seeing the value in running servers in your browser. Presumably, most sites will still need to connect to a remote back-end services to get at data and whatnot, and that back-end isn't going to be running locally on a customer's web browser.
I don't know about your dev environment, but in mine we run server code separate from web code. So you could toggle your local WebContainer to access either localhost:3001 or a remote branch, staging or prod environment for whatever server you wanted to test against.
Isn't the opposite to "running remotely" "running locally"? Meanwhile if pressed for the opposite of "running native" I would answer "running virtualized".
The web devs where I work have started calling pretty much everything "native" as long as it interacts with the host in some way since people assume it means fast.
People have made Linux run on the browser, so... Although I don't think this is what they did here.
Man, the yet another layer they've made between code and the CPU makes me uncomfortable. Imagine finding a bug and going through 20 layers to troubleshoot it.
node.js, as any other native Linux/Windows/Mac application, talks to the OS via a system API. You can emulate that API using WASM, mapping I/O to browser APIs, which is why they mention network is implemented via service workers, for example. File I/O can be mapped to IndexDB... Threads can also be mapped to service workers, I guess... and so on.
emscripten[1] did this a long time ago for C, which is how a bunch of native applications have already been ported to run on the browser.
webcontainers[2] seem to do a similar thing but focused on exposing the browser API to the native apps in a way that integrates well with the JS environment.
basically node exposes all these other APIs like reading a file from disk. Browser also has APIs like detecting when the page is fully loaded that wouldn't be available on Node. What JavaScript syntax is supported can also vary between Node and browsers.
It can't access the file system, but it could implement an API that emulates a file system inside the browser sandbox. Presumably that's what they're doing here.
This is going full inception mode. Nodejs is compiled to wasm (like assembly for your browser), and then loaded inside the browser, which can then run javascript.
So a full JS engine is loaded, completely separate from the built-in one.
yeah that's what I gathered: javascript already runs on the browser, then we got node.js because people liked javascript and wanted to run it natively, and then finally we got this which allows us to run node.js on the browser again? it looks to me like we've gone full circle with extra overhead :D but I guess people have reasons to run node.js on the browser and not javascript so idk
I guess the idea is to access some Apis that are not present in the browser, like fs. This was mentioned in a question to StackBlitzs CEO, but he seems to be an AI that just pastes the same text over and over again.
> I guess the idea is to access some Apis that are not present in the browser, like fs.
That's why electron exists. But this can't do that, it's still limited by the APIs that are present in the browser. It's seemingly just full-overhead for... some... reason?
It isn't open source. It has a name, WebContainers, that implies it's based on the web, but it's designed around Node.js which isn't built on browser technologies as much as something like Skypack or Deno.
It also seems like it's going to be a memory, disk, and CPU hog, and that it's going to be pretty complex. I like where Skypack is headed and this seems like the opposite direction.
It's pretty cool but it seems like they're trying to create a lot of hype around it.
> I like where Skypack is headed and this seems like the opposite direction.
I did a quick google search because I had never heard of Skypack.
Skypack seems to be a far cry away from what WebContainers is claiming to do. Like, they're completely different products. I guess I don't understand why you're even comparing the two.
The future is going to be wild. You could bundle V8 compiled to WASM with a graphics toolkit that uses a syscall like interface to access a canvas backed by WebGPU?
At that point you could replace the browser with ZINE (Zine is not Electron) and run webapps natively just how WINE works.
ELI5... why would you want to run an Electon app in the browser? Isn't that just a website at that point?
EDIT: Upon closer inspection, it looks like this is an online IDE meant to mimic a desktop environment. I don't know if there's any demand for that, but sure, why not. I'm sure someone will come up w/ a reason to run WebAssembly in a jQuery plugin, too.
Thank you! Our entire team has definitely averaged 3hrs of sleep every night the past week to make this launch happen :) Incredible how much work goes into launching something like this.
As far as I understand JS code is being run natively in the browser. Node has been hooked to the native engine, so V8 is not involved.
At LeaningTech we have since several months been able to run nodejs, but also python and ruby using our Wasm based x86 VM (CheerpX), the demo is available here: https://repl.leaningtech.com/?nodejs
With our solution the full Linux x86 binary of nodejs is running including the whole of V8 and its JIT engine. Of course fully client side.
Not an expert on PyTorch in particular, the python + native (if any) components would work right now. If GPU acceleration is required some way to bridge into WebGL or (much better) WebGPU would need to be engineered, certainly doable but not existing right now.
Except for figuring out Pytorch's blas, the rest should be pretty straightforward to use CPU-only, I'd love to use that to (even pay to) compile some side projects to wasm apps in the browser :)
It looks like the difference is you went for full x86 system call level emulation to handle existing binaries whereas stackblitz decided to build a container for things like Node compiled directly to WASM with no static assumptions of a base system. The latter approach with 1 less level of indirection should be much more efficient, as is evident by startup times of the 2 demos. Not to mention the ability to hook into things above the syscall level.
I used to teach web development and was always frustrated with just how much effort went into setting up a machine. It would be really nice if a developer could just include a script tag in their project that setup a TypeScript compiler inside service worker. I managed to get the basics down, even fixing relative import statements. Requests like localhost:3000/main.tsx would compile and cache on the fly.
There were a few native file system issues I encountered that this would be perfect for!
PHP is actually pretty great these days, you should have a look at what modern PHP has become if you’ve been out of the loop for a while. Have a look at https://phptherightway.com/ if you’re interested.
VS Code seems to allow this with Dev Containers. I can download a repo with a `.devcontainer.json` and be up and running in virtually no time, in a container.
I wonder if that is the ideal path though, I was an instructor for a few subjects in my university and sometimes the biggest detractor for students was the lack of familiarity with the tool set used by the language,
for example, people sometimes did not know how to freeze their dependencies and use python virtual environments when working with python projects, that lead to problems in collaboration, handing students "magic" prefabricated environments could lead them to believe that that's all there is to it, and explaining that it is not to the student in an edge case might end up being less productive than understanding this process from the get go
you could say that the _effort_ to set up a development machine should be part of the learning experience, at least that's what I think.
I remember seeing my classmates having their first freelance jobs and editing minified css/javascript directly because they did not know anything about the transpiling that goes on or the toolset surrounding javascript and web development
It’s even funnier when those same devs who struggle with packaging and distribution in their best language go on to work at Fortune 500 companies and require constant devops assistance to actually make what they wrote work in even a test env. “But it works on my laptop!”
This is super cool! I think pairing this with the upcoming native support for modules and a CDN like skypack could lead to a truly powerful browser development experience.
No bundlers and a full IDE? Maybe this could even mean that there's a chance for a some of the 'just fiddle with the page' experience to come back to the web! (/rainbows and unicorns)
Wow, the PWA is really impressive. Honestly it's starting to get hard to tell the difference between this and Electron-based editors like VS Code. I'm really excited for integration with the File System Access API; that's the final puzzle piece I'd need to use something like this for serious work.
So it's 8 dollars for the astronaut plan. That's what I pay for robux to my kids, per week. It's jaw dropping to me how such advanced tech is made available at a single digit price. Thanks for sharing.
Me too and I was surprised to find that Chrome lists this as "ready" or "released" (I don't recall the exact term). I tried the example web based editor and found that the file system access API worked in Firefox too.
I'm working on a note taking project right now that I think would be great as a web app, but I'd like to store the result in a flat text file and the file system access API looks like it will let me do exactly that.
> ..found that the file system access API worked in Firefox too
For now, File System Access API seems to be available only in desktop Chromium browsers: Edge, Chrome, Opera. I believe Firefox has not implemented it yet.
> The ability to read and write from the filesystem is potentially very dangerous. We will need to carefully consider any solution in light of the security and privacy implications. We recognize that the spec authors take this issue seriously, but we are concerned that any solution will increase the risk of security incidents more than we are willing to tolerate.
> Right now, there isn't enough detail in the specification to make an assessment of these risks, so we will defer our decision until we have more information.
> Honestly it's starting to get hard to tell the difference between this and Electron-based editors like VS Code
Since Electron is just chromium, I'm not sure why you would expect a difference in the first place? The differences that previously existed between Electron & Web hasn't changed - Electron's only real purpose was to basically turn off the browser permission model & let the "web app" do whatever it wants. That's still true with WebContainers, it still has its hands & features tied to whatever Chrome & browsers in general are comfortable letting websites ask permission to do.
237 comments
[ 2.7 ms ] story [ 275 ms ] threadI'm fascinated in learning more about how you run a web server with this - the article says "WebContainers include a virtualized TCP network stack that's mapped to your browser's ServiceWorker API" but I'd love to understand a bit more about how that works.
Since this is all done with WebAssembly are you planning on targeting other stacks such as Python? The amount of time I lose helping other people getting their Python development environments to work remains horrifying.
We also did an hour long podcast a few months back that goes into pretty deep detail: https://www.youtube.com/watch?v=5F9qH-ea5Qk
Any plan to work towards supporting Firefox & other browsers?
Can you link to the blog post where you actually discuss literally any of the technical stack in any depth further than one sentence of wow more pizzaz much fast.
It sounds like each native binary gets compiled to a WebAssembly binary, but how do they communicate? How are the network and file system implemented? Are parts of the file system persistent? How does a system call work?
We also did an hour long podcast a few months back that goes into pretty deep detail: https://www.youtube.com/watch?v=5F9qH-ea5Qk
I found this on the GH repo: "Is this open source? Today no, but the API will be open source to developers at GA."
I'm confused by what "the API" is, exactly. Is WebContainers a technology you plan to make available for others to use node, in the browser, in their own apps?
If they make it Open Source, Microsoft will add it to VScode and eat their lunch.
If they don't, developers might be afraid of lock in.
Best exit for them is to just grow users and not make a decision either way until they get acquired by Github (aka Microsoft) and folded into VScode.
Put the code under the GPL so no one can use it to build proprietary software. Visual Studio Code contains proprietary components, so this would prevent that.
do you think that a docker.js running containers in the browser is something that could become a reality soon if we follow you line of work ?
It looks like the shell is JSH. Is this an in house shell? Is there any more information on it?
With all the security and design problems of Node.js, when there are actually secure and reliable ways (harder ways, yes. Security and reliability are hard) to do all of this. Why?
Above is a quote from the post. I feel like this is a stupid question, but how can running yarn/npm in a browser on my machine be faster than running yarn/npm on my machine? Particularly when each page load runs a fresh npm/yarn install?
Nonetheless, this is a incredible piece of software that i'll be following closely
If that is the reason, then if your machine becomes memory constrained, the performance will drop through the floor quicker than with full-local installs.
It could be a mix of this and your suggestion (lower network latency related bottlenecks than experienced with non-local deployment).
I dont see any way to set breakpoints. When i use a debugger statement and open inspector it breaks on transpiled/packaged code (its different to the actual file). This is a deal breaker for me at present. Am I doing something wrong?
(Using the http server template)
We have a CLI tool built with node.js, could we build one of these containers with with the CLI tool setup and expose just a terminal with access to the local file system?
Supporting chrome/chromium doesn't seem like a bad place to start with a prototype with that kind of potential adoption.
How? Isn't this using npm?
We haven't released any more info on Turbo v2 yet but will soon.
Rather, I assume they compiled a library or two, but otherwise they use the JS VM in the browser, and they've ported the Node.js runtime scripts to that environment.
That doesn't take anything away from the achievement here - it's really impressive! It's better to use the VM in the browser, if you can get those runtime scripts portable enough, which from other comments it seems like the answer is (or will be) yes.
Full WASI support in WebContainer is landing in the next 1-2 months.
Is wasm in a browser somehow considered native now?
I don't think this is universally true.
https://sql.js.org/#/
https://github.com/sql-js/sql.js/
Man, the yet another layer they've made between code and the CPU makes me uncomfortable. Imagine finding a bug and going through 20 layers to troubleshoot it.
Also: https://www.theregister.com/2017/03/23/cursor_devours_cpu_cy...
We also did an hour long podcast a few months back that goes into pretty deep detail: https://www.youtube.com/watch?v=5F9qH-ea5Qk
emscripten[1] did this a long time ago for C, which is how a bunch of native applications have already been ported to run on the browser.
webcontainers[2] seem to do a similar thing but focused on exposing the browser API to the native apps in a way that integrates well with the JS environment.
[1] https://emscripten.org/
[2] https://github.com/stackblitz/webcontainer-core
So a full JS engine is loaded, completely separate from the built-in one.
That's why electron exists. But this can't do that, it's still limited by the APIs that are present in the browser. It's seemingly just full-overhead for... some... reason?
My assumption was that the node APIs are simply being exposed to v8.
It isn't open source. It has a name, WebContainers, that implies it's based on the web, but it's designed around Node.js which isn't built on browser technologies as much as something like Skypack or Deno.
It also seems like it's going to be a memory, disk, and CPU hog, and that it's going to be pretty complex. I like where Skypack is headed and this seems like the opposite direction.
It's pretty cool but it seems like they're trying to create a lot of hype around it.
I did a quick google search because I had never heard of Skypack.
Skypack seems to be a far cry away from what WebContainers is claiming to do. Like, they're completely different products. I guess I don't understand why you're even comparing the two.
At that point you could replace the browser with ZINE (Zine is not Electron) and run webapps natively just how WINE works.
EDIT: Upon closer inspection, it looks like this is an online IDE meant to mimic a desktop environment. I don't know if there's any demand for that, but sure, why not. I'm sure someone will come up w/ a reason to run WebAssembly in a jQuery plugin, too.
Or a desktop-only Electron App would be usable from a mobile phone browser etc.
At LeaningTech we have since several months been able to run nodejs, but also python and ruby using our Wasm based x86 VM (CheerpX), the demo is available here: https://repl.leaningtech.com/?nodejs
With our solution the full Linux x86 binary of nodejs is running including the whole of V8 and its JIT engine. Of course fully client side.
How close is CheerpX to getting (for example) an app that uses a PyTorch model running in the browser? :)
What is the "native engine" if not v8?
There were a few native file system issues I encountered that this would be perfect for!
https://github.com/nirrius/my-server
sudo apt install apache2 php libapache2-mod-php; vim /var/www/html/index.php
Most things can be simplified to the raw essentials, doesn't mean you're going to achieve a result that people want or need.
Worked like a charm.
for example, people sometimes did not know how to freeze their dependencies and use python virtual environments when working with python projects, that lead to problems in collaboration, handing students "magic" prefabricated environments could lead them to believe that that's all there is to it, and explaining that it is not to the student in an edge case might end up being less productive than understanding this process from the get go
you could say that the _effort_ to set up a development machine should be part of the learning experience, at least that's what I think.
I remember seeing my classmates having their first freelance jobs and editing minified css/javascript directly because they did not know anything about the transpiling that goes on or the toolset surrounding javascript and web development
No bundlers and a full IDE? Maybe this could even mean that there's a chance for a some of the 'just fiddle with the page' experience to come back to the web! (/rainbows and unicorns)
Maybe one day, smartphones and tablets are just browsers in hand.
I'm working on a note taking project right now that I think would be great as a web app, but I'd like to store the result in a flat text file and the file system access API looks like it will let me do exactly that.
For now, File System Access API seems to be available only in desktop Chromium browsers: Edge, Chrome, Opera. I believe Firefox has not implemented it yet.
https://caniuse.com/native-filesystem-api
---
Mozilla's current position on the specs:
> The ability to read and write from the filesystem is potentially very dangerous. We will need to carefully consider any solution in light of the security and privacy implications. We recognize that the spec authors take this issue seriously, but we are concerned that any solution will increase the risk of security incidents more than we are willing to tolerate.
> Right now, there isn't enough detail in the specification to make an assessment of these risks, so we will defer our decision until we have more information.
https://mozilla.github.io/standards-positions/#native-file-s...
Since Electron is just chromium, I'm not sure why you would expect a difference in the first place? The differences that previously existed between Electron & Web hasn't changed - Electron's only real purpose was to basically turn off the browser permission model & let the "web app" do whatever it wants. That's still true with WebContainers, it still has its hands & features tied to whatever Chrome & browsers in general are comfortable letting websites ask permission to do.