You can see when JIT is disabled, the upcoming Static Hermes (Hermes V1) engine from Meta, built specifically for React Native, outperforms both V8 and JSCore on Apple Silicon
It'll be interesting to see how much it will affect React Native apps as it gets more and more optimized for this use case
I am pleased to see how complete this table really is. I recently migrated a tool from Otto to Modernrc's QuickJS transpile of QuickJS to pure Go. Both are represented.
It’s fascinating to see so many implementations targeting the same thing, along with the crazy variation in runtime size. I’d love to see memory usage comparisons too but I suppose you’d need to establish what you’re actually measuring first.
A few years ago I started work on a kind of abstraction layer that would let you plug Rust code into multiple different engines. Got as far as a proof of concept for JavascriptCore and QuickJS (at the time I had iOS and Android in mind as targets). I still think there’s some value in the idea, to avoid making too heavy a bet on one single JS engine.
What surprises me is that under the "Show variants" checkbox, SpiderMonkey 24 from 2013 outperforms 147alpha by ~2000 pts -- almost 10% --, while having 1/4 LOC, 1/3 binary size and almost 1:1 on other metrics. (SM 24 targets ES5, however)
All these JavaScript engines and it's still remarkably hard to find a robust solution for executing JavaScript from untrusted sources inside my own server-side applications.
Every time I look I find repos that look promising at first but are either unmaintained or have a team or just one or two maintainers running them as a side project.
I want my sandbox to be backed by a large, well funded security team working for a product with real money on the line if there are any holes!
(Playing with Cloudflare workerd this morning, which seems like it should cover my organizational requirements at least.)
To add some color to "workerd is not a hardened sandbox":
workerd does not include any sandboxing layers other than V8 itself. If someone has a V8 zero-day exploit, they can break out of the sandbox.
But putting aside zero-day exploits for a moment, workerd is designed to be a sandbox. That is, applications by default have access to nothing except what you give them. There is only one default-on type of access: public internet access (covering public IPs only). You can disable this by overriding `globalOutbound` in the config (with which you can either intercept internet requests, or just block them).
This is pretty different from e.g. Node, which starts from the assumption that apps should have permission to run arbitrary native code, limited only by the permissions of the user account under which Node is running.
Some other runtimes advertise various forms of permissions, but workerd is the only one I know of where this is the core intended use case, and where all permissions (other than optionally public internet access, as mentioned) must be granted via capability-based security.
Unfortunately, JavaScript engines are complicated, which means they tend to have bugs, and these bugs are often exploitable to escape the sandbox. This is not just true of V8, it's true of all of them; any that claims otherwise is naive. Cloudflare in production has a multi-layer security model to mitigate this, but our model involves a lot of, shall we say, active management which can't easily be packaged up into an open source product.
With all that said, not all threat models require you to worry about such zero-day exploits, and you need to think about risk/benefit tradeoffs. We obviously have to worry about zero-days at Cloudflare since anyone can just upload code to us and run it. But if you're not literally accepting code directly from anonymous internet users then the risk may be a lot lower, and the overall security benefit of fine-grained sandboxing may be worth the increased exposure to zero-days.
I got a question for everyone: as a web user, have you been affected by performance limitations of a particular JS engine? Have you switched browsers b/c of JS speed?
My n=1 as a long time Firefox user is that performance is a non-issue (for the sites I frequent). I’m much more likely to switch browsers because of annoying bugs, like crashes due to FF installed as a snap.
It honestly is pretty surprising, given that JS runtime runs website code single-threaded.
genuine question, let us say you took the source code of every single engine you see here and feed it to all the 10000 llms and ask them to analyze the code, optimize every function the best way they see fit, make architectural changes as and when they see appropriate, what do you think will be the result from cutting edge models?
22 comments
[ 3.3 ms ] story [ 36.9 ms ] threadAnd SpiderMonkey seems... not up there compared to the other 2
That's a staggering accomplishment.
https://github.com/Hans-Halverson/brimstone
It'll be interesting to see how much it will affect React Native apps as it gets more and more optimized for this use case
The amount of work just to aggregate and compare is admirable, let alone the effort behind the engines themselves.
How many of these engines are chasing benchmarks at the cost of increased memory usage?
A few years ago I started work on a kind of abstraction layer that would let you plug Rust code into multiple different engines. Got as far as a proof of concept for JavascriptCore and QuickJS (at the time I had iOS and Android in mind as targets). I still think there’s some value in the idea, to avoid making too heavy a bet on one single JS engine.
https://github.com/alastaircoote/esperanto
Every time I look I find repos that look promising at first but are either unmaintained or have a team or just one or two maintainers running them as a side project.
I want my sandbox to be backed by a large, well funded security team working for a product with real money on the line if there are any holes!
(Playing with Cloudflare workerd this morning, which seems like it should cover my organizational requirements at least.)
Update: Classic, even Cloudflare workerd has "WARNING: workerd is not a hardened sandbox" in the README! https://github.com/cloudflare/workerd?tab=readme-ov-file#war...
workerd does not include any sandboxing layers other than V8 itself. If someone has a V8 zero-day exploit, they can break out of the sandbox.
But putting aside zero-day exploits for a moment, workerd is designed to be a sandbox. That is, applications by default have access to nothing except what you give them. There is only one default-on type of access: public internet access (covering public IPs only). You can disable this by overriding `globalOutbound` in the config (with which you can either intercept internet requests, or just block them).
This is pretty different from e.g. Node, which starts from the assumption that apps should have permission to run arbitrary native code, limited only by the permissions of the user account under which Node is running.
Some other runtimes advertise various forms of permissions, but workerd is the only one I know of where this is the core intended use case, and where all permissions (other than optionally public internet access, as mentioned) must be granted via capability-based security.
Unfortunately, JavaScript engines are complicated, which means they tend to have bugs, and these bugs are often exploitable to escape the sandbox. This is not just true of V8, it's true of all of them; any that claims otherwise is naive. Cloudflare in production has a multi-layer security model to mitigate this, but our model involves a lot of, shall we say, active management which can't easily be packaged up into an open source product.
With all that said, not all threat models require you to worry about such zero-day exploits, and you need to think about risk/benefit tradeoffs. We obviously have to worry about zero-days at Cloudflare since anyone can just upload code to us and run it. But if you're not literally accepting code directly from anonymous internet users then the risk may be a lot lower, and the overall security benefit of fine-grained sandboxing may be worth the increased exposure to zero-days.
How much are you ready to pay for a license?
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
My n=1 as a long time Firefox user is that performance is a non-issue (for the sites I frequent). I’m much more likely to switch browsers because of annoying bugs, like crashes due to FF installed as a snap.
It honestly is pretty surprising, given that JS runtime runs website code single-threaded.