ForeverVM: Run AI-generated code in stateful sandboxes that run forever (forevervm.com)
While this works, it was clunkier than what we would have wanted in a first-principles code execution product. We built ForeverVM from scratch to be that product.
In particular, it felt clunky for app developers to have to think about sandboxes starting and stopping, so the core tenet of ForeverVM is using memory snapshotting to create the abstraction of a Python REPL that lives forever.
When you go on our site, you are given a live Python repl, try it out!
---
Edit: here's a bit more about why/when/how this can be used:
LLMs are often given extra abilities through "tools", which are generally wrappers around API calls. For a lot of tasks (sending an email, fetching data from well-known sources), the LLM knows how to write Python code to accomplish the same.
Any time the LLM needs to do a specific calculation or process data in a loop, we find it is better to generate code than try to do this in the LLM itself.
We have an integration with Anthropic's Model Context Protocol, which is also supported by a lot of IDEs like Cursor and Windsurf. One surprising thing we've found is that once installed, when we ask a question about Python, the LLM will see that ForeverVM is available as a tool and verify it automatically! So we cut down on hallucinations that way.
54 comments
[ 2.2 ms ] story [ 101 ms ] threadYC lists all their companies here: https://www.ycombinator.com/companies.
Many companies are likely happy to take your small check if you are a nice person and can be even minimally helpful to them. Note that for YC companies you'll probably have to swallow the pill of a $20M valuation or so.
LLMs are generally quite good at writing code, so attaching a Python REPL gives them extra abilities. For example, I was able to use a version with boto3 to answer questions about an AWS cluster that took multiple API calls.
LLMs are also good at using a code execution environment for data analysis.
isn't that very dangerous? The LLM may do anything, e.g. create resources, delete resources, change configuration etc
Then of course there's the whole UX problem of even when you restrict the interface to safe queries, the LLM may still generate queries which are completely incorrect. The best idea I can come up with there is to dump the query text to an editor where the user can review it for correctness.
So it's not really "natural language queries" more like "natural language SQL generation" which is a completely different thing and absolutely should not be marketed as the former.
People bring up this concept as a way to make systems "more friendly to novice users" which tbh makes me a little uncomfortable, because it seems like just a huge footgun. I'd rather have novice users struggle a bit and become less novice, than to encourage them to run and implicitly trust queries which are likely incorrect.
So it's a bit difficult to tell how much value is added here over some basic intellisense style autocomplete.
Looking to the world of "real tools" like hammers and saws, we don't see "novice hammers" or "novice saws". The tool is the tool, and your skill using it grows as you use it more. It seems like a bit of a boondoggle to try to guess what might be good for a novice and orient your entire product experience around that, rather than simply making a tool that's good for experts doing real work and trusting that the novices will put in the effort to build expertise.
It makes for a flashy demo, though.
Since LLM context is limited, at some point the LLM will forget what was defined at the beginning so you will need to reset/ remind the LLM whats in memory.
The utility here is mostly that you're not paying for compute/memory when you're not actively running a command. The "forever" aspect is a side effect of that architecture, but it also means you can freeze/resume a session later in time just as you can freeze/resume the LLM session that "owns" it.
That seems to be a common misunderstanding.
Smalltalk implementations are usually 4 files:
-- the VM (like the J VM)
-- the image file (which you mention)
-- the sources file (consolidated source code for classes/methods/variables)
-- the changes file (actions since the source code was last consolidated)
The sources file and changes file are plain text.
https://github.com/Cuis-Smalltalk/Cuis7-0/tree/main/CuisImag...
So when someone says they corrupted the image file and lost all their work, it usually means they don't know that their work has been saved as re-playable actions.
https://cuis-smalltalk.github.io/TheCuisBook/The-Change-Log....
> Is this the best way to build applications? Almost assuredly not.
False premise.
Looking at the docs, it seems only suited for interpreted code, but I’d be interested to know if this was feasible or almost feasible with a little work.
Compare/contrast: the Smalltalk software distribution model, where rather than shipping a VM + a bunch of code that gets bootstrapped into that VM every time you run it, you ship an application (or more like, a virtual appliance) as a VM with a snapshot process-memory image wherein the VM has already preloaded that code [and its runtime!] and is "fully ready" to execute that code with no further work. (Or maybe — in the case of server software — it's already executing that code!)
In complex use cases like building a bi engineer, it’s useful to persist state across multiple function calls within the same interpreter.
I forget who (lcamtuf?) made a VM thing for fuzzing that used it.
For a clean implementation I’d look at socket-activated rootless podman with a wasi-sdk build of Python.
We mention Claude a lot because it is a good general coding model, but this works with any LLM trained for tool calling. Lately I've been using it as much with Gemini Flash 2.0, via Codename Goose.
What are some of the edge cases where ForeverVM works and doesn't work? I don't see anything in the documentation about installing new packages, do you pre-bake what is available, and how can you see what libraries are available?
I do like that it seems the ForeverVM REPL also captures the state of the local drive (eg. can open a file, write to it, and then read from it).
For context on what I've tried: I used CRIU[1] to make the dumps of the process state and then would reload them. It worked for basic things, but ran into the issues stated above and abandoned the project. (I was trying to create a stack / undo context for REPLs that LLMs could use, since they often put themselves into bad states, and reverting to previous states seemed useful). If I remember correctly, I also ran into issues because capturing the various outputs (ipython capture_output concepts) proved to be difficult outside of a jupyter environment, and jupyter environments themselves were even harder to snapshot. In the end I settled for ephemeral but still real-server jupyter kernels where I via wrapper managed locals() and globals() as a cache, and would re-execute commands in order to rebuild state after the server restarts / crashes. This allowed me to also pip install new packages as well, so it proved more useful than simply static building my image/environment. But, I did lose the "serialization" property of the machine state, which was something I wanted.
That said, even though I personally abanonded the project, I still hold onto the dream of a full Tree/Graph of VMs (where each edge is code that is executed), and each VM state can be analyzed (files, memory, etc.). Love what ForeverVM is doing and the early promise here.
[1] https://criu.org/Main_Page
We've generally prioritized edge case handling based on patterns we see come up in LLM-generated code. A nice thing we've found is that LLM-generated code doesn't usually try to hold network connections or file handles across invocations of the code interpreter, so even though we don't (currently) handle those it tends not to matter. We haven't provided an official list of libraries yet because we are actively working on arbitrary pypi imports which will make our pre-selected list obsolete.
> Love what ForeverVM is doing and the early promise here.
Thank you! Always means a lot from someone who has built in the same area.
This is interesting! How did you end up achieving this? What tools are available for rolling back LLMs doing?
What are you doing to prevent the abuse?
We give every repl its own network namespace and virtual ethernet device. We also apply a set of firewall rules to lock it out from making non-public-internet requests.
This is meant to be usable for those use cases, but also to allow apps/agents to make API requests, load data from various sources, etc. It can also run in a company's cloud account, for compliance situations where they are running inference on their cloud account and want a ChatGPT-like code interpreter where data never leaves their VPC.
So I can tell it to install asw cli for example and have it control my instances and such? Cool