Launch HN: Drifting in Space (YC W22) – A server process for every user (driftingin.space)
Many high-end web apps give every user a dedicated connection to a server-side process. That is how they get the low latency that you need for ambitious products like full-fledged video editing tools and IDEs. This is hard for smaller teams to recreate, because it takes a significant ongoing engineering investment. That’s where we come in—we make this architecture available to everyone, so you can focus on your app instead of its infrastructure. You can think of it like Heroku, except that each of your users gets their own server instance.
I realized that something like this was needed while working on data-intensive tools at a hedge fund. I noticed that almost all new application software, whether it was built in-house or third-party SaaS, was delivered as a browser application rather than native. Although browsers are more powerful than ever, I knew from experience that industrial-scale data-heavy apps posed problems, because neither the browser or a traditional stateless server architecture could provide the compute resources needed for low-latency interaction with large datasets. I began talking about this with my friend Taylor, who had encountered similar limitations while working on data analysis and visualization tools at Datadog and Uber. We decided to team up and build a company around solving it.
We have two products, an open source package and a managed platform. Spawner, the open source part, provides an API for web apps to spawn a session-lived process. It manages the process’s lifecycle, exposing it over HTTPS, tracking inbound connections, and shutting it down when it becomes idle (i.e. when the user closes their tab). It’s open source (MIT) and available at https://github.com/drifting-in-space/spawner.
Jamsocket is our managed platform, which uses Spawner internally. It provides the same API, but frees you from having to deal with any cluster or network configuration to ship code. From an app developer’s point of view, using it is similar to using platforms like Netlify or Render. You stay in the web stack and never have to touch Kubernetes.
Here's an example. Imagine you make an application for investigating fraud in a large transaction database. Users want to interactively filter, aggregate, and visualize gigabytes of transactions as a graph. Instead of sending all of the data down to the browser and doing the work there, you would put your code in a container and upload it to our platform. Then, whenever a fraud analyst opens your application, you hit an API we provide to spin up a dedicated backend for that analyst. Your browser code then opens a WebSocket connection directly to that backend, which it uses to stream data as the analyst applies filters or zooms/pans the visualization.
We're different from most managed platforms because we give each user a dedicated process. That said, there are a few other services that do run long-lived processes for each user. Architecturally, we're most similar to Agones. Agones is targeted at games where the client can speak UDP to an arbitrary IP; we target applications that want to connect directly from browsers to a hostname over HTTPS. In the Erlang world, the OTP stack provides similar functionality, but you have to embrace Erlang/Elixir to get the benefits of it; we are entirely language-agnostic. Cloudflare Durable Objects support a form of long-lived processes, but are focused on use cases around program state synchronization rather than arbitrary high-compute/memory use cases.
We hav...
57 comments
[ 3.4 ms ] story [ 121 ms ] threadI am a little confused about the product purpose and the definition of who your competition is in the market. I think new SaaS hosting providers are interesting, so please don't take any of this the wrong way, just hoping to give you some space to expand on your ideas more.
> Here's an example. Imagine you make an application for investigating fraud in a large transaction database. Users want to interactively filter, aggregate, and visualize gigabytes of transactions as a graph. Instead of sending all of the data down to the browser and doing the work there, you would put your code in a container and upload it to our platform. Then, whenever a fraud analyst opens your application, you hit an API we provide to spin up a dedicated backend for that analyst. Your browser code then opens a WebSocket connection directly to that backend, which it uses to stream data as the analyst applies filters or zooms/pans the visualization.
You say "put your code in a container", but... wouldn't you basically have to put all your gigabytes of data into a container? The bottleneck to the types of analytic applications you're describing seems unlikely to be the custom backend code, and far more likely to be whatever database is powering the application, which means that each interactive instance really needs to spin up a complete copy of the dataset to gain any performance benefit for these on-demand analytic workloads.
I've worked with a number of high-scale applications, and scaling the backend API server has never been even remotely the main challenge... plus, having dedicated instances of the web server process wouldn't make anything faster than just having an appropriate number of instances, it would just make it more expensive. It's almost always a question of scaling the database -- not the API layer. For offline analytic workloads like you describe, you could potentially spin up fresh copies of the database for each user, and that would make things better, but the challenge of scaling (online) OLAP and OLTP comes from the shared-everything nature of the database itself. If you're intending to provide unique database instances to each user, then all the data needs to either be packaged up with the application, or stored somewhere that the application can retrieve it on startup and load the database, which could be a time-consuming process that creates painfully long cold starts.
> Many high-end web apps give every user a dedicated connection to a server-side process. That is how they get the low latency that you need for ambitious products like full-fledged video editing tools and IDEs.
> We have not solidified a price plan yet, but we’re aiming to provide an instance capable of running VS Code (as an example) for about 10 cents an hour, fractionally metered.
Since you bring up the examples of running GUI desktop applications, I'm wondering if your competition isn't actually AWS WorkSpaces. Someone could build an image for a WorkSpace that includes everything the analyst needs, and then AWS will manage the lifecycle of that instance as the analyst connects and disconnects, billing entirely based on usage. That image could even include vast quantities of data pre-populated into a database, along with a web server that offers local dedicated processes to serve requests from the browser in the WorkSpace, if the company prefers to develop their application's GUI using the web as a platform.
Obviously the challenge with WorkSpace is if you want to offer it to parties outside your company, but AWS does address this use case to some extent: paulgb ↗ > You say "put your code in a container", but... wouldn't you basically have to put all your gigabytes of data into a container? The bottleneck to the types of analytic applications you're describing seems unlikely to be the custom backend code, and far more likely to be whatever database is powering the application, which means that each interactive instance really needs to spin up a complete copy of the dataset to gain any performance benefit for these on-demand analytic workloads.
You're right that it does depend a lot on the needs of a specific application. If a bunch of users are accessing the same dataset, and can constantly access the subset of data they need with low latency through a global index, and there isn't much need to do computation interactively at runtime, then a standard architecture is probably a better fit.
Where this approach is useful is if every user needs access to a different subset of the data (e.g. if the underlying dataset is petabytes, and each user needs to interactively explore a different gigabytes-big subset of it). Or if there is a lot of derived compute on top of it, for example, a graph visualization that needs to be updated when the user changes the subset of data in focus.
> I'm wondering if your competition isn't actually AWS WorkSpaces
The general approach of "run and render elsewhere and stream the pixels back" is definitely our competition in the sense that it's something companies currently do. What we provide is a way of moving the client/server boundary to wherever makes sense for your app: if it makes sense to render server-side and stream pixels, you can do that (although we don't yet support UDP, which would be useful in this case); if it makes sense to do data aggregation server-side but render through WebGL, that's also an option.
obviously, but it does'r mean certain HEAVY algorithms wont benefit from judicious use of mutability and getting closer to the metal. you can't always fan out your computation and sometimes you need to get the result out to the user fast. even if you dont NEED to, the user appreciates an experience that feels snappy.
> CPU usage becomes an architectural concern at a certain point,
depends what you're trying to do and the value of getting it done fast. Otherwise we'd never use lower level languges. python calls out to c for certain things for a reason.
> Even the fastest languages can't make e.g. a game server scale the way a CRUD app does.
true a slow language isn't going to be much worse than fast language for io, but a high performance system might be able to update that game state faster and passit back to the controller in a slow language that can take care of sending teh data out.
> CPU usage becomes an architectural concern at a certain point, not a language concern
thats a blunt assertion. I can think of plenty of use cases where its just as much a language concern. There's a reason dropbox and figma wrote performance critical parts of their system in rust.
for reference: I built my entire startup in elixir and have managed to get by using just elixir, judicious use of architecture and writing really tight sql queries. Yes, there are certain performance bottlenecks that can be addressed with architecture but to say that applies to all cases is foolish.
Luckily, our roadmap won't need any of these high cpu demands for ahwile. at some point, we want to use some sort of machine learning. There is just no way you're going to scale up large scale matrix operations over large data sets in pure elixir. elixir is copy om wrote and evaluate by value. Great ergonomics for day to day work but they have an overhead that is unacceptable under certain contexts. One being matrix operations. Luckily we have Nx which calls out to low level c code. Otheriwse we'd be using rust or python calling out to tensor-flow on a separate microservice.
I think we might be arguing past each other in more-or-less-agreement so I might bow out at this point.
[1]: https://sandstorm.io/
Exactly right. We do not actually require that every user gets their own container; that's a decision that's entirely up to your app. Our API spins up an instance and returns its hostname, and then you can connect to it from as many clients as you like.
I work on a data-intensive app that fits the use-case you describe but I'm confused about the benefits for performance. (can certainly see how the code would end up nice/simpler) Is this mostly applicable to certain stacks?
Not yet, but we're working on some demos of things that are easier with session-lived backends. One way to think about it is that it's good for repeated queries against the same subset of data -- if you have a dataset of petabytes and your typical use case has users (through filters or queries) repeatedly accessing a sample of ~gigabytes of that data throughout a use session, you could use a session-lived backend to materialize that subset of data in-memory and quickly serve queries off of it without hitting the global index.
Another case where it comes up is when you need to do some stateful computation after loading the data, for example, if you need to generate a graph or embedding layout of some data and refine the layout when users select/deselect data.
That's actually a very good way of putting it to people who understand the reference!
One of the things I've been playing with is actually using Spawner to spin up Jupyter Lab notebooks with their new(ish) collaboration feature. Jupyter and VS Code both work very nicely with Spawner's architecture out-of-the-box, since they can be put into a container and accessed entirely through an HTTPS connection.
Good luck!
I've experienced the need first-hand as well as talked to people who experienced it. The most prominent group of users are development tools, because that world has already embraced this architecture -- software like VS Code and Jupyter already takes the same approach, we just generalized it. One way of looking at it is that our bet is that applications other than dev tools will embrace this architecture too.
The example is only partly contrived; I began my career doing fraud analysis on ad market data and would run jobs overnight that computed an embedding layout, I wished for a way to recompute the embeddings on-the-fly as I filtered the data.
Did you guys build this on top of beam? my startup had a similar need for opening a process per user and we ended up using a combination of horde + genserver to accomplish something similar. In our case, we spawn a process that mainitains a websocket connection to an external service, maintain some state in there and relay updates to the user over a channel. There is one per client.
Have you guys tested Drifting in Space with executing users code and opening ports? (like replit)
So I guess the answer is “probably not in the near future, but maybe eventually” :)
I'm a solo open-source maintainer and have a popular project that people want to orchestrate many instances of. Each instance (a.k.a session) is stateful and individually configurable. I'm excited to test out spawner. Any company that makes it super simple for open-source maintainers make money by providing a managed service will be a huge success - from my initial thoughts, this looks to fit the bill.
If I can help with anything as you look into it, do let me know!
Just to illustrate where I'm coming from, what I have so far mimics the pm2 cli as an API with built-in reverse-proxy, with create (similar to init), reload, restart, start, stop and delete.
Some questions:
- Why do you need one process per user? For low latency, would you just need to make sure you have idle CPU to serve their request, even if that CPU time is multiplexed onto an event loop (one event loop serves many users)?
- Wouldn't this "event loop" actually be more efficient that one user/process, as there would be less context switching cost from the OS?
- Can I just keep a map of (connection, thread_id) on my server, and spawn one thread per user on my own server?
- Could I just load up my server with many cores, and give each user a SQLite database which runs each query in its own thread?
- This way a multi GB database would not be loaded into RAM, the query would filter it down to a result set.
> Why do you need one process per user? / Wouldn't this "event loop" actually be more efficient that one user/process, as there would be less context switching cost from the OS?
We're particularly interested in apps that are often CPU-bound, so a traditional event-loop would be blocked for long periods of time. A typical solution is to put the work into a thread, so there would still be a context switch, albeit a smaller one.
The process-per-user approach makes the most sense when a significant amount of the data used by each user does not overlap with other users. VS Code (in client/server mode) is a good example of this -- the overhead of siloing each process is relatively low compared to the benefits it gives. We think more data-heavy apps will make the same trade-offs.
> Can I just keep a map of (connection, thread_id) on my server, and spawn one thread per user on my own server?
If you don't have to scale beyond one server, this approach works fine, but it makes scaling horizontally complicated because you suddenly can't just use a plain old load balancer. It's not just about routing requests to the right server; deciding which server to run the threads on becomes complicated because you ideally want to decide based on the server load of each. We started going down this path, realized we'd end up re-inventing Kubernetes, so decided to embrace it instead.
> Could I just load up my server with many cores, and give each user a SQLite database which runs each query in its own thread? This way a multi GB database would not be loaded into RAM, the query would filter it down to a result set.
If, for a particular use case, it's economical to keep the data ready in a database that supports the query pattern users will make, it's probably not a good fit for a session-lived backend. In database terms, where our architecture makes sense is when you need to create an index on a dataset (or subset of a dataset) during the runtime of an application. For example, if you have thousands of large parquet files in blob storage and you want a user to be able to load one and run Falcon-type[1] analysis on it.
[1] https://github.com/vega/falcon
- Persists for the lifetime of the user session.
- Only processes a single user session.
- Has large amounts of CPU/RAM and writable disk to handle large datasets.
Car wasn't fast enough, so we removed the rear view mirror to lower weight. You are looking at the sexy fun to solve problem rather than the useful boring solution of throwing away the stack. Users can already run things like Solidworks in a web browser with near native performance using VDI.
> deciding which server to run the threads on becomes complicated because you ideally want to decide based on the server load of each
High end load balancers have done this since the 90s. This is now easily done with the nginx API.
Honestly I am sure there is some need somewhere for your stack. But hiring a good server/network operations team instead would have saved you a lot of code.
> High end load balancers have done this since the 90s. This is now easily done with the nginx API.
A load balancer doesn’t (or at least shouldn’t) do everything we need to do, which involves statefully mapping hostnames generated on-the-fly to servers in a cluster. This allows our users to create instances that multiple clients can connect to, as opposed to just using “sticky sessions” or something like that.
Our approach takes less code than you might think —- we lean heavily on nginx and Kubernetes where we can, so we only need to fill in the missing pieces.
> For quite a long time (and especially in the webdev world), there exists a perception that to achieve scalability, all our request handlers need to be as stateless as possible. In the world of the all-popular Docker containers, it means that all the app containers need not only to be immutable, but also should be ephemeral ... keeping our request handlers stateless, does NOT really solve the scalability problem; instead it merely pushes it to the database.
Though the problems and solutions pointed out in that article don't mean you have to go straight to process-per-X. One solution might be, as mentioned in passing in the OP's launch blog, to keep state in a cache like Redis. If the data fits this approach, it would ease load on the database while allowing each request handler to remain stateless.
Durable Objects seem less focused on heavy computation, but I think they're really interesting as points of synchronisation for e.g. collaborative editing. Having all requests go into a _single thread_ seems important.
[1]: http://ithare.com/scaling-stateful-objects/
2. Is the data processing stream or batch?
3. Could Mighty + DiS work together to completely accelerate a data- and UI-heavy application?
Context: I have been working recently with a reporting-heavy company that is continually using data analysis to understand risk, combat fraud, and identify key patterns in user actions and data.
[0]: Mighty Makes Google Chrome Faster (YC S19) -- https://news.ycombinator.com/item?id=26957215
Since our product is built directly into the SaaS app, it's up to the app's developer to decide at what level they want to split the work between client and server. Doing everything on the server and streaming pixels is one option, but I suspect most applications will want to take a hybrid approach where they do some CPU/memory-intensive work on the server, stream the data to the client, and use the client's GPU (via WebGL/WebGPU) to render it. So that's the approach we're currently optimizing for, but better support for pixel streaming is on our radar too.
2. It's up to the application layer; we just provide a way to run a container and the data layer is up to the app.
3. Yes, an app served by DiS is just a regular web app so you could use it in Mighty. Our hope is that because we shift some of the heavy computation to the server there are less use-cases where this makes sense, but there could be cases where you want to do GPU-heavy rendering which we don't yet provide.
Main tweak is our model grew to "small client browser GPU/CPU session <> serverside multi-node multi-GPU time sharing." Current cloud services (lambda etc) fail here: cold-start, mostly CPU, etc, vs bursty sticky GPU sessions. A lot more power when you can scale resource use... so 1 server process is kind of dinky. Good backend abstractions are tricky here though, so starting with 1 process makes sense as they figure out a sustainable revenue model, e.g., powering demanding visual intelligence apps is vastly different from powering commodity netlify CRUD apps.
The fraud analysis use-case is actually semi-based on a real world experience I had building tools for fraud detection in adtech in 2013, where I often found myself taking a time-slice of a graph and loading it up in Gephi to compute a layout. I'd written other browser-based tools to make my work easier, but because I was shoehorning everything into a stateless backend, computing a large graph layout as part of it was tough. So when I saw what you were up to with Graphistry, it immediately resonated with me (though I was no longer working on fraud at the time).
Funny enough... We've stacked up ~4 incarnations of how we back our different kinds of sticky live GPU session workloads, and ironically, a big one we aren't doing but I keep wanting to see solved is user-defined GPU containers (vs our own). So, good luck!
https://0pointer.net/blog/dynamic-users-with-systemd.html
The specific example use case that matches here:
> By combining dynamic user IDs with socket activation you may easily implement a system where each incoming connection is served by a process instance running as a different, fresh, newly allocated UID within its own sandbox. Here's an example waldo.socket:
> With a matching waldo@.service: > With the two unit files above, systemd will listen on TCP/IP port 2048, and for each incoming connection invoke a fresh instance of waldo@.service, each time utilizing a different, new, dynamically allocated UID, neatly isolated from any other instance.By allocating a new user ID for every invocation, you definitely limit the number of instances you can run—systemd only has a pool of 4336 dynamic user IDs (61184–65519) to allocate from, beyond which point I presume it’d refuse to accept any more connections. But it’s cool stuff, anyway.
(You could also just go for socket activation without a dynamic user, but I was thinking of this from the dynamic user perspective because that’s the more novel thing; socket activation has been around for much longer.)
In general I haven’t seen any really data-heavy apps in Elixir, are there examples I should be looking at? It could be interesting to compare performance.
Erlang/Elixir is a very neat little ecosystem, but in a lot of ways it's a dead end now. It was alone in its space for so long that it built a lot of ways of doing things that are kinda closed in on its own ecosystem, because there was no other ecosystem to reach out to to speak of, but now there's an abundance of choices and choosing Erlang means choosing something that is built on a lot of assumptions that don't match the world anymore. There may be some "reinvention" in building something on WASM and other communication mechanisms, but it's one with a path forward.
In particular, Erlang/Elixir have a lot of integrated solutions for modern code problems, but being either first or very early, none of them are best-of-breed anymore. You could think of them as the first draft of a lot of modern techs. Between that, and the fact that you can't build a business based on going to your customers and saying "Hey, everyone, I've got a great platform, just allocate the budget to rewrite your entire codebase into this somewhat obscure language and we'll make everything all better!", it just isn't a viable choice for a business, or at least not one that has any plans for growth. (And I don't mean VC-funded hypergrowth... I mean, the regular kind too.)
This is definitely overdue for a re-examination. "Web handlers should be stateless" goes all the way back to the 1990s, when a server system was lucky to have a single gigahertz, and even server systems could be looking at being loaded with the obscene quantity of maybe 128MB of RAM.
The solution is obviously not to just flip all the way to the other side. But the landscape has changed a lot since then. I've made a lot of hay out of very selectively stateful web services. It takes some care, but sometimes it honestly takes less care than trying to build completely pristinely stateless servers, because it's not like that's trivial all the time either!