Either the crap truly works and nobody is needed or it does not work. Where is this half-arsed human-agent hybrid vision coming from? The land of plateaued LLM gains?
IMHO, the goals is not to have to watch what agents do and let them do the work.
I would personally invest into making agents more autonomous (yes hard problem today) then building a desktop video session protocol to watch them do the work.
We're opening the private beta where we provide a hosted environment for testing, or you can install the latest Helix release and run the installer with --code to try it on your own GPUs
Ha, we'll take it! It's definitely ambitious - we're essentially building a full remote desktop streaming stack on top of an AI stack on private data.
The 'madness' part is probably the part where we said 'let's do hardware-accelerated H.264 encoding and real-time agent control and make it all work with sub-100ms latency.' We think the problem space is worth the complexity.
Hi, quite interesting project but have a hard time to understand why would stream a desktop.
From my (ignorant) understanding, the important part is the context of the LLM in the task. Some conversations you need visuals, some you don't. What's the advantage of giving a full desktop streaming instead of using integrations?
I’m curious how far are we from giving coding agents access to these desktop agents so that when we are using say Claude Code to build a native desktop app, the coding agents can actually see and act on the desktop UI that it is building
For web apps, I'd guess that many of us already do that via Playwright or other MCPs. I'd bet there are people doing something similar with desktop apps too.
The streaming architecture is designed for exactly this - we originally built it for autonomous agents that need persistent development environments. The missing pieces are mostly integration work (mapping Claude's tool use format to our desktop APIs). Would be very interested to hear if others are working on similar integrations - the combination of LLM coding agents + real desktop environments feels like it unlocks a lot of interesting workflows.
I commend the fact they acknowledge the maintainer's work, but seeing the singular 'maintainer', I can't help but notice the weight on that one person's shoulders.
I find this very curious. I don’t think agents care about UI, humans do. So in the end the UI is not required. As soon as the ai can get into the physical world. The whole IT world is done for. All of this will be automated away. The IT and CS only ever started to make us more productive more connected to improve our physical well being. When we don’t need to touch computers anymore there is no need for …
I’ve also independently concluded Moonlight was the best way to go after trying my hand at a very similar task. I didn’t want to dig through moonlight’s source, but I’m sure if you’re dedicated enough it would pay dividends later on, it basically does everything you’d need for realtime control in the setting of simulating human input.
20 comments
[ 3.2 ms ] story [ 40.6 ms ] threadI will join the woodworking people before that happens, thanks.
> Moonlight expects: Each client connects to start their own private game session
Nope, it's a Wolf design choice, eg. Sunshine allows users to concorrenly connect to the same instance/game
I would personally invest into making agents more autonomous (yes hard problem today) then building a desktop video session protocol to watch them do the work.
We're opening the private beta where we provide a hosted environment for testing, or you can install the latest Helix release and run the installer with --code to try it on your own GPUs
No shoehorns needed. Just take what you like and build what you need.
The 'madness' part is probably the part where we said 'let's do hardware-accelerated H.264 encoding and real-time agent control and make it all work with sub-100ms latency.' We think the problem space is worth the complexity.
From my (ignorant) understanding, the important part is the context of the LLM in the task. Some conversations you need visuals, some you don't. What's the advantage of giving a full desktop streaming instead of using integrations?
I commend the fact they acknowledge the maintainer's work, but seeing the singular 'maintainer', I can't help but notice the weight on that one person's shoulders.
Are there appliances or easy to deploy hardware that allow one to run these private models on-premise vs cloud
[0] https://kasm.com