A generative operating system that directly predicts screen images based on mouse and keyboard inputs, powered by an RNN for state modeling and a diffusion model for image generation.
I tried to use this but the lag made it impossible to even click on an icon. On top of that, a message that other people were waiting popped up intermittently, pushing the emulation down the page, away from the mouse pointer. I'm not sure what sort of experience you're aiming for, but this probably isn't it.
I opened the terminal and typed `which bash`. This was interpreted as `ls`.. It's a very fun demo, but the utility of disregarding my input and trying to guess what I wanted to type is very questionable.
Tried to use this, also found lag made it basically impossible, and felt uncomfortable being reminded that other people might be waiting for me to get on with it.
However, I was able to click on a folder, it opened and looked fairly convincing. Only indicator that something was off - other than lag - was the at the bottom of the file browser, it mentioned how much diskspace was available: the first digit was clearly 6, the second was flickering and blurring between different numbers.
Pretty interesting idea though. What framerates should it run at? I felt I was getting <5fps.
This brings personal nostalgia to when I was very young and made an "OS" in PowerPoint using links between slides, animations, and the embedded internet explorer object. Similarly, I'm not sure I see any practical use in this. Still it's a really fascinating conceptual demonstration of networks understanding intent in the complex state-machine that is a graphical user interface.
Thanks everyone for trying out NeuralOS, and apologies for the frustrating user experience!
I coded up the demo myself and didn't anticipate how disruptive the intermittent warning messages about waiting users would become. The demo is quite resource-intensive: each session currently requires its own H100 GPU, and I'm already using a dispatcher-worker setup with 8 parallel workers. Unfortunately, demand exceeded my setup, causing significant lag and I had to limit sessions to 60 more seconds when others are waiting. Additionally, the underlying diffusion model itself is slow to run, resulting in a frame rate typically below 2 fps, further compounded by network bottlenecks.
As for model capabilities, NeuralOS is indeed quite limited at this point (as acknowledged in my paper abstract). That's why the demo interactions shown in my tweet were minimal (opening Firefox, typing a URL).
Overall, this is meant as a proof-of-concept demonstrating the potential of generative, neural-network-powered GUIs. It's fully open-source, and I hope others can help improve it going forward!
This reminds me of a recent conversation we had at work where someone suggested that at some point all backend APIs are going to get replaced by a single LLM that'll just do anything (if you ask it nicely enough).
I didn't get to do much. Had a hard time clicking on Firefox and then getting to the nav bar and type in "Hackernews". Boy was that wild watching it type. Those definitely weren't letters. Then it tried to translate the page for me into Finish and weirdly the "I'm not a robot" box would appear, disappear, and then I'd see the title of some paper. I never actually made it to the Google results...
It's an interesting project. I'll totally accept "for fun" or "because" but I'm interested in the why. Even if just a very narrow thing, is there any benefits we would get from using a ML based OS? I mean it is definitely cool and that has merit in its own right, but people talk about Neural OSs and I just don't "get it"
This is a very inspiring concept. Although it is still in its infancy, it demonstrates the generative capabilities of AI in the field of interaction and inspired me to think of some new possibilities. However, I admit that I was surprised when I first saw the title: Neural Network Operating System? Such nonsense marketing and misuse of terminology actually appearing on HN? :)
Very interesting idea. The entire computing experience, os applications, potentially even file systems etc exists only in the “imagination “ of the model.
Although this is of course ridiculously wasteful right now, I can see this being the optimal solution for many things if a technology like thermodynamic well based neural networks get to the point of viability.
A thermodynamic well based model could have a trillion parameters in the size of an so card at a few milliwatts of power.
In case like that it’s easy to imagine that mass produced implementations could be a one size fits all solution for all but the most trivial or advanced computing tasks. For perhaps less than a dollar for a 100b sized chip, you get the ability to “imagine” video, sound, etc and a strong general purpose “reasoning” capability imbedded into everything right down to children’s toys and toasters.
Kinda makes me think of Rick and Morty with the butter passing robot. A lot of pointless capabilities, but still cheaper than a purpose built deterministic computing device. OTOH having embedded knowledge as an ambient part of everyday life would be kinda neat, even if it would almost surely mean the end of human civilization lol.
What does it mean when you say "operating system powered by neural network"? Does it have a kernel space and user space with hard defined boundaries or is the network determining what function call is being made and switches the space based on it? what about security? what about networking? what about program execution? how does this actually work?
Note: The Space is intended as a template, so please duplicate it and run with your own GPU for a better experience. (The default Space has only one worker.)
Recommended GPU: At least an L40, ideally an A100-large. (The original demo at neural-os.com used H100s.)
All code and models are self-contained in the huggingface space.
Give it a permanent pre-prompt of “no bugs, no security holes, no ads, no tracking, no feed manipulation, no spam in search, online and subscription tools all backed up to run local, all online data also backed up local, all interactions or tasks to display in completed form within 100ms or less, … (burn everyone else’s Bitcoin when they attempt to transact, but not mine), …”
Looks like the entire mucky internet will be fixed with just some careful prompting as soon as this thing runs efficiently!
More seriously, it would be fun - and probably instructive - to play with a system that consistently (shallowly) simulated that. A kind of oasis.
Ok tried to execute 'ls'. Worked and it showed me some files. Tried to type 'less' and it just kept doing weird stuff. Removed my input and tried it again; it interpreted it as 'ls' and again showed me the content.
30 comments
[ 4.5 ms ] story [ 52.3 ms ] threadSee my tweet for more details: https://x.com/yuntiandeng/status/1944802154314916331
However, I was able to click on a folder, it opened and looked fairly convincing. Only indicator that something was off - other than lag - was the at the bottom of the file browser, it mentioned how much diskspace was available: the first digit was clearly 6, the second was flickering and blurring between different numbers.
Pretty interesting idea though. What framerates should it run at? I felt I was getting <5fps.
although i wasn't able to really use it due to lag
I coded up the demo myself and didn't anticipate how disruptive the intermittent warning messages about waiting users would become. The demo is quite resource-intensive: each session currently requires its own H100 GPU, and I'm already using a dispatcher-worker setup with 8 parallel workers. Unfortunately, demand exceeded my setup, causing significant lag and I had to limit sessions to 60 more seconds when others are waiting. Additionally, the underlying diffusion model itself is slow to run, resulting in a frame rate typically below 2 fps, further compounded by network bottlenecks.
As for model capabilities, NeuralOS is indeed quite limited at this point (as acknowledged in my paper abstract). That's why the demo interactions shown in my tweet were minimal (opening Firefox, typing a URL).
Overall, this is meant as a proof-of-concept demonstrating the potential of generative, neural-network-powered GUIs. It's fully open-source, and I hope others can help improve it going forward!
Thanks again for the honest feedback.
It's an interesting project. I'll totally accept "for fun" or "because" but I'm interested in the why. Even if just a very narrow thing, is there any benefits we would get from using a ML based OS? I mean it is definitely cool and that has merit in its own right, but people talk about Neural OSs and I just don't "get it"
Although this is of course ridiculously wasteful right now, I can see this being the optimal solution for many things if a technology like thermodynamic well based neural networks get to the point of viability.
A thermodynamic well based model could have a trillion parameters in the size of an so card at a few milliwatts of power.
In case like that it’s easy to imagine that mass produced implementations could be a one size fits all solution for all but the most trivial or advanced computing tasks. For perhaps less than a dollar for a 100b sized chip, you get the ability to “imagine” video, sound, etc and a strong general purpose “reasoning” capability imbedded into everything right down to children’s toys and toasters.
Kinda makes me think of Rick and Morty with the butter passing robot. A lot of pointless capabilities, but still cheaper than a purpose built deterministic computing device. OTOH having embedded knowledge as an ambient part of everyday life would be kinda neat, even if it would almost surely mean the end of human civilization lol.
What are the implications of relying on deep networks for instantiating and running the abstractions we usually hand upon physics and transistors
Is this a type of VM
Is an imagined VM Turing complete?
Note: The Space is intended as a template, so please duplicate it and run with your own GPU for a better experience. (The default Space has only one worker.)
Recommended GPU: At least an L40, ideally an A100-large. (The original demo at neural-os.com used H100s.)
All code and models are self-contained in the huggingface space.
Looks like the entire mucky internet will be fixed with just some careful prompting as soon as this thing runs efficiently!
More seriously, it would be fun - and probably instructive - to play with a system that consistently (shallowly) simulated that. A kind of oasis.
https://eaglemode.sourceforge.net/