Terminal world models have many applications, but empirical observation of successful pretraining or fine-tuning can be significant for encapsulating from rather expensive Bash traces, such as ones obtained from systems with scarce accelerators, etc.
Thank you for linking that. I distinctly remember this moment as an important point in time, played around with emulating a terminal for hours, couldn’t believe how realistic it was, it even remembered when I changed a file earlier etc. Another point was when people on twitter with early access were like “oh my god it can write code too! I asked it to design a website and it just works!”, programming wasn’t even what LLMs were about, and now it’s one of the main use cases. Another revelation was when it could kinda play chess. A single model that can one shot a huge array of different problems, still mind blown how well all of this works to this day.
That seems as likely as anything, but that just moves the problem to counting, doesn't it? The model gets the duration, but I was under the impression LLMs struggled with the arithmetic that would make that useful.
I regretfully suspect this actually is the future of UI. You just need smart local models running fast enough to keep the framerate high, but that’s inevitable in the long term, isn’t it?
Why would you do that instead of having the model do the task you're using the UI for? It could generate UI widgets on-the-fly as needed to take richer inputs.
It might be fine for one-off applications, but it's incredibly inefficient for something that has to be done over and over.
Why would you use a GPU and gigabytes of VRAM running a non-deterministic process for something a program can do on the CPU in milliseconds with kilobytes of memory?
The logic here is a bit different. Consumer software is slow because it's the user paying for the compute to run those bloated Electron apps. Unless we're all running local inference in the future, it would be the vendor paying for the compute to run the AI-based UI.
Of course they would pass that along to the consumer, but nobody is going to pay token-based pricing for something that used to be free and ad-supported, or a cheap monthly subscription.
Yes, Tauri is awesome. But now you’re reliant on hiring Rust developers instead of cheaper JS developers. Which comes back to my point about cost savings.
Nowadays you hire a developer that can use Claude or Cursor out whatever and they will use any language. There are no expensive Rust developers anymore. Or even JavaScript ones.
i did this for my company and now im considering we need an ui, the reason is 80% of the tasks everyone uses this thing for are the ame tasks, but with different prompts, often bad prompts, so a button that would say do X instead of the "hello ai how are you today i was thinking y but maybe z actually do x" may be better. havent tested it yet though
We had decades to work on making software more performant, but instead most users' space-age 2nm chips spend most of their time running Javascript because it was faster to make software that way.
I think it's hard to understand how much better your experience could be if all your software was 10x faster and using 0.1x the resources, since you can't test both experiences side by side
It's like riding a bike with a tiny bit of resistance in the bearings. You will get used to it, and will probably feel like your bike is fine. But if you replace the hub and remove that friction, you will wonder how you ever tolerated riding a bike that way.
I just wonder if it points at something more fundamental. We could have always written the code to use less memory, for example embedded code, but unless constrained we don't.
There's a good reason why the UI for Excel, Photoshop and Word has hardly changed and remains consistent for years and that has worked for billions of people.
Many HN readers have forgotten the Windows 8 UI disaster already.
The office UI has changed dramatically over the years. We have floating toolbars, live preview, ribbons, collapsing menus, an undo button in the title bar, a tabbed interface. We've moved from SDI to MDI and back to SDI. If you had a customized toolbar layout in office, upgrading to 2007 meant you had to learn to use office the way Microsoft wanted it. The default font changed multiple times. Chart wizard, clip organizer (and later clip art as a whole) were removed. Print preview changed to a tab. Contextual help (the `?` button) was removed when Microsoft got rid of winhlp, followed by the office assistant, followed by offline help as a whole.
Want to extend the idea to office as a whole? Lync was replaced with Skype, OneNote came and went and then came back again. Groove, infopath, publisher, frontpage, SharePoint, various versions of Outlook, the office shortcut bar, photo editor, picture manager - they've all disappeared (some have come back again or been replaced with something else).
I used office XP and 2000 (blocked at the firewall; I mostly saved files as rtf for personal use) for years (admittedly this is mainly because they loaded in 0.5s on my crappy laptop and PC) until I was forced into upgrading to office 2007 by the need to pass around files for class with someone who complained incessantly about their stylesheets being lost every time I converted the file.
I'm still salty I can't waste my time editing 16x16 (32x32?) 256-color dicks into the toolbar icons i when I'm supposed to be writing essays or setting up autocorrect to quietly put typos into a shared computer (the computer isn't shared anymore, and it's no fun doing it to myself)
Even assuming that the hardware and models catch up, wouldn't there be an issue with the model not being completely deterministic? You don't really want any sort of randomness in your terminal commands.
very good question, but during training it gets the actually characters as input, and then * as what gets rendered, so I think it should understand the difference.
The mistake correction capabilities are fascinating, have you experimented around that more, like breaking it up in 2 models? one that generates one that repairs?
Wondering if would make any difference in results, which could be impactful given your observation that "Self-correction was one of the more effective training changes"
>I used Qwen2.5-Coder-0.5B as the starting point. I needed a small model to get close to realtime performance. Being a coding model, it had also already seen plenty of CLI usage.
Interesting. I wanted to ask, how that model compares with the fine-tuned version, but I just saw the author provided benchmarks. Interesting that fine-tuning improved the memory so significantly, wouldn't have expected that.
Though, on second thought, I had an experience where I was able to, as a human, fine-tune my memory beyond what I previously thought was possible!
Reminds me of one of my favorite jokes.
Excuse me, can you tell me how to get to Carnegie Hall?
I'm not the person you're replying to, but there are techniques like memory palaces that can let many people pull off impressive memory tricks like memorizing a deck of cards.
Joshua Foer did some articles where he was looking into how people do the memorization tricks, ultimately concluded that most anyone can do it, and he actually went on to win the US memory competition after a year of practice. He wrote several articles and a book about it I think, so I'm not sure which one I originally read about it.
Oh, so I was smoking weed with a friend and went "what were we just talking about?"
The guy tells me to rewind. You can just rewind the conversation. Put in a little effort and run it backwards.
That's a skill and apparently I'd never practiced it before.
After a few weeks, I was being complimented for my "excellent memory", while I was high (as someone who naturally had a poor memory even while sober).
--
Also worth mentioning, I met a guy once who was much like me, he'd go on like 5 different tangents... but then he did something spectacular, he'd tie them all together into one glorious point.
I assumed that was because he was some kind of genius, but now I'm wondering if that's learnable as well.
I wonder how much a small model could do if it had a tool calling API for a filesystem, an embedded busybox, and a single output token for 'emit what busybox would do here'. Then presumably the model parameters could learn everything that diverges from what busybox can handle.
Very very cool! Impressed that it can be done with only fine tuning
On a very similar theme, I recently built a “lossy git”, which stores your code by training a neural network, then it generates the code back to you on checkout
The web demo runs entirely on the browser, both the version control and the neural network training/inference. You can commit and checkout both images or code - you can also paste/“upload” your own images, no server requests (everything clears from the page on reload)
That's very cool project.
I read this blog post a few days ago about one way to look at memory encoding in models https://alexwang.ai/posts/associative-memory/ that you might find relevant
Thank you for the link. Yes! loved that post. I had just created the cli version (https://nicobrenner.github.io/lossy-git/) and it inspired me to create the web version
Also, I really appreciate the excellent detailed writeup on your project. Curious about the mistake correction capabilities, it's fascinating, did you try breaking it up in 2 models? one that generates one that repairs? Just wondering if would make any difference in results, which could be impactful given your observation that "was one of the more effective training changes"
You jest, but that actually sounds like a really good research direction. Imagine what you could do if merging two codebases was as easy as merging with neural git.
Love it. In fact the demo already does something like that. It trains 2 models, the neural net that saves the data, and a classifier that trains on correcting the data - and it has a huge impact, you can go from 30% fidelity to 90-100% just with that
That's actually a really interesting idea. If we do end up in a "we don't look at the code ever" future, then something like this could actually be sensible.
> Commit code, it trains. Checkout, it reconstructs from memory.
I see a "Commit" button, but I don't see any checkout, I guess that is what happens automatically? Although that panel says "Reconstruction", not "Checkout"... You might want to do a pass after the AI created the initial version, and make it "humanly consistent" or similar, as right now it seems a bit incoherent.
> Run the JS code before and after a commit
How am I suppose to "run the JS code", manually writing it in my local JS console? Or is this just "Click the green button in the UI" but expressed in the way only a LLM would write it?
> How am I suppose to "run the JS code", manually writing it in my local JS console? Or is this just "Click the green button in the UI" but expressed in the way only a LLM would write it?
Probably by clicking the button that says run. It is not a weird way to say what they did.
I don't see such a button at all, all I see is text/image pills, samples and a "Commit" button, which the text correctly mentioned, but the other things the text mentions, don't seem to actually exist?
It is weird, a human wouldn't write such text on that page, or if they did, make the text reference actual UI elements, not what they talked about while building it.
There's two sections, an Image one and a Text one. The image one doesn't have a Run button, maybe that's where you are getting confused? The text tab does have it, so you can Run the code shown in the box. The button is at the top right of the element with the code
Just to point out your site text appears too small on mobile (no mobile view) and it doesn't have a direct link to the repo. If either is by choice then that's fine of course.
and what are the savings in this case? the size of the decompressor must be included with the size of the compressed data. and "i lossily compressed a few kilobytes into a couple of gigabytes" sounds a lot less impressive
I tried it and is not working remotely as close as the demo gif. It keeps telling me that "test1" folder that I want to create already exists (not true), then letting me do "mkdier test2" (sic) but then the folder is not there when ls.
I mean all its expected, this has no state, follow no rules and is making up things from the current context, but is not as "magical" as I thought when watching the demo.
No, the model is simulating the entire TUI, not just terminal commands. It also actually is that slow (because it's generating a diff for the entire screen each time you make a keypress).
i agree absolutely with you. and this kind of reinforce my idea that, like this predicted terminal could not be considered a real terminal, predicted intelligence could not be considered real intelligence
Comparing a deterministic system with a non-deterministic one is relatively easier, especially if we know how the deterministic system works. I don't think you can use the same reasoning to extend it to compare two systems we have no idea of how they work.
Although finding out how they work internally might be easier now than initially thought? [0]
Thats an very interesting way to think about this, never thought to invert it that way.
I allways found it interesting to look at this issue in the way the imitation game was modeled. For me I concluded, that as long as we can replicate the intution we associate with intelligence, there is not much room left to debate about "real intelligence".
The chinese room experiment I find in that sense a valuable extension, cuz the turing model more or less is a very conversational "voided of action" kind of test. The room model adds the act of translating without having the knowledge of translation.
But your take invalidates my stances here clearly. There is a possible scenario, in which we can replicate the intuition for operating a real terminal but still keeping the act of the terminal itself a "voided" sort of action by itself, cuz well, its not a real terminal, reading real files and so on...
Thank you for this, I will have to chew on that :)
thanks for the great comment! You answer is great too, I will probably think about it for the next 4 nights or more, triggering my wife's comment "why are you so weird and distant? are you cheating on me?"
And for me this is the reason we need to learn stuff several times. In layers.
The first time we both improvise and memorise. But we have no idea of the underlying complexity of the edge cases. This matches what the imitation game does.
Then we practice and practice, and we start to understand, to really understand what's going on. This is in parallel with learning to deal with the edge cases.
Then, many years later, we learn everything again, but based on an internal world model that can predict and explain just everything about what we are learning. And that's when we actually learn. After we already know it :)
Like with many things, it depends entirely on how you define "real terminal"; that is, what a "real terminal" must do before it satisfies your definition. If it's "takes input and deterministically executes its instructions, producing output along with side effects," a well-trained model with argmax sampling could well meet the bar. if given the appropriate external interfaces.
generally i expect if i invoke `touch file` to see my file update date as now, on the filesystem. so i want to have a mapping between physical world (in this case, memory on my machine) and intelligence representation. If the LLM terminal, upon invoking `touch file` does not update the physical file, it is not a real terminal. you can argue that i will use the terminal itself to invoke `ls -la` and check the updated file, but verifying from an external scope is mandatory imho otherwise it's obvious you cannot logically demonstrate that. hope i was clear
One of the first very cool demos I've seen of ChatGPT was in a post where the author prompted AI "pretend to be a terminal". Then went on to execute commands like `cd .fbi`, `ls files`, `cat files/confidential.txt`. Very fun demo. Looking forward to exploring similar idea with this terminal
I think this was right at the beginning, in the GPT-3.5 era still.
I always wanted to put it behind an open SSH server as a "fake" honeypot of sorts, see what attackers and script kiddies do with it. But I guess by now someone's tried that already..
The craziest thing with that is you could "install" lynx browser and "browse" the alternate universe web within this terminal. At the time -- the whole chat bot and LLM thing was very new-- it seemed crazy and impossible that that kind of behavior was possible from "just some next token prediction"! Like, if you "created" a file on this terminal and it was saved , or downloaded a webpage etc. It sucked when they put in some guardrails and the terminal was no more
202 comments
[ 3.0 ms ] story [ 35.6 ms ] threadhttps://www.youtube.com/watch?v=z3pV6FHvcgM
Edit: Maybe this was it—https://youtube.com/watch?v=7NfyZhV1dKM
Not unlike that American tractor pulling, but for geeks.
Clouds of smoke, weeeeee!
https://www.microsoft.com/en-us/research/articles/whamm-real...
The tmux thing had me confused for a moment until i realized it was emulating tmux. Is the model window width aware?
``` BOS SCREEN_START 32 96 ```
I trained it on three different resolutions, but it should kind of work with in between ones too.
Time is an input too. If nothing is pressed, the model gets a WAIT token with a duration instead of a key, which is how snake keeps moving on its own.
Why would you use a GPU and gigabytes of VRAM running a non-deterministic process for something a program can do on the CPU in milliseconds with kilobytes of memory?
Edit: oh, I saw you wrote a related comment after I posted this. So you already know that direct LLMs for UIs are all but inevitable...
It’s cheaper to hire developers, and quicker to ship software. The user experience be damned.
Of course they would pass that along to the consumer, but nobody is going to pay token-based pricing for something that used to be free and ad-supported, or a cheap monthly subscription.
The second one it's much, much faster.
Nowadays you hire a developer that can use Claude or Cursor out whatever and they will use any language. There are no expensive Rust developers anymore. Or even JavaScript ones.
If I had to complain about any site/app it'd be datadog... each tab uses like 1gb in firefox.
It's like riding a bike with a tiny bit of resistance in the bearings. You will get used to it, and will probably feel like your bike is fine. But if you replace the hub and remove that friction, you will wonder how you ever tolerated riding a bike that way.
There's a good reason why the UI for Excel, Photoshop and Word has hardly changed and remains consistent for years and that has worked for billions of people.
Many HN readers have forgotten the Windows 8 UI disaster already.
Want to extend the idea to office as a whole? Lync was replaced with Skype, OneNote came and went and then came back again. Groove, infopath, publisher, frontpage, SharePoint, various versions of Outlook, the office shortcut bar, photo editor, picture manager - they've all disappeared (some have come back again or been replaced with something else).
I used office XP and 2000 (blocked at the firewall; I mostly saved files as rtf for personal use) for years (admittedly this is mainly because they loaded in 0.5s on my crappy laptop and PC) until I was forced into upgrading to office 2007 by the need to pass around files for class with someone who complained incessantly about their stylesheets being lost every time I converted the file.
I'm still salty I can't waste my time editing 16x16 (32x32?) 256-color dicks into the toolbar icons i when I'm supposed to be writing essays or setting up autocorrect to quietly put typos into a shared computer (the computer isn't shared anymore, and it's no fun doing it to myself)
Gimp hasn't changed it's UI either, and no one uses it, because it sucks.
Maybe I do, if by "randomness" you also mean the ability to handle unexpected usecases. Imagine running something like:
$ ps "which processes are slowing my PC down??"
Wondering if would make any difference in results, which could be impactful given your observation that "Self-correction was one of the more effective training changes"
Interesting. I wanted to ask, how that model compares with the fine-tuned version, but I just saw the author provided benchmarks. Interesting that fine-tuning improved the memory so significantly, wouldn't have expected that.
Though, on second thought, I had an experience where I was able to, as a human, fine-tune my memory beyond what I previously thought was possible!
Reminds me of one of my favorite jokes.
Excuse me, can you tell me how to get to Carnegie Hall?
—Practice, man... Practice...
Please elaborate.
Joshua Foer did some articles where he was looking into how people do the memorization tricks, ultimately concluded that most anyone can do it, and he actually went on to win the US memory competition after a year of practice. He wrote several articles and a book about it I think, so I'm not sure which one I originally read about it.
The guy tells me to rewind. You can just rewind the conversation. Put in a little effort and run it backwards.
That's a skill and apparently I'd never practiced it before.
After a few weeks, I was being complimented for my "excellent memory", while I was high (as someone who naturally had a poor memory even while sober).
--
Also worth mentioning, I met a guy once who was much like me, he'd go on like 5 different tangents... but then he did something spectacular, he'd tie them all together into one glorious point.
I assumed that was because he was some kind of genius, but now I'm wondering if that's learnable as well.
On a very similar theme, I recently built a “lossy git”, which stores your code by training a neural network, then it generates the code back to you on checkout
* web demo: https://nicobrenner.github.io/lossy-git/demo.html * cli version: https://nicobrenner.github.io/lossy-git/
The web demo runs entirely on the browser, both the version control and the neural network training/inference. You can commit and checkout both images or code - you can also paste/“upload” your own images, no server requests (everything clears from the page on reload)
Here's the HN discussion of Alex Wang's article: https://news.ycombinator.com/item?id=50122878
Also, I really appreciate the excellent detailed writeup on your project. Curious about the mistake correction capabilities, it's fascinating, did you try breaking it up in 2 models? one that generates one that repairs? Just wondering if would make any difference in results, which could be impactful given your observation that "was one of the more effective training changes"
You could have invented transformers from memory - https://news.ycombinator.com/item?id=50122878 - Oct 2026 (149 comments)
I see a "Commit" button, but I don't see any checkout, I guess that is what happens automatically? Although that panel says "Reconstruction", not "Checkout"... You might want to do a pass after the AI created the initial version, and make it "humanly consistent" or similar, as right now it seems a bit incoherent.
> Run the JS code before and after a commit
How am I suppose to "run the JS code", manually writing it in my local JS console? Or is this just "Click the green button in the UI" but expressed in the way only a LLM would write it?
Probably by clicking the button that says run. It is not a weird way to say what they did.
I don't see such a button at all, all I see is text/image pills, samples and a "Commit" button, which the text correctly mentioned, but the other things the text mentions, don't seem to actually exist?
It is weird, a human wouldn't write such text on that page, or if they did, make the text reference actual UI elements, not what they talked about while building it.
Just to point out your site text appears too small on mobile (no mobile view) and it doesn't have a direct link to the repo. If either is by choice then that's fine of course.
The link to the repo is at the top of the article: https://github.com/nicobrenner/lossy-git
Love it :)
I did it for web apps last year, basically a web server that feeds all requests into an LLM.
https://www.ohad.com/blog/voidware/
>What's the difference between a model that predicts intelligent output, and intelligence itself?
It doesn't directly attack the question, but gives a neatly constrained thought experiment which you can see running.
Although finding out how they work internally might be easier now than initially thought? [0]
[0] https://already-clean.goodfire.com/
I allways found it interesting to look at this issue in the way the imitation game was modeled. For me I concluded, that as long as we can replicate the intution we associate with intelligence, there is not much room left to debate about "real intelligence".
The chinese room experiment I find in that sense a valuable extension, cuz the turing model more or less is a very conversational "voided of action" kind of test. The room model adds the act of translating without having the knowledge of translation.
But your take invalidates my stances here clearly. There is a possible scenario, in which we can replicate the intuition for operating a real terminal but still keeping the act of the terminal itself a "voided" sort of action by itself, cuz well, its not a real terminal, reading real files and so on...
Thank you for this, I will have to chew on that :)
The first time we both improvise and memorise. But we have no idea of the underlying complexity of the edge cases. This matches what the imitation game does.
Then we practice and practice, and we start to understand, to really understand what's going on. This is in parallel with learning to deal with the edge cases.
Then, many years later, we learn everything again, but based on an internal world model that can predict and explain just everything about what we are learning. And that's when we actually learn. After we already know it :)
I always wanted to put it behind an open SSH server as a "fake" honeypot of sorts, see what attackers and script kiddies do with it. But I guess by now someone's tried that already..