I was going to comment how it reminded me of TempleOS and the author should look into that, but the accompanying blog post explains how it was inspired by it https://blog.orhun.dev/introducing-ratty/
- rendering capabilities of this seem like it should also be able to handle 2d well, or am I mistaken? every solution I see for getting high quality 2d images or rasterization in terminal is all pretty bad. Could this do better than other solutions or is there a fundamental limit being hit somewhere?
- What happens with ssh given that this is gpu accelerated?
Really fun project! Dude, I spent the last week implementing Kitty Graphics and Clipboard protocols in ghostty-web in the Canvas render.
Then I added WebGL and WebGPU renderers [1], including support for Kitty.
Then I see this this project on a Monday morning... so now I have to implement Ratty Graphics Protocol?!?! [2].
ETA: I looked into this; Ghostty would need patched to support Ratty since Ghostty-Web now defers APC handling there. It would also require pulling in a 3D engine like three.js or otherwise implementing file parsing, lighting, etc. Finally, since local filenames are part of the protocol, a browser would need some file resolver helper, either to get the data over the APC channel or via a URL.
Glyph rendering in three.js, fully instanced and addressable and positionable instances. Handles tens of millions. Sample app loads up full GitHub repositories in the web in a few seconds.
I have been thinking about this for a while. It's not as crazy as it may sound, especially in light of the other comments making a parallel between terminals and notebooks.
A few thoughts:
1. Linux VTs kind of have this feature already: there is the normal buffer, the alternate buffer (that something like htop would draw on), and an IOctl can change them to/from graphics mode.
2. It makes sense for interactivity. Kitty's graphics protocol is quite useful for static shapes, can be abused for animations, but doesn't really cut it for interactivity (say, pan a graph around). Wayland is designed for this.
3. Wayland would be a good fit: isolate each command from another, let them request buffers, but keep control of where to display them, do not update them when off screen, etc.
4. One downside is that terminals excel for one-shot tasks. What's the purpose of the display when you are done with it? Should you kill the process driving it? Due to this, it may make more sense to delegate more features to the terminal emulator (displaying the 3D model, etc). Or maybe just allow the app to temporarily take over the window.
5. Once you have it up and running, have it talk directly to the direct rendering manager. Your "kmscon" is now your compositor / desktop environment. That's a fun thought! Add some basic terminal features like tabs and tiling, and you've inverted the usual setup.
6. One downside is accessibility. I really like that I can copy-paste any part of the interface for reference, "screenshots", etc. It's good for screen readers, too. You lose these advantages by going to Wayland.
7. Another current terminal limitation is fonts. Power line, yazi & other make use of custom fonts for drawing part of the interface, logos, etc. AFAIK there is no good way to query their availability (which is also an issue for color emoji). Custom fonts or a new protocol could be useful, but client apps could draw it themselves if given a surface (that can already do that with the kitty graphics protocol, mind you)
Obviously I am not seriously considering to make such a terminal emulator, but it would be an interesting experiment (heck, maybe something I should try this "vibe coding" with, since I wouldn't want to spend too much time on it).
This is a great idea. I always wanted KDE konsole to e. g. show images inlined as is. This is possible via magick six:-, but I wanted this to be natively. I want the terminal to be able to work with any data and display it in any way. No need to simulate the 1980s era anymore (except for backwards/legacy support). So great idea here really.
I like this. No reason the terminal should only support text. Data science notebooks show one way the terminal can evolve. Lots of interesting stuff happening in this space, with Kitty probably being the most aggressive innovator here [1]. I'm not sure there is an overall vision, though.
95 comments
[ 2.7 ms ] story [ 84.2 ms ] threadedit: But your spirit lives on ( based on the project:D )
Questions:
- rendering capabilities of this seem like it should also be able to handle 2d well, or am I mistaken? every solution I see for getting high quality 2d images or rasterization in terminal is all pretty bad. Could this do better than other solutions or is there a fundamental limit being hit somewhere?
- What happens with ssh given that this is gpu accelerated?
Then I added WebGL and WebGPU renderers [1], including support for Kitty.
Then I see this this project on a Monday morning... so now I have to implement Ratty Graphics Protocol?!?! [2].
ETA: I looked into this; Ghostty would need patched to support Ratty since Ghostty-Web now defers APC handling there. It would also require pulling in a 3D engine like three.js or otherwise implementing file parsing, lighting, etc. Finally, since local filenames are part of the protocol, a browser would need some file resolver helper, either to get the data over the APC channel or via a URL.
[1] https://github.com/NimbleMarkets/ghostty-web/tree/nm-webgpu
[2] https://github.com/orhun/ratty/blob/main/protocols/graphics....
Glyph rendering in three.js, fully instanced and addressable and positionable instances. Handles tens of millions. Sample app loads up full GitHub repositories in the web in a few seconds.
https://github.com/tikimcfee/glyph3d-js https://ivanlugo.dev/ide
A few thoughts:
1. Linux VTs kind of have this feature already: there is the normal buffer, the alternate buffer (that something like htop would draw on), and an IOctl can change them to/from graphics mode.
2. It makes sense for interactivity. Kitty's graphics protocol is quite useful for static shapes, can be abused for animations, but doesn't really cut it for interactivity (say, pan a graph around). Wayland is designed for this.
3. Wayland would be a good fit: isolate each command from another, let them request buffers, but keep control of where to display them, do not update them when off screen, etc.
4. One downside is that terminals excel for one-shot tasks. What's the purpose of the display when you are done with it? Should you kill the process driving it? Due to this, it may make more sense to delegate more features to the terminal emulator (displaying the 3D model, etc). Or maybe just allow the app to temporarily take over the window.
5. Once you have it up and running, have it talk directly to the direct rendering manager. Your "kmscon" is now your compositor / desktop environment. That's a fun thought! Add some basic terminal features like tabs and tiling, and you've inverted the usual setup.
6. One downside is accessibility. I really like that I can copy-paste any part of the interface for reference, "screenshots", etc. It's good for screen readers, too. You lose these advantages by going to Wayland.
7. Another current terminal limitation is fonts. Power line, yazi & other make use of custom fonts for drawing part of the interface, logos, etc. AFAIK there is no good way to query their availability (which is also an issue for color emoji). Custom fonts or a new protocol could be useful, but client apps could draw it themselves if given a surface (that can already do that with the kitty graphics protocol, mind you)
Obviously I am not seriously considering to make such a terminal emulator, but it would be an interesting experiment (heck, maybe something I should try this "vibe coding" with, since I wouldn't want to spend too much time on it).
It is just another graphical application window on the OS.
Inline graphics from 1981,
https://youtu.be/o4-YnLpLgtk?t=376
[1]: https://sw.kovidgoyal.net/kitty/protocol-extensions/
Terminals on other operating systems that grew up with a framebuffer don't have this limitation.
That had me in stitches.