Great find! I haven't done much work GUIs myself, but I would love to read more about alternative design philosophies in human-centered computing and their downfalls.
Also to clarify, UIElements (Unity's new retained mode UI framework) is available now in 2019.1 for Editor UI. Runtime UI will also be using UIElements in the future.
I think the author is over exaggerating the problem of object creation and destruction in traditional gui frameworks. List/collection views are designed to reuse objects as you scroll. Secondly I think the author is also downplaying the fact that retained GUIs can also cache object rendering. Just as the gpu doesn't have to draw the whole screen when only the cursor is blinking, it also doesn't have to redraw widgets unless their size changes.
Immediate vs retained is a simple case of budgeting against cpu usage or memory usage, and it should be considered in that light. (immediate uses more processing, retained uses more memory)
Thinking about this decision just as a performance one disregards the fact that the code is substantially different. It does seems likely that one way is more intuitive / easier to work with than the other, I wouldn't know which though.
It depends on the probkem younare trying to solve with your GUI, I suppose. Sometimes, it is better to just recreate the whole GUI to adapt to a model change (e.g. the user moved half of the tree nodes somewhere else), sometimes it is easier/faster/... to just update the existing GUI (e.g. update the text of a label inside a complex dialog widget).
It's important to point out why games use immediate mode GUIs:
1. The GUI needs to be overlaid on the game image (OpenGL/DirectX). This is difficult with traditional GUIs like QT.
2. The GUI needs to be updated in sync with the game, again, it's difficult to integrate traditional GUIs event loops into the game loop, especially with stuff like double/triple buffering.
3. The GUI needs to be as fast as possible, games are severely CPU bound.
A retained mode GUI is typically easier to use, convenience is not why people use immediate mode GUIs.
It's worth pointing out that the immediate/retained split doesn't apply only to the GUI - there are retained mode graphical APIs - DirectX used to have one. They are only used in low-demand games, they sacrifice a lot of speed for the convenience of using a retained mode.
I think 1, 2 and maybe 3 could all be done with a traditional retained mode GUI, and indeed there are retained mode GUI systems for games. The big traditional UI frameworks just have not been made to work well with games. Which is fair, game engines are alien environments that may as well be considered their own platforms.
> It's worth pointing out that the immediate/retained split doesn't apply only to the GUI
Indeed. Modern game engines, and in particular almost all 3D games, use what could be described as a “retained mode” system for the rendering of the game itself — the scene graph — so it is interesting that the UI doesn't always match that.
Generic retained mode is dead, but creating a scene graph driven by something other than imperative updates (eg constraint satisfaction via a physics engine) is alive and well. The problem is that you can’t overlay the former very well in the latter, so things like OGL retained mode are never used anymore.
IMGUI is easy to implement so its the easiest to put in a custom game engine.
IMGUI is not particularly fast in practice. Unity's certainly isn't. However, they are simple, which approximates performance for small cases and it also lets you write your own tuned implementation.
Traditionally screen orientations for games were pretty simple so doing layouts in a single pass was feasible. These days, when you want to fit your game on every platform you need to do a lot of layout work. At that point you're just making an incomplete retained gui implementation.
I've written real-time game UIs before so I think I have some relevant experience here.
1. It is very possible to write a retained-mode GUI in a graphics API like DirectX or OpenGL. In fact, a retained GUI would typically wipe immediate GUIs in terms of performance in this context. In immediate mode, the GUI's vertex buffers need to be completely reconstructed from scratch every single frame, which is slow, CPU bound, and cannot be (easily) parallelized. It's like reconstructing the game world every frame -- that would be ludicrous for any non-trivial game.
2. I don't think there would be that much of a difference between the two UI models, since data updates can be dispatched from the event loop. It would be faster, too, because only UI components that need updating could be redrawn. This is far faster than updating the entire UI every single frame.
3. As mentioned earlier, immediate mode GUIs are going to be a lot slower than retained mode, when implemented properly. Immediate mode GUIs put most of the work on the CPU instead of offloading most of the work to the GPU like in the retained model.
I think developers that are using immediate mode GUIs are doing so because of their ease of use. I think retained mode is typically harder for a game developer to conceptualize because immediate mode is conceptually similar to a game loop. Also, I don't know of any free & open source retained mode GUIs for DirectX and OpenGL and the like.
Also, DirectX at least (and probably OpenGL) encourages a retained-like model for general rendering. The only way to get decent performance is to re-use vertex buffers between frames and only update them when something changes.
I didn't say that it's not possible to solve these problems in a retained GUI, just that existing ones, QT, Win32, WPF have these problems.
And since writing a full retained GUI is not exactly trivial, people just wrote mini-GUIs using immediate mode.
When I talked about retained mode APIs, I was thinking about scene graphs, where you say "addMesh" or "addSphere" and then just call "renderFrame". I'm aware that most game engines implement their own scene graph anyway, but it's game specific, not some generic one provided by the OpenGL/DX API.
Sorry if I misinterpreted your post but I don't think that changes my response much.
You can absolutely use Win32 Forms and WPF (and probably Qt) with a game. They can be overlaid on top of a DirectX or OpenGL window (with a transparent background) -- I've done it before! I don't think it would have any of the downsides you mentioned, either, except that it wouldn't be GPU accelerated or actually rendered inside the graphics context, which is why nobody actually does this in practice.
But that ignores the dozens of GUI middleware specifically designed for games. A cursory Google search reveals that most of these are going to be retained mode. There's a reason for that. Projects like ImGUI appeal to mostly indie devs who don't have the time or resources to write their own GUI library or license some third-party middleware. And it's probably going to be just fine for their use case. But it's definitely not a perfect solution and we definitely shouldn't throw away decades worth of knowledge and experience like the article is implying.
Qt's QML uses OpenGL and can at least render into an FBO in your existing OpenGL context more or less out of the box. Integrating it with a game render loop deeper should be possible too, but more effort.
Sure, for Qt 6, so at some not-yet known point in the next few years (I doubt they'll get that done next year). Whereas both the FBO method or a custom renderer are things you can do today.
I've written real-time game UIs too. I think you underestimate just how ludicrously fast CPUs and GPUs are, and overestimate the complexity of your average GUI. What does your average screen's-worth of GUI consist of, after all? How many widgets are there? I double dare you to tell me that a modern computer or games console can't handle 500 widgets per frame. And I now triple dare you to tell me that your UI designer has put that many damn widgets on one stupid screen in the first place.
(Every UI I worked on actually did redraw everything every frame anyway. It's really not a big deal. Your average GPU can draw a monstrous amount of stuff, something quite ridiculous, and any sensible game UI that's actually usable will struggle to get anywhere near that limit.)
I'm sure many games can get away very well with an immediate mode GUI. I think the question is not can you, but rather should you. My last project used a custom immediate-mode GUI. At the absolute pinnacle of optimization, it was pushing 2,000+ FPS on my machine with something like 3-4k vertices, with heavy texture mapping and anti-aliasing. But the problem was that even with peak optimization, the CPU was spending 15-20% of its time every frame recreating the UI's vertex buffer. Now imagine if we had done a retained-mode GUI instead. That 15-20% overhead would be reduced to near 0% on a typical frame. For nearly any type of game, that kind of savings is really significant. Think of how many more vertices your artists can add, or cool gameplay elements you can add that you didn't have the CPU time available before, and how much better it will run on lower-end hardware.
I don't understand the problem. 2000Hz = 0.5ms/frame; 20% of this = 0.1ms. That sounds like a great result. Your target frame rate is presumably 60-100Hz, assuming it's a PC game, meaning your frame budget is 10-16ms. If your UI takes 0.1ms, you've got >99% of your budget left.
(Also: 3-4K vertices for UI was about what you could expect to budget for a PS2 or Xbox game! - max throughput for PS2 was something like 250,000 vertices/frame at 60Hz, and this is <2% of that. I struggle to believe this is any kind of an issue for anything modern.)
I should mention that 2000 Hz when only running the UI -- with the full game running at 150 FPS. So really the CPU time is 6.66ms*0.2 = 1.33 ms just for the UI!
Of course, that's on my beefy machine with a octo-core overclocked CPU and a couple 1080Tis. But what about the players who are trying to play on their mobile CPUs with integrated graphics? That margin could be the difference between 45 and 60 FPS.
This is wrong, just because it uses 20% CPU when rendering at 2000Hz doesn't mean it will consume 20% CPU when running the game too. Running the game alongside the UI is what makes it drop to 150FPS from 2000FPS. So if the UI consumes the 20% of the frame time at 2000FPS, that is indeed 0.1ms and this number will not magically increase when you add the game. So when you're running at 150FPS your frame time is 6.66ms and you're using 0.1 for the UI. That seems pretty good for me to be honest.
Basically yes. If an immediate mode API is much easier to use, and a retained mode underlying implementation has much better performance, then putting a React-style VDOM layer in-between could get the best of both worlds, depending on how well the middle layer is implemented.
The catch is that games follow the law of "MOAR!": more details, more effects, more postprocessing, mpre abimation, more simulation... this means that the engine will almost always be pushed to max out the hardware long before the designers are happy with what they have.
Heck, I can think of pretty easy ways to improve sound and video quality of games in ways that are quite fundamental, but that would easily max out the best gaming rigs with either sound or graphics alone.
Bottom line: the need to optimize always comes sooner than you would expect.
Performance is not the only consideration, even in most demanding games. IM GUIs simplify UI code a lot, by not having to rely on message systems that make the code harder to follow and less predictable.
As most things in life, it's a matter of trade-offs.
Every game I've worked on didn't actually use an IMGUI for anything you'd ever show a player; (unless it's a style of game that really doesn't need much UI), it's just custom gui implementations.
A lot of modern games actually put the UI in with the scene graph of the game itself (IE: Unity's new GUI system)
This is definitely the way to go, especially with the more recent trend of embedding UI elements in world space, and VR/AR essentially requiring it.
Internal tools tend to be IMGUI, Winforms, or WPF from what I've seen. Essentially whatever the original author is most productive in. No need to be super performant because your only users are other devs with really high end systems and rarely an actual need for a constant 30/60fps.
Your engine likely already has a graph structure, event system, and input handling. The core pieces of a retained mode UI system are already there.
> You have to write lots of code to manage the the creation and destruction of GUI objects. [..]
> The creation and destruction problem leads to slow unresponsive UIs [..]
> You have to marshal your data into and out of the widgets. [..]
My biggest pain point with the retained mode GUIs I worked with was none of the issues mentioned above. It was always the centralized GUI thread and the consequential synchronization complications. I don't know if this is an inherent problem of retained mode GUI frameworks and if there are some that don't force all widgets into a single thread. If not, this alone is a reason to for me to find immediate mode interesting.
Immediate mode, if anything, makes this harder. (You absolutely have to run the GUI on one thread, for instance). Really though you can get around that on either of them by spawning a worker thread/coroutine/etc. on button clicks and so on.
The real problems start, when the worker thread needs to update the GUI, for example to advance a progress bar. With the frameworks I know I have to build a communication channel between the threads and tell the GUI thread to show and update the progress bar.
My hope was that with an immediate mode framework I could just show a progress bar on top of the existing GUI right from the worker thread. I don't know enough about immediate mode to say if this is really possible. It would simplify a lot of my code though.
This is really reads like someone trying to sell you something. I've done work on frameworks for both immediate mode and retained mode GUIs. They both absolutely allocate memory behind the scene. There absolutely is state being marshaled around. Caching commonly used state is important. Performance can be bad and great in both. You're really just subscribing to different sets of opinions
It will. In fact, even a simple one will require this. But it's not much of a problem - 99% of the time, the UI is the same from one frame to the next, and it isn't much work to detect this. So even if every change requires a complete rebuild of everything, it's not much of a problem.
(I've written systems like this for Cocoa and Win32, and it never turned out to be necessary to do anything other than just regenerate the entire UI any time anything changed. The update runs at 30Hz or 60Hz, and when anything is changing, the UI gets regenerated a lot! - but so what? Most of the time, the UI doesn't get regenerated at all. Then something happens, and the code spends 2 seconds getting absolutely hammered continuously, malloc malloc malloc malloc malloc, god help us all... and then, once again, nothing. The operator puts their fingers to one side and stares at the result with their eyes. Repeat.)
I recall reading this article in your comments on the last GUI-specific link posted here... where you just kept disagreeing with comments that took the time to point out how this stuff is largely off base.
"More research into ImGUI style UIs could lead to huge gains in productivity."
Don't tell this cat that the research on this stuff goes back >40 years and that the introductory chapter of any book on computer graphics would have talked about all of this. Not like it would help him - he hasn't read anything about it, didn't even do a cursory Google search. Sheesh. A low, low bar.
One of my favorite is "Don't Fidget with Widgets, Draw!" (https://www.hpl.hp.com/techreports/Compaq-DEC/WRL-91-6.pdf) It's modern enough to be understandable, and while it's referencing Ezd (a Scheme drawing system) it greatly influenced Tk (which is still used in all kinds of heavy-hitting EDA software).
That one's only been around for 28 years though (well, the paper was published in 1991, so code was before that...) but let's go further:
Which if you haven't watched MOAD before, give it a spin. It will still blow your mind, but the interactive graphical drawing mechanisms will be recognizable.
Also, this "paint the screen every time" method is how a tremendous number of people who cut their teeth on DOS did things on the screen. DOS release date: 1981. So you don't have to have been an academic (which I am NOT) to have tried these techniques while solving practical problems.
As far as books:
Computer Graphics
Principles and Practice in C (later editions use C#/C++/etc. - the ideas are the same)
The 1995/96 one talks about retained mode graphics directly. That book has been standard in "intro computer graphics" courses for as long as it's been out. So at least 20+ years it's been the "start here" book.
Recently I started playing with https://github.com/ajnsit/concur-documentation/blob/master/R... which has been the most refreshing UI paradigm i've used in a while. and it reminds me a lot of this ImGUI approach, but behind the scenes it uses coroutines instead.
The idea is that a button is a UI element that _blocks_ until an event is fired. You can then compose elements in time like:
button "hey" >> label "clicked"
which is a program that displays a button, you click it, the button goes away and the text "clicked appears"
Or you can compose programs in space:
(button "hey" <> label "not clicked")
this is a program that displays both a button, and a label at the same time.
Now, by combining both space and time, we can create a program that changes the label when clicking as follows:
program = (button "toggle" <> label "off") >> (button "toggle" <> label "on") >> program
This is an application that toggles a label on and off when you press a button. (Note that the definition is recursive)
In the context of creating debugging UIs for games and graphics applications, Dear imGUI is a godsend. Programmers love it because there is literally only the code to worry about. It's very easy to get it up and running, and all the code that handles the UI drawing and interaction is in one place so it's easy to reason about.
It works very well in the context where you already have fast graphics and an update loop, and you're already expecting to redraw the whole screen every frame. It does not really suit more complex, text-heavy UIs where you're rendering thousands of glyphs with proper kerning and ligatures and anti-aliasing, etc, and want the result of that hard work to be retained in the framebuffer unless it absolutely needs to change.
> want the result of that hard work to be retained in the framebuffer unless it absolutely needs to change
I think immediate mode GUI libraries can get around this issue by still caching and reusing between frames. Conrod does this by still having the state in the background although you are programming to an immediate mode API:
There's a difference bt poor impl and poor design.
As the author points out, HTML has a poor design (eg. if you want to have a 1000x1000 cell table, you have to have 10^6 actual cells - that's a lot of tds or whatever to parse).
Modern OO GUI frameworks don't do this - they say something like:
cellRenderer.draw(target, row,col,position,size)
No creation of objects required. Of course since it's so easy to create OO programs a lot of code isn't great... and then others copy that code and so it goes.
Seems like we keep re-creating software b/c we haven't taken the time to look at what exists and only then decide on what to keep and what to change. "This is too complex - I'll re-write it!". 10 years later: "We added all the features that the existing software had and now the new one... is just as slow... but we did sell a lot of conference tickets and books so... totally worth it."
When I was 20 I also thought I knew better so I get it.
Conrod is an immediate mode GUI library for rust[1]. I've been using it for an image processing app[2] and have enjoyed the way the code turns out. Everything is much more straightforward as you don't have to reason about callbacks interacting with a loop that's not yours to control and the performance seems good.
The author does a good job explaining some benefits of an immediate mode renderer but vastly misses the disadvantages.
The immediate mode renderer is great for toy programs. Similar to how you could reproduce 'look here is how simple it is to write hello world and compute the millionth digit of PI' in a new esoteric language...
Occlusion, hit-testing, state changes, scrolling/animations even in the simplest forms will fall over. Infact, that's why we have every major browser move their touch and animation systems into a layer based compositor system (into a separate thread / process).
The author also grossly misses their own example of 'how a spreadsheet with many rows and columns will update faster using ImgUI' and how Instagram styled apps will fare better ImgUi.
A retained mode renderer will use virtual scrollers, efficient culling of nodes for both display and hit-testing (scale to billions of virtual cells) and more importantly help a team of people coordinate their work and get things done.
In fact, with Javascript JIT engines (whose teams deeply understand the use of libraries like React) relentlessly attacking the overhead of DOM nodes and their initialization, and with users who actually expect the design flexibility provided by CSS... a system like React is actually an ideal layer of abstraction for modern UI implementation on practically any platform.
Sure, if you're on an embedded platform, somewhere a JIT can't run, or if you're doing something with real-time rendering requirements (and honestly modern React Suspense should even make that feasible), you may want to use something lower-level. But most people won't need to do this.
The DOM being slow is really more an artifact of browsers/HTML and the history behind doing document based layout. React is a nice solution to that specific problem; but this article suggests immediate mode gui's are the future, which, I don't think that's really the case. And I'm not sure I'd really call React an IMGUI anyway. It's similar, but it's also pretty different.
> A retained mode renderer will use virtual scrollers, efficient culling of nodes for both display and hit-testing (scale to billions of virtual cells)
A good IMGUI will do this too (e.g. Dear IMGUI already does)
The problem with ImGUI is that your gui tree is function call tree and you can't do much with that unless you are in lisp.
Function call tree has some advantages. Functions are wonderfully composable and flexible.
React is such a revolution because it translates ImGUI usage into whatever insane retained mode API is at the bottom of the stack, through intermediate representation of virtual dom.
Presto. You have flexibility and composability of ImGUI without disadvantages of keeping your gui tree only in your function calls, and you can use it without ability to directly control how the GUI is drawn in the engine.
Many of authors most significant criticisms on retained GUIs are implementation considerations. GUI frameworks exist that solve his key criticisms of complexity and are pleasant to work with.
Criticisms that target core architecture of retained GUI I don't consider to be valuable design goals, at least in settings where I work on GUIs. e.g. memory usage.
Alot of things are glossed over that remain challenges in both, e.g. layout management.
HTML is an interesting example. First iteration of HTML was essentially immediate mode if you think about a single client/server interaction as a GUI update cycle. Server sends draws to client browser and client browser sends back to server user selections. There is no retained state on gui side. Now with programmatic access to DOM, ability to attach events to DOM elements from client side it is now a retained GUI. Seems to be where things evolve to naturally.
The GUI framework I use nearly daily is retained and very pleasant to work with in terms of ease of belting out screens & readability/maintainability of code. The simplicity comes with compromises though as there are limits on GUI idioms that can be expressed. Occasionally run into those boundaries and resulting GUI does look a little plain and unexciting, but for something that is substantially about data entry its fine.
Really probably not. I love them and they're very handy in certain situations (debugging tools, quick UIs) but once you need a lot of customization they become extremely cumbersome. Also really kind of makes it impossible for designers or less technical people to do anything. Also, while I think separating view logic is slightly overrated, it is useful, and it's very hard to do that with IMGUI. Also it doesn't thread well. Also... Look there's like a million downsides.
Also saying there's no memory allocation is really misleading. There's PLENTY of memory allocation, per frame, you're just not explicitly doing it yourself. It's actually much worse than an RMGUI in this regard, because at least with an RMGUI you get the allocations over with once. With an IMGUI you're allocating things all the time. They're probably much smaller allocations, but lots of small allocations does not make for good performance.
One final note, the Unity 3D example always gets used. If you've ever written a plugin for unity or a custom editor, you're very familiar with the fact that it's editor gui system is extremely limiting and kind of sucks. I mean, it's an example, but once you're past the basics it's kind of a bad example.
On the allocation point, the efficient way to handle this is to use a per-frame memory pool. Because nothing in the IMGUI can persist between frames, you can allocate a single arena of memory for any UI elements that need to store bounding boxes or callbacks or whatever. Each frame just reset your next_ptr to the start of the arena. Technically you are allocating memory, but in practice your allocations are free.
Adding a separate complex ad hoc memory allocation scheme is not what I would call free. Granted, computationally-wise it may be relatively cheap but it does add multiple forms of complexity to a problem that doesn't exist in rmguis.
// Start of frame
void *arena = malloc(LOTS_OF_MEMORY);
void *next_ptr = arena;
// Allocate something
object = *next_ptr; // return that value
next_ptr = object_size + alignment; // crash if out of memory
// End of frame
free(arena);
By the way, such "separate complex ad hoc" memory allocation schemes are the reason why manual memory management is faster than garbage collection. If you did everything with malloc(), it would be slower (unless the GC language allocates much more than C, which they often do).
Nope, just set the pointer back to where it started from. You do need to be super careful doing this though, as anything that relies on RAII (in c++ land) will be busted. You could manually call the destructor on the object in that case, but kind of defeats the purpose of the "no allocation" goals
C++ includes it's "placement new" feature specifically to cater to the memory pool needs. There's no allocation, and only constructor and destructor calls.
Specifically Unity would have been much better served using something like Electron. Unity's UI looks bad, feels bad, and is a huge pain in the ass if you're authoring extensions.
Unity launched in 2005. Electron released in 2013. Still, saying Unity would have been "better served" by using Electron is more than a little unreasonable!
Using a primarily JavaScript environment for primarily C# content sounds like an absolute nightmare to me.
Weirdly, when Unity was first starting out it had this "UnityScript" thing, which was javascript-but-not-quite. Luckily it's mostly been phased out (I don't think newer versions even support it)
I have seem other programs doing stuff like that before, and I have also done some of that in my own programming (although not with this or any other library). I did not know what it is called, until I read this today. It look like good to me. Also, you will still need to add some extra variables if you are doing such thing as tab to focus, I think.
I feel that the future lies in combining retained and immediate interfaces, preferably with the granularity to allow deeply nested retained interfaces that are fast for complex ui (or for a realtime system with memory to spare), whilst still allowing one to go the other direction, such that a simple ui can be written cleanly and logically for low-memory systems (such as embedded or boot guis). It would need a very well designed api for this, but I feel the benefits are worth the effort (and I'll probably look into this next time I have the freedom to choose ui apis).
A balance may be letting people define it either way, so that manually written ui still can have auto-layout yet intuitive code (following control flow primitives), whilst allowing generated retained uis to be manually editable -- perhaps even allowing one to then embed one within the other, a boon for scripted interfaces that perhaps have people of various levels of experience producing ui elements, such as a musician with little experience being able to add a simple visualiser in an immediate manner to a deeply retained daw gui.
Of course, there's a lot here that is implementation, and some criticism either way can be optimised out. Immediate mode can still cache its rendering, we've had optimised blitting since the early days, and is only usually a problem with complex ui. Retained would get fewer cache misses if we weren't allocating madly across the heap and took a more disciplined approach allocating to contiguous memory -- which is almost entirely a language/api problem (in my experience) that can also happen with immediate but we typically don't see since it's often done in a more procedural style that is allocating to some pool.
Other api elements, such as handling lists etc aren't really a differentiation between retained and immediate, those can be made in either.
For me, I often find that the ability to write out prototype ui code in an immediate style in very quick and satisfying (exactly what I want in prototyping), however once I start to expand upon a ui, I find it best to over time refactor towards a retained style, since by then I will typically have some templates for what ui elements look like, and so I just have to pass a string and function pointer to fill in the template.
Can't see why we can't have nice things and let both coexist...
Anyone experienced with ImGUI ever use Rebol and Red's DRAW DSL?
I believe Rebol's GUI support is even easier to use than ImGUI, but of course it can't be embedded and used in the same way as ImGUI either. I wonder if non Red projects could possibly hook into Red's system once Red/System gets closer to C level performance?
Probably not. With technologies like React that make retained-mode UIs look more like immediate-mode ones, there is less need for full blown immediate-mode UIs. React achieves the programmability of an immediate-mode UI without sacrificing the performance of a retained-mode UI (at least, that’s the goal).
Say I want to make a program, X, that draws a 50x50px square over another program, Y, at (200,100) and only program Y. Do i need a low level stuff for this, can imGUI be used here? or is it possible with electronjs etc... Also, would program Y be able to detect, with administrator privileges that another program was targeting its pixel space and drawing over it?
I'm having a hard time to understand what is ImGUI (and what is the opposite RmGUI)... any could help me with ELI5? It sounds like ImGUI is reactive while RmGUI is not?
Yeah I'd appreciate an ELI5 source for this also. The article mostly assumes that you already understand the concept and the linked video is incredibly dry. Some visuals alongside the code would be nice.
Games (normally) re-render the whole scene every frame.
ImGUI exposes that to their API users: you have to re-render and check for clicks on every frame. The code looks like React, (but not as optimized, it re-renders every frame!), and normally you have to keep state by yourself. Code example: [1].
Retained Mode is closer to the DOM, Cocoa or WPF: you create objects and there's an abstraction between the API and the renderer: they get re-rendered every frame for you. Componentes normally have events and state by themselves. Sometimes there's a visual editor too.
The main difference is the API. One is lower level than the other. In practice, the APIs aren't that different, except when it comes to event handling.
> Games (normally) re-render the whole scene every frame
They don't re-upload the whole scene to the GPU every frame, they don't recreate objects all the time. Immediate mode was only used in, like, the GL 1.x times.
A retained mode GUI is defined by writing initialization code that sets up a model of the UI elements in memory, attaching data and linking callbacks to interactive elements like buttons. The UI library uses the model in memory to draw the UI and handle user interactions.
An immediate mode GUI is defined by writing an update function that draws the UI, passing in data and conditionally running callbacks for interactive elements like buttons. There is no model saved in memory, rather the structure of the UI is implied through the code path taken by the update function. The UI library uses this function to both draw the UI and handle user interactions.
It comes from 3D gaming and hardware acceleration.
In 3D you use hardware acceleration that is 100-200 times more energy efficient than drawing in the CPU. But you loose flexibility.
It is actually way cheaper to actually clear the screen and redraw it again each 1/60 of a second than to complicate the design of the drawing.
If you make it complex, the GPU could not draw it, because the GPU is not flexible, so only the CPU could draw it.
With GPUs, those tricks do not make sense at all.
In the past, there was a lot of optimization for things like drawing windows on the screen, things like circular buffers, "happy ideas" everywhere that made it efficient in the CPU at the expense of complexity, that the CPU could handle.
Imagine solving a mathematical equation. With matrices we fill a table with zeros that represents the elements of the table that do not exist. This is inefficient, but with a matrix a machine can solve it automatically without "thinking".
Immediate Gui uses the same concept, it draws without caring for last frames or states, making it way simpler.
127 comments
[ 2.4 ms ] story [ 196 ms ] threadImmediate vs retained is a simple case of budgeting against cpu usage or memory usage, and it should be considered in that light. (immediate uses more processing, retained uses more memory)
1. The GUI needs to be overlaid on the game image (OpenGL/DirectX). This is difficult with traditional GUIs like QT.
2. The GUI needs to be updated in sync with the game, again, it's difficult to integrate traditional GUIs event loops into the game loop, especially with stuff like double/triple buffering.
3. The GUI needs to be as fast as possible, games are severely CPU bound.
A retained mode GUI is typically easier to use, convenience is not why people use immediate mode GUIs.
It's worth pointing out that the immediate/retained split doesn't apply only to the GUI - there are retained mode graphical APIs - DirectX used to have one. They are only used in low-demand games, they sacrifice a lot of speed for the convenience of using a retained mode.
> It's worth pointing out that the immediate/retained split doesn't apply only to the GUI
Indeed. Modern game engines, and in particular almost all 3D games, use what could be described as a “retained mode” system for the rendering of the game itself — the scene graph — so it is interesting that the UI doesn't always match that.
IMGUI is easy to implement so its the easiest to put in a custom game engine.
IMGUI is not particularly fast in practice. Unity's certainly isn't. However, they are simple, which approximates performance for small cases and it also lets you write your own tuned implementation.
Traditionally screen orientations for games were pretty simple so doing layouts in a single pass was feasible. These days, when you want to fit your game on every platform you need to do a lot of layout work. At that point you're just making an incomplete retained gui implementation.
1. It is very possible to write a retained-mode GUI in a graphics API like DirectX or OpenGL. In fact, a retained GUI would typically wipe immediate GUIs in terms of performance in this context. In immediate mode, the GUI's vertex buffers need to be completely reconstructed from scratch every single frame, which is slow, CPU bound, and cannot be (easily) parallelized. It's like reconstructing the game world every frame -- that would be ludicrous for any non-trivial game.
2. I don't think there would be that much of a difference between the two UI models, since data updates can be dispatched from the event loop. It would be faster, too, because only UI components that need updating could be redrawn. This is far faster than updating the entire UI every single frame.
3. As mentioned earlier, immediate mode GUIs are going to be a lot slower than retained mode, when implemented properly. Immediate mode GUIs put most of the work on the CPU instead of offloading most of the work to the GPU like in the retained model.
I think developers that are using immediate mode GUIs are doing so because of their ease of use. I think retained mode is typically harder for a game developer to conceptualize because immediate mode is conceptually similar to a game loop. Also, I don't know of any free & open source retained mode GUIs for DirectX and OpenGL and the like.
Also, DirectX at least (and probably OpenGL) encourages a retained-like model for general rendering. The only way to get decent performance is to re-use vertex buffers between frames and only update them when something changes.
And since writing a full retained GUI is not exactly trivial, people just wrote mini-GUIs using immediate mode.
When I talked about retained mode APIs, I was thinking about scene graphs, where you say "addMesh" or "addSphere" and then just call "renderFrame". I'm aware that most game engines implement their own scene graph anyway, but it's game specific, not some generic one provided by the OpenGL/DX API.
You can absolutely use Win32 Forms and WPF (and probably Qt) with a game. They can be overlaid on top of a DirectX or OpenGL window (with a transparent background) -- I've done it before! I don't think it would have any of the downsides you mentioned, either, except that it wouldn't be GPU accelerated or actually rendered inside the graphics context, which is why nobody actually does this in practice.
But that ignores the dozens of GUI middleware specifically designed for games. A cursory Google search reveals that most of these are going to be retained mode. There's a reason for that. Projects like ImGUI appeal to mostly indie devs who don't have the time or resources to write their own GUI library or license some third-party middleware. And it's probably going to be just fine for their use case. But it's definitely not a perfect solution and we definitely shouldn't throw away decades worth of knowledge and experience like the article is implying.
(Every UI I worked on actually did redraw everything every frame anyway. It's really not a big deal. Your average GPU can draw a monstrous amount of stuff, something quite ridiculous, and any sensible game UI that's actually usable will struggle to get anywhere near that limit.)
Why settle for "good-enough" performance?
(Also: 3-4K vertices for UI was about what you could expect to budget for a PS2 or Xbox game! - max throughput for PS2 was something like 250,000 vertices/frame at 60Hz, and this is <2% of that. I struggle to believe this is any kind of an issue for anything modern.)
Of course, that's on my beefy machine with a octo-core overclocked CPU and a couple 1080Tis. But what about the players who are trying to play on their mobile CPUs with integrated graphics? That margin could be the difference between 45 and 60 FPS.
Not saying it is easy, but it's possible to optimize and cache vertex buffers by using something similar to React's VDOM.
Because its good enough? Perfection is the enemy good.
I can't see 2000 FPS, 60 FPS is good enough for me. Similarly the 20% vertex buffer hit might not matter.
Heck, I can think of pretty easy ways to improve sound and video quality of games in ways that are quite fundamental, but that would easily max out the best gaming rigs with either sound or graphics alone.
Bottom line: the need to optimize always comes sooner than you would expect.
As most things in life, it's a matter of trade-offs.
A lot of modern games actually put the UI in with the scene graph of the game itself (IE: Unity's new GUI system)
Internal tools tend to be IMGUI, Winforms, or WPF from what I've seen. Essentially whatever the original author is most productive in. No need to be super performant because your only users are other devs with really high end systems and rarely an actual need for a constant 30/60fps.
Your engine likely already has a graph structure, event system, and input handling. The core pieces of a retained mode UI system are already there.
That's not actually that hard to do. You just need to split the UI on layers.
Here is for example HTML/CSS is rendered below and on top of 3D scene: https://sciter.com/sciter-and-directx/
> You have to write lots of code to manage the the creation and destruction of GUI objects. [..]
> The creation and destruction problem leads to slow unresponsive UIs [..]
> You have to marshal your data into and out of the widgets. [..]
My biggest pain point with the retained mode GUIs I worked with was none of the issues mentioned above. It was always the centralized GUI thread and the consequential synchronization complications. I don't know if this is an inherent problem of retained mode GUI frameworks and if there are some that don't force all widgets into a single thread. If not, this alone is a reason to for me to find immediate mode interesting.
My hope was that with an immediate mode framework I could just show a progress bar on top of the existing GUI right from the worker thread. I don't know enough about immediate mode to say if this is really possible. It would simplify a lot of my code though.
(I've written systems like this for Cocoa and Win32, and it never turned out to be necessary to do anything other than just regenerate the entire UI any time anything changed. The update runs at 30Hz or 60Hz, and when anything is changing, the UI gets regenerated a lot! - but so what? Most of the time, the UI doesn't get regenerated at all. Then something happens, and the code spends 2 seconds getting absolutely hammered continuously, malloc malloc malloc malloc malloc, god help us all... and then, once again, nothing. The operator puts their fingers to one side and stares at the result with their eyes. Repeat.)
We moved away from WM_PAINT for a reason.
"More research into ImGUI style UIs could lead to huge gains in productivity."
Don't tell this cat that the research on this stuff goes back >40 years and that the introductory chapter of any book on computer graphics would have talked about all of this. Not like it would help him - he hasn't read anything about it, didn't even do a cursory Google search. Sheesh. A low, low bar.
One of my favorite is "Don't Fidget with Widgets, Draw!" (https://www.hpl.hp.com/techreports/Compaq-DEC/WRL-91-6.pdf) It's modern enough to be understandable, and while it's referencing Ezd (a Scheme drawing system) it greatly influenced Tk (which is still used in all kinds of heavy-hitting EDA software).
That one's only been around for 28 years though (well, the paper was published in 1991, so code was before that...) but let's go further:
The drawing system(s) that greatly influenced Ezd came largely from Xerox PARC, such as ALTO: https://www.computerhistory.org/atchm/xerox-alto-source-code...
There's code in there for a vector drawing program (in 1980!), as well as interacting widgets. Let's go back further...
Finally, we have to mention the Mother of All Demos, in 1968: https://en.wikipedia.org/wiki/The_Mother_of_All_Demos
Which if you haven't watched MOAD before, give it a spin. It will still blow your mind, but the interactive graphical drawing mechanisms will be recognizable.
Also, this "paint the screen every time" method is how a tremendous number of people who cut their teeth on DOS did things on the screen. DOS release date: 1981. So you don't have to have been an academic (which I am NOT) to have tried these techniques while solving practical problems.
As far as books: Computer Graphics Principles and Practice in C (later editions use C#/C++/etc. - the ideas are the same)
The 1995/96 one talks about retained mode graphics directly. That book has been standard in "intro computer graphics" courses for as long as it's been out. So at least 20+ years it's been the "start here" book.
So yea, this stuff has been around for a while...
The idea is that a button is a UI element that _blocks_ until an event is fired. You can then compose elements in time like:
which is a program that displays a button, you click it, the button goes away and the text "clicked appears"Or you can compose programs in space:
this is a program that displays both a button, and a label at the same time.Now, by combining both space and time, we can create a program that changes the label when clicking as follows:
This is an application that toggles a label on and off when you press a button. (Note that the definition is recursive)It works very well in the context where you already have fast graphics and an update loop, and you're already expecting to redraw the whole screen every frame. It does not really suit more complex, text-heavy UIs where you're rendering thousands of glyphs with proper kerning and ligatures and anti-aliasing, etc, and want the result of that hard work to be retained in the framebuffer unless it absolutely needs to change.
I think immediate mode GUI libraries can get around this issue by still caching and reusing between frames. Conrod does this by still having the state in the background although you are programming to an immediate mode API:
https://docs.rs/conrod/latest/conrod/guide/chapter_1/index.h...
That said it's a good middle ground. Use whatever API you prefer over a well optimized implementation.
As the author points out, HTML has a poor design (eg. if you want to have a 1000x1000 cell table, you have to have 10^6 actual cells - that's a lot of tds or whatever to parse).
Modern OO GUI frameworks don't do this - they say something like:
cellRenderer.draw(target, row,col,position,size)
No creation of objects required. Of course since it's so easy to create OO programs a lot of code isn't great... and then others copy that code and so it goes.
Seems like we keep re-creating software b/c we haven't taken the time to look at what exists and only then decide on what to keep and what to change. "This is too complex - I'll re-write it!". 10 years later: "We added all the features that the existing software had and now the new one... is just as slow... but we did sell a lot of conference tickets and books so... totally worth it."
When I was 20 I also thought I knew better so I get it.
[1] https://github.com/pistondevelopers/conrod
[2] https://github.com/pedrocr/chimper
The immediate mode renderer is great for toy programs. Similar to how you could reproduce 'look here is how simple it is to write hello world and compute the millionth digit of PI' in a new esoteric language...
Occlusion, hit-testing, state changes, scrolling/animations even in the simplest forms will fall over. Infact, that's why we have every major browser move their touch and animation systems into a layer based compositor system (into a separate thread / process).
The author also grossly misses their own example of 'how a spreadsheet with many rows and columns will update faster using ImgUI' and how Instagram styled apps will fare better ImgUi.
A retained mode renderer will use virtual scrollers, efficient culling of nodes for both display and hit-testing (scale to billions of virtual cells) and more importantly help a team of people coordinate their work and get things done.
We are no longer in the 90s.
Sure, if you're on an embedded platform, somewhere a JIT can't run, or if you're doing something with real-time rendering requirements (and honestly modern React Suspense should even make that feasible), you may want to use something lower-level. But most people won't need to do this.
A good IMGUI will do this too (e.g. Dear IMGUI already does)
Function call tree has some advantages. Functions are wonderfully composable and flexible.
React is such a revolution because it translates ImGUI usage into whatever insane retained mode API is at the bottom of the stack, through intermediate representation of virtual dom.
Presto. You have flexibility and composability of ImGUI without disadvantages of keeping your gui tree only in your function calls, and you can use it without ability to directly control how the GUI is drawn in the engine.
Many of authors most significant criticisms on retained GUIs are implementation considerations. GUI frameworks exist that solve his key criticisms of complexity and are pleasant to work with.
Criticisms that target core architecture of retained GUI I don't consider to be valuable design goals, at least in settings where I work on GUIs. e.g. memory usage.
Alot of things are glossed over that remain challenges in both, e.g. layout management.
HTML is an interesting example. First iteration of HTML was essentially immediate mode if you think about a single client/server interaction as a GUI update cycle. Server sends draws to client browser and client browser sends back to server user selections. There is no retained state on gui side. Now with programmatic access to DOM, ability to attach events to DOM elements from client side it is now a retained GUI. Seems to be where things evolve to naturally.
The GUI framework I use nearly daily is retained and very pleasant to work with in terms of ease of belting out screens & readability/maintainability of code. The simplicity comes with compromises though as there are limits on GUI idioms that can be expressed. Occasionally run into those boundaries and resulting GUI does look a little plain and unexciting, but for something that is substantially about data entry its fine.
Also saying there's no memory allocation is really misleading. There's PLENTY of memory allocation, per frame, you're just not explicitly doing it yourself. It's actually much worse than an RMGUI in this regard, because at least with an RMGUI you get the allocations over with once. With an IMGUI you're allocating things all the time. They're probably much smaller allocations, but lots of small allocations does not make for good performance.
One final note, the Unity 3D example always gets used. If you've ever written a plugin for unity or a custom editor, you're very familiar with the fact that it's editor gui system is extremely limiting and kind of sucks. I mean, it's an example, but once you're past the basics it's kind of a bad example.
Electron is a catastrophic of hog of resources and performance. Electron for mobile game UI would be a disaster!
(Plenty of criticisms of Electron still hold, of course.)
Unity launched in 2005. Electron released in 2013. Still, saying Unity would have been "better served" by using Electron is more than a little unreasonable!
Using a primarily JavaScript environment for primarily C# content sounds like an absolute nightmare to me.
A balance may be letting people define it either way, so that manually written ui still can have auto-layout yet intuitive code (following control flow primitives), whilst allowing generated retained uis to be manually editable -- perhaps even allowing one to then embed one within the other, a boon for scripted interfaces that perhaps have people of various levels of experience producing ui elements, such as a musician with little experience being able to add a simple visualiser in an immediate manner to a deeply retained daw gui.
Of course, there's a lot here that is implementation, and some criticism either way can be optimised out. Immediate mode can still cache its rendering, we've had optimised blitting since the early days, and is only usually a problem with complex ui. Retained would get fewer cache misses if we weren't allocating madly across the heap and took a more disciplined approach allocating to contiguous memory -- which is almost entirely a language/api problem (in my experience) that can also happen with immediate but we typically don't see since it's often done in a more procedural style that is allocating to some pool.
Other api elements, such as handling lists etc aren't really a differentiation between retained and immediate, those can be made in either.
For me, I often find that the ability to write out prototype ui code in an immediate style in very quick and satisfying (exactly what I want in prototyping), however once I start to expand upon a ui, I find it best to over time refactor towards a retained style, since by then I will typically have some templates for what ui elements look like, and so I just have to pass a string and function pointer to fill in the template.
Can't see why we can't have nice things and let both coexist...
I believe Rebol's GUI support is even easier to use than ImGUI, but of course it can't be embedded and used in the same way as ImGUI either. I wonder if non Red projects could possibly hook into Red's system once Red/System gets closer to C level performance?
ImGUI exposes that to their API users: you have to re-render and check for clicks on every frame. The code looks like React, (but not as optimized, it re-renders every frame!), and normally you have to keep state by yourself. Code example: [1].
Retained Mode is closer to the DOM, Cocoa or WPF: you create objects and there's an abstraction between the API and the renderer: they get re-rendered every frame for you. Componentes normally have events and state by themselves. Sometimes there's a visual editor too.
The main difference is the API. One is lower level than the other. In practice, the APIs aren't that different, except when it comes to event handling.
[1] - https://docs.unity3d.com/Manual/gui-Basics.html
They don't re-upload the whole scene to the GPU every frame, they don't recreate objects all the time. Immediate mode was only used in, like, the GL 1.x times.
An immediate mode GUI is defined by writing an update function that draws the UI, passing in data and conditionally running callbacks for interactive elements like buttons. There is no model saved in memory, rather the structure of the UI is implied through the code path taken by the update function. The UI library uses this function to both draw the UI and handle user interactions.
In 3D you use hardware acceleration that is 100-200 times more energy efficient than drawing in the CPU. But you loose flexibility.
It is actually way cheaper to actually clear the screen and redraw it again each 1/60 of a second than to complicate the design of the drawing.
If you make it complex, the GPU could not draw it, because the GPU is not flexible, so only the CPU could draw it.
With GPUs, those tricks do not make sense at all.
In the past, there was a lot of optimization for things like drawing windows on the screen, things like circular buffers, "happy ideas" everywhere that made it efficient in the CPU at the expense of complexity, that the CPU could handle.
Imagine solving a mathematical equation. With matrices we fill a table with zeros that represents the elements of the table that do not exist. This is inefficient, but with a matrix a machine can solve it automatically without "thinking".
Immediate Gui uses the same concept, it draws without caring for last frames or states, making it way simpler.