Back in the WebVR days the sentiment was we can never do this because security.
In my experience with standards bodies/Chrome, anything outside of the owning company's produce goals is "no, because security", and then then they flip-flop on that when the browser feature becomes a dependency on something an internal team is working on (which happened many times with WebVR, now WebXR).
So I'm curious what is Google working on that has this as a dependency, now that they've 180'd on it?
This had to do with leaking privacy sensitive information such as visited links by reading back pixels from a rendered <a>. They fixed this with a concept they call "read-back-allowed rendering"
"Security" is usually not a binary, but how much money you are willing to spend to figure out a way to do it securely. It shouldn't be surprising that chrome spends more resources on things that have a business case than things that do not.
The solution was simple and always possible. It is to make it only work for same-origin content. Aka it can't work for ads or closed source external web content.
Imagine being able to declare a list of paths with fill/stroke-paints to draw, filled in from attributes, variables and `calc()` values... one can only dream.
Which requires doing a ton of math/layout in JavaScript, since I need those shapes to go between, around and behind the normal HTML elements (working on games and editors here).
Doing all that is a colossal PITA, since SVGs aren't built to act as "fragments" I can arbitrarely draw with CSS or Canvas.
This new html-in-canvas API will alleviate the pain a bit, at least.
I would use it to render HTML elements such as inputs or a website in videos. Even though HTML is powerful the control over every pixel is important for high quality videos.
I wanted to implement a whole canvas native workspace, I started with a html only implementation and "fake canvas" approach but even for moderate number of panels (~50) the responsiveness dropped quite a lot. So I went with a new implementation with "real canvas" and worked out the panels/interaction.
In terms of performance this approach worked out much better (able to easily handle 1000+ panels/nested panels at 60+ fps) but the complexity goes very high in the implementation of each panel. As in I cant just use for eg a markdown rendering js library or any third party ui component. And frequently make a change would result in massive performance degradation.
Currently I have text/markdown/image/audio/video and pdf panel (you can drag and drop files)
I wonder if i should take another go at it and implement using html-in-canvas api. Afraid of the api being not maintained if they dont see benefit or usage?
The DOM in modern browsers is backed by texture compositing just like people hand-roll with canvas and webgpu, so it can be extremely fast.
If you can get by with canvas rendering, you could instead use HTML and eliminate tons of CSS and layout cost by using CSS containment and absolute positioning:
Thanks for the tip, I did not know about contain: strict. position absolute ofcourse everyone knows. Will checkout contain. But doesnt sound like worth trying out html based canvas again just for contain. I feel like I have come quite far ahead with the native canvas approach.
Honestly, I think the push to constantly add browser features (and thereby expand the already massive web API) is misguided. This feature, in particular, seems like it should be solved with library, rather than by further bloating the browser.
If I put on my conspiracy hat, it looks like Google is pushing to expand the web API as a means to maintain its lead in the browser market by making it harder for other browsers (i.e. Firefox) to maintain feature parity.
I have the same theory but for different reasons(commented something similar to a recent css functions thread), in this case though I don't thing you can solve it with a library, the canvas api is very limited, text layouting is very hard especially if you want to support different languages and directions. Let alone rendering a text input, if the use case is games, those usually don't have very complex inputs and that would be ok I think to implement a simpler one. IMO we should just have exposed the Skia text layout api in javascript.
You can't solve this with a library without essentially implementing a separate browser engine in JS (or WASM). Doing this directly in the browser engine is far simpler than that.
TBH, the first time I read about this, I had to check the date to make sure it wasn't an April Fool's joke I had missed. I'm sure there are some things this would be useful for, but the sheer amount of complexity it would add to web renderers doesn't seem commensurate with how useful it would be.
This kind of stuff just puts even more distance between established engines and smaller projects like Servo.
> the sheer amount of complexity it would add to web renderers doesn't seem commensurate with how useful it would be.
Why? The browser is effectively already rendering HTML in a “canvas”, via tiled rendering etc. This isn’t a whole new rendering path, just a redirection of the existing one.
Since then I've actually met these two and had to conclude that:
- They didn't have any use-cases in mind that weren't cosmetic
- They don't quite understand how being part of an advertising company shapes how they build and conceive of web tech
- They react quite poorly to reactions that aren't cheerleading
The answer for curved elements was handwaving and "just approximate it with flat projective transforms" which tells you this is not a serious attempt to expose mature UI internals, but just another shiny epicycle on the creaky leviathan that is the HTML5 web api.
The list of features that "automatically work" rests on assumptions that are only valid for minor visual adjustments. The moment you try to use HTML-in-Canvas as a more basic building block, a lot of that falls away.
e.g. Suppose I render a piece of text and then use that as a mask for visual effects. If the browser now draws a text selection highlight into that mask, the text becomes unreadable.
I've read your blog enough times to recognize it instantly, yet it's the first time I notice that you blog header is an animated and interactive 3D demo. Very cool!
The new API confuses me because I can't see what it can really do that you couldn't already achieve by adding some position:absolute elements that float above the canvas. I mean, yes, since it's rendered as a texture, you can apply an arbitrary visual transform… but apparently the input handling is outside your control, so what's even the point? You won't be able to interact with the element properly. I also assume you can't even really occlude things properly? It doesn't make any sense to me.
The thing about HTML in Canvas is that canvas should always have had the ability to add html to its content, and there would be no debate about whether it's a good idea or not.
Pretty sweet! I'm building out an AI powered motion graphics & video generator and the biggest limitation is not having HTML and Canvas play well. Some visual effects, while cool looking, really impact performance so I have to compromise. I genuinely think this is the last thing we need before we have After Effects level capabilities for AI motion graphics on the web.
Hmm, not immediately clear how this avoids potential privacy leaks, such as drawing DOM hyperlinks to the canvas and then scraping their visited-status colour from the pixel data.
Or maybe I'm behind the times; I've not done serious webdev for a long time.
This is the best news ever! I can't believe it was implemented in just one year after that blogpost... Boy I'm going to be spending good portions of my life making with this API...
This is cool! Unfortunately a side effect is going to be more flashy vibe coded product landing pages adding expensive shader effects to all of their HTML. If you thought scrolljacking was bad, we're going to be seeing some interesting hover / mouse / navigation effects.
The problem with doing apps on the web is the DOM. It’s wild how much complication browsers will add instead of just adding a new UI rendering method/substrate.
The DOM is a slow annoying pile when you want to make actual applications.
> Which GUI toolkits over internet do you refer to?
My point is there is no GUI toolkit for the internet and there should be. They should stop trying to morph the document object model into a platform for delivering applications, let it do documents, and give us something else/new. It'd also be the proper path to the new thing having accessibility support. Instead people raw-dog the GPU for UI and you get no screenreader support.
What they're doing here is a half-assed attempt: letting people draw but using the DOM for accessibility. The worst of both worlds I guess?
Remember when looking up a restaurant online loaded a full screen flash movie with an outdated menu and loud music you could not stop? The reason why we abolished everything else apart from the DOM was that as soon as we had anything else it just marginalized the open web by not being flexible and adaptable. Active X controls, Silverlight, Flash and Java applets all did more or less what you are requesting while not being fit for mobile.
I don't see how Canvas or WebGPU is any different. People want a way to write high-performance UIs without all the complexity/legacy of the DOM. It makes everything more difficult, and because it makes things easier for browser developers isn't a valid justification to make the user experience worse. The users should drive the technology. I'm sorry this architecture you built almost 30 years ago isn't adaptive enough to meet the needs of your users. Instead they should create a new one for this use case (GUIs/applications) and keep it focused on documents.
> isn't adaptive enough to meet the needs of your users
I imagine if this conversation continues this will be quoted and argued against. My point is that it's not working well. People find workarounds to do amazing things with the browser, but they shouldn't have to. Give developers what they've wanted for 15 years. The various committees and working groups keep missing the mark, by a lot. See WASM GC, WebGPU shader language, etc., etc.
Ok. Cool and all. But isn't the need for this a hack to begin with? Why isn't there some more unified graphics system by now? When it comes to presentation, the entire industry is fragmented that necessitates these things.
Because no one can agree on what that graphics system should look like.
WebGPU is too low level for most users so it needs a framework on top, and that would mean changing the way code distribution works (from bundling to packages) if you want it to be efficient.
Skia is at the right level but then you need to maintain two very different graphics APIs in parallel and somehow make them work together. You can't just give up WebGPU because games and graphics editors are starting to require it.
You've committed to building your app using canvas but realize you actually wanted the DOM. Now you can dig yourself deeper rather than fixing the original mistake!
...all the advantages listed under "The advantages of bringing the DOM to Canvas" apply to using the DOM generally.
Actually, I can see the case for having a primarily canvas-based app that, nevertheless, sprinkles in some DOM. It's the presentation here that rankles.
(Extra points lost for using "intractable" where interactive could be used; intractable isn't an English word while interactive is. Now, since I'm not a purist, interactable can be used as a noun in contexts like game programming that need such a word, since no such English word exists. But using it as an adjective only seems OK when not a native speaker and when not making an announcement. It just reinforces the impression that they don't know what they are doing.)
What? Intractable (meaning difficult to manage or solve), is a valid English adjective, the opposite of tractable. It has nothing to do with the word interact.
71 comments
[ 6.1 ms ] story [ 106 ms ] threadHere's hoping for it being DOM-in-Canvas all they way down.
In my experience with standards bodies/Chrome, anything outside of the owning company's produce goals is "no, because security", and then then they flip-flop on that when the browser feature becomes a dependency on something an internal team is working on (which happened many times with WebVR, now WebXR).
So I'm curious what is Google working on that has this as a dependency, now that they've 180'd on it?
What you described is the CSS painting api (part of CSS Houdini)
Imagine being able to declare a list of paths with fill/stroke-paints to draw, filled in from attributes, variables and `calc()` values... one can only dream.
What I really want is something along the lines of...
And yeah, fully aware that this looks abominable, but I'd rather this than stringly/DOMly generating SVGs or drawing in a canvas.Doing all that is a colossal PITA, since SVGs aren't built to act as "fragments" I can arbitrarely draw with CSS or Canvas.
This new html-in-canvas API will alleviate the pain a bit, at least.
Related bunch of comments from that time: https://news.ycombinator.com/item?id=48201222
In terms of performance this approach worked out much better (able to easily handle 1000+ panels/nested panels at 60+ fps) but the complexity goes very high in the implementation of each panel. As in I cant just use for eg a markdown rendering js library or any third party ui component. And frequently make a change would result in massive performance degradation.
https://canaster.in/
Currently I have text/markdown/image/audio/video and pdf panel (you can drag and drop files)
I wonder if i should take another go at it and implement using html-in-canvas api. Afraid of the api being not maintained if they dont see benefit or usage?
If you can get by with canvas rendering, you could instead use HTML and eliminate tons of CSS and layout cost by using CSS containment and absolute positioning:
If I put on my conspiracy hat, it looks like Google is pushing to expand the web API as a means to maintain its lead in the browser market by making it harder for other browsers (i.e. Firefox) to maintain feature parity.
The browser API has become so big because the implementers are quite tenacious, with multiple parties doing “extend” in parallel…
This kind of stuff just puts even more distance between established engines and smaller projects like Servo.
Why? The browser is effectively already rendering HTML in a “canvas”, via tiled rendering etc. This isn’t a whole new rendering path, just a redirection of the existing one.
Since then I've actually met these two and had to conclude that:
- They didn't have any use-cases in mind that weren't cosmetic
- They don't quite understand how being part of an advertising company shapes how they build and conceive of web tech
- They react quite poorly to reactions that aren't cheerleading
The answer for curved elements was handwaving and "just approximate it with flat projective transforms" which tells you this is not a serious attempt to expose mature UI internals, but just another shiny epicycle on the creaky leviathan that is the HTML5 web api.
The list of features that "automatically work" rests on assumptions that are only valid for minor visual adjustments. The moment you try to use HTML-in-Canvas as a more basic building block, a lot of that falls away.
e.g. Suppose I render a piece of text and then use that as a mask for visual effects. If the browser now draws a text selection highlight into that mask, the text becomes unreadable.
I've read your blog enough times to recognize it instantly, yet it's the first time I notice that you blog header is an animated and interactive 3D demo. Very cool!
https://www.vibemellow.ai
Very stoked for this!
Or maybe I'm behind the times; I've not done serious webdev for a long time.
:D.
https://github.com/WICG/html-in-canvas/#read-back-allowed-re...
When activated in chrome, nesting currently works, but somehow breaks events.
https://johnhenry.github.io/html-in-canvas-lab/
The DOM is a slow annoying pile when you want to make actual applications.
Which GUI toolkits over internet do you refer to?
My point is there is no GUI toolkit for the internet and there should be. They should stop trying to morph the document object model into a platform for delivering applications, let it do documents, and give us something else/new. It'd also be the proper path to the new thing having accessibility support. Instead people raw-dog the GPU for UI and you get no screenreader support.
What they're doing here is a half-assed attempt: letting people draw but using the DOM for accessibility. The worst of both worlds I guess?
> isn't adaptive enough to meet the needs of your users
I imagine if this conversation continues this will be quoted and argued against. My point is that it's not working well. People find workarounds to do amazing things with the browser, but they shouldn't have to. Give developers what they've wanted for 15 years. The various committees and working groups keep missing the mark, by a lot. See WASM GC, WebGPU shader language, etc., etc.
Can you be more specific? Because to me the DOM is one of the simplest conceptual representations of UI one could imagine, a tree of nodes.
WebGPU is too low level for most users so it needs a framework on top, and that would mean changing the way code distribution works (from bundling to packages) if you want it to be efficient.
Skia is at the right level but then you need to maintain two very different graphics APIs in parallel and somehow make them work together. You can't just give up WebGPU because games and graphics editors are starting to require it.
https://github.com/Jack000/PartKAM
You've committed to building your app using canvas but realize you actually wanted the DOM. Now you can dig yourself deeper rather than fixing the original mistake!
...all the advantages listed under "The advantages of bringing the DOM to Canvas" apply to using the DOM generally.
Actually, I can see the case for having a primarily canvas-based app that, nevertheless, sprinkles in some DOM. It's the presentation here that rankles.
(Extra points lost for using "intractable" where interactive could be used; intractable isn't an English word while interactive is. Now, since I'm not a purist, interactable can be used as a noun in contexts like game programming that need such a word, since no such English word exists. But using it as an adjective only seems OK when not a native speaker and when not making an announcement. It just reinforces the impression that they don't know what they are doing.)
The word bothering me is: interactable