Exactly what I needed, just as I was about to start screwing with distance fields or the like. THANK-YOU!
Edit: I may have spoken slightly too soon, not since the technique isn't awesome, but because this isn't yet usable without an atlas generation tool that I don't know how to build.
As far as performance goes, this is rendering a 124-page PDF (I know, just the glyphs...) utterly without lag using the GPU, on a 16 Gbyte quad i7 box that can barely render a couple of pages under OS X Preview without significant pauses. Impressive and Incredible.
Performance is pretty decent (around 48fps for a full page with lots of glyphs, 1080p) on my Intel HD 4400. This is on linux though. I hear the OS X intel drivers have some shader compiler pitfalls.
I find it curious the number of people in this thread being amazed that high end i7 processors and graphics cards designed for gaming can passably render some text. Is there something I'm missing?
It's not so much they are amazed it capable of doing this. But more of a WOW this is fast compared to rendering theses slow PDF pages on the CPU.
It's a bit like saying: wow this rocket propelled car is fast. Are we surprised it is faster ? No. Are we like: "wow we could go faster !"? Yes
To be fair, I don't understand why we don't have a lib, or even this stuff build into OpenGL/Direct3D. We are doing raster work CPU's. It's relatively slow.
The something you're missing is that we're used to see this done on the CPU, and it's not something CPUs are good at: Pages full of text rendered with scalable fonts involves either a huge number of bezier curves etc. or a huge number of small blits, though you can trade some of it off by spending huge amounts of memory pre-rendering and caching (but if you cache bitmaps at the character level, you will find it harder to do really nice typography - ligatures etc.).
It's not that it's that amazing that it's possible on a GPU, but that it's amazing to see the difference.
Sorry to sound like an old fart, but computers have been doing this for years, and no, caching rendered glyphs does not eat huge amounts of memory. e.g. RiscOS on ancient ARM processors, drawing sub-pixel anti-aliased text on a CPU that had no floating point. Even just 64kb of font cache to store the bitmaps could keep everything running blazingly fast.
Of course, the font renderers have gotten smarter, and of course, screen resolutions have increased, but the difficulties inherent in font renderering are all at the small scale, where you want text to remain crisp even when using only a few pixels.
I have to quibble with blazing fast. What was the resolution on an A3000 or whatever. Pretty low. My recall was the anti-aliased text rendering was technically impressive, but practically less-so due to low-resolution. A SUN or Mac was quite a bit nicer. On the Archimedes rendering against a non-solid non-monochrome color was especially nasty.
In that case, caching very expensive rendering to improve performance makes complete sense. Nice Impression summary and screenshot here. http://www.iconbar.com/forums/viewthread.php?threadid=10197 Bear in mind running on a 4 MHz ARM3 I would think.
Now, with a sufficiently high-resolution display and a performant pixel shader, all the stupid caching and hinting logic can be thrown away, and effort can go instead into optimizing this technique, with no loss of generality, when rendering to arbitrary structures in the page.
Since screen resolutions have scaled far more slowly than memory sizes in the last couple of decades, surely it makes even more sense to cache font rendering nowdays? If we both agree that it was a win for low resolutions (say 640x480), back when a computer might have 2MB of RAM, surely it's far less of a cost now that we have 4k displays (only a few factors bigger) and well over 1000x the amount of memory.
Just about everything in computing is a form of caching. It's rarely stupid, so throwing it out is short-sighted.
> Sorry to sound like an old fart, but computers have been doing this for years
With the CPU, yes, but not on the GPU. The point is that we've generally been limited to doing this serially, which ties into this:
> and no, caching rendered glyphs does not eat huge amounts of memory
It does the moment your user starts to zoom rapidly in and out like in this demo and you want smooth zooming without rendering artefacts, potentially with multiple fonts, and potentially dealing with large font sizes and multi-colour fonts.
I'm an old enough fart myself to know that at least the earliest ARMs with RiscOS (as well as the Amiga's) had little enough memory/cpu bandwidth to make fast, smooth zooming of lots of text a challenge even with pre-cached font-bitmaps. And it took a long time before most systems would have enough memory to be able to treat this that casually.
> Of course, the font renderers have gotten smarter, and of course, screen resolutions have increased, but the difficulties inherent in font renderering are all at the small scale, where you want text to remain crisp even when using only a few pixels.
I've written terminal emulator code, and I've written GUI toolkit code several times, and text rendering speed on CPUs is still something that needs a lot of optimization to avoid it being a bottleneck. At small sizes you obviously can easily afford to cache, but then you end up having many more small blits - the overhead is still painful but for different reasons. E.g. intelligently blitting the existing window content to minimize re-rendering tends to be a major speedup vs. re-rendering every glyph even with fixed width bitmap fonts.
It does the moment your user starts to zoom rapidly in and out
Well yes, that'll wipe out the cache hit rate, but when has the use case of 'zooming rapidly in and out of text' ever been a particular point on contention for any application? Even the original iphone's web zoom was good enough, a smooth zoom with temporarily blurry text until you pick a good zoom point. Seems like a very obscure niche functionality to optimise, one where no-one apparently cares about right now.
I'm an old enough fart myself to know that at least the earliest ARMs with RiscOS (as well as the Amiga's) had little enough memory/cpu bandwidth to make fast, smooth zooming of lots of text a challenge even with pre-cached font-bitmaps.
True enough! I can't even think of any app that had smooth zooming... most apps had a zoom factor (e.g. x1, x2, etc) that would work perfectly with a font cache. The main crunch point was the simple, intelligent, partial rendering of text as you pan around a page.
At small sizes you obviously can easily afford to cache, but then you end up having many more small blits
Going back to my nostalgia for RiscOS, I wrote a dumb anti-aliased font renderer that would draw fonts dot-by-dot rather than by blitting a 2d transparent bitmap to the screen, it turned out to be massively faster than even a simple cached bitmap font renderer, even at larger sizes. Definitely not a good approach today with higher-res screens though... (and the storage format was massively wasteful at larger sizes)
Terminal text rendering is a bone of contention for me, as it's frustrating how some modern-day terminals can be utterly appalling at doing their job (e.g. typing 'dmesg' and having to wait several seconds for it to complete), so I'm all in favour of anyone who takes an interest in making it fast. Most terminals unfortunately seem to try and redraw every new line to the screen, rather than realising that they've got another 10000 lines to print and perhaps the user might be content with a (say) 30fps update interval...
Very clever technique! It seems more true to letterforms than SDF. Seems like it might need more texture bandwidth/dependent reads than SDF, but less texture memory as a whole?
I've been looking at different text rendering methods for a VR engine. I wonder if this technique works in 3D space? If so I'll give it a shot, benchmark it against SDF :)
I'm using TextMesh Pro for a VR/AR application in Unity called Pantomime, and it works great for text in the 3d world space, as well as 2d gui dialogs. You can get up really close to the text, and it looks perfect. I posted some links to demos on the TextMesh Pro forum here:
Just had a quick read though that paper, and it doesn't really look like the same thing.
This technique treats a pixel as a unit circle, coloring it according to its length of intersection, along multiple axes, with a glyph simply described by 3 point beziers, oriented to keep 'inside' on the right.
A point fully inside a glyph has a trivial intersection, and is fully shaded.
Points on the edge of a glyph intersect a bezier, with shading of the pixel being proportionate to the amount of intersection, giving anti-aliasing.
The product I work on in my day job has similar code to handle rendering filled polygons, with similar constraints on widdershins/anti-widdershins to turn off filling for 'holes'. The 'magic' of this technique seems to be applying that technique in a shader, on a per-pixel basis.
There's no triangulation that I can see (compare with the elaborate schemes Loop-Blinn paper above), just a decomposition of glyphs to these ordered collections of 3 point bezier curves, and some fiddling to get it all saved in a texture.
Well now that you said that it's 3x damages, whether or not you read it (at least in the US, the standard is whether there's "an objectively high likelihood" of infringing some patent that is "either known or so obvious that it should have been known" to you, so knowing that a thing's been patented counts even if you don't read the patent).
But even 1x damages are pretty high. Read it carefully and do something sufficiently different and avoid infringement in the first place.
Although cool, this is not a particularly novel idea.
I remember reading a 2007 SIGGRAPH paper about something similar Valve did in HL2 (or maybe it was Orange Box). Either way, it was a similar method of rendering vector textures via distance fields. Here it is: http://www.valvesoftware.com/publications/2007/SIGGRAPH2007_...
This is a cool technique. The WebGL demonstration is pretty compelling!
It's a little odd to me that they describe distance fields as having an unavoidable problem with sharp corners, though. The Valve paper on signed distance fields for text demonstrates a simple workaround that fixes sharp corners...
Signed Distance Field fonts are wonderful! I've been using a Unity extension called TextMesh Pro, which works very efficiently, and renders beautiful text that you can configure and decorate in many ways.
Beyond the obvious benefits of SDF fonts, it also has excellent and extensive layout, formatting and other useful features, like automatically scaling the text to fit in a given area.
The source code for the shaders and formatter and ugui integration is included, but not the atlas generator (although you might ask the developer if you need it -- he's engaged and helpful, and provides good support and quick fixes for Unity updates). It includes desktop shaders with many fancy features (outlines, drop shadows, beveling, glow, bump mapping, a surface shader that reacts to lighting and throws shadows into the world), and simplified optimized shaders for mobile.
I've used it (and read the source code to see how it works), and can testify that it works great on Unity's desktop, iOS, Android and WebGL backends. The code itself and the way it offloads so much work to the GPU is very beautiful and elegant.
Here's a benchmark that compares 5000 crisp TextMesh Pro objects rendering at 70 FPS, versus 5000 fuzzy Unity text objects rendering at 42 FPS: https://www.youtube.com/watch?v=rdc8UkxuSZc
Here's the same benchmark compiled with Unity's WebGL backend, with the same number of 2500 static and 2500 dynamic text objects. Nowhere near the 70 FPS of native code of course, but not bad for WebGL and so many objects -- zoom in with the mouse wheel to see the text up close: http://donhopkins.com/home/TextMeshProBenchmark/
no subpixel rendering or hinting. Im confused - is there a problem with cpu rendered fonts that needs fixing? I didnt notice anything wrong last time I was reading schematics and datasheets on 4K monitor.
This is useful for gaming or any other application where text is integrated into a GPU rendered 3D environment and may be subject to affine transformations.
> Im confused - is there a problem with cpu rendered fonts that needs fixing?
Performance. Even if an inefficient painting algorithm manages to paint at 60 FPS (which is very much not the case in, for example, common Web apps), improving its performance can free up the CPU to do more things and can reduce power consumption.
Think about it. If your GPU is sitting there idle, why not use it and free up the CPU to do other things? Today, most apps that aren't games are woefully underutilizing the GPU. Programs ideally should be utilizing all the silicon in your device while working so the work can be done faster and the hardware can enter low-power mode quicker.
On a modern PC games use a small fraction of the CPU and the GPU is the bottleneck. The only applications that really strain the CPU (i.e. all cores) are trivially parallelizable ones like transcoding, compilation, test suites, etc.
For games specifically I'd prefer more work done on CPU and free up the GPU, provided the communication overhead isn't too much and latency doesn't become a problem.
Low power is less of a problem than thermal limiting for compute intensive work. I see about 90% variation in test times on my laptop for unit tests, for example.
So overall I see more application for this in apps than games, and only then when the tradeoffs are right. It's not a simple call.
Games graphics engine guy here. I'm very happy to move text rendering from the CPU to the GPU. Done well it moves milliseconds of CPU work to tenths (or less) of milliseconds of GPU work.
Also, I care quite a bit about power draw because my customers on laptops and mobile are very sensitive to it. If my game is a battery hog, it doesn't get played as long or as often.
Sure, for games the tradeoffs may be different. For apps, though, we regularly see 1% GPU time, 99% CPU time. Even "GPU-accelerated" painting backends like Skia-GL can easily spend 90% of their time on the CPU--in the driver, issuing state changes, rasterizing fonts, building batches, creating display lists. It's not an exaggeration to say apps are desperate to find work that can be offloaded to the GPU.
Don't know why you're getting downvoted. More performance is certainly always good, but without subpixel anti-aliasing the text will tend to look pretty crappy absent a ridiculously high resolution display.
Eh, nowadays you're probably rendering on a phone screen with a PenTile display and 300ppi, so the entire thing is moot.
Plus, every implementation of subpixel hinting I've seen (and I've seen them all, on all three major desktop platforms), produces absolutely disgusting colour fringing.
Just lay off the ClearType(tm) kool-aid and do normal anti-aliasing. We don't do colour-fringing anti-aliasing for any other geometric primitive, I see no reason to implement a font-specific approach that only works if the user is colour-blind.
I assume you are thinking through it's implications for WebVR+servo?
I can see questioning the utility for current 2d applications but this is really critical speed when the large amount of translation and scaling of many pages in a 3d environment is the expectation.
The css3d "portal" solution for webgl doesn't seem like it can get us very far at all...
I'd use this today in my webvr/gl project if it was source complete.
The demo that zooms in and out of a PDF is much faster than it would be with CPU rendering. When you zoom in and out in an existing PDF viewer, it takes much longer than 1/60 sec to re-rasterize the page.
The CPU takes a long time to rasterize fonts. It only seems fast because the glyphs are cached. If you need text drawn in so many different sizes, like in that demo, the cache is not as effective.
Subpixel rendering isn't implemented in the demo but just requires running the calculations three times with 1/3 pixel offsets. It still would only need to read the texture data once. I didn't implement it in the demo because there's no way to ask the browser what the display's subpixel order is. If you're viewing it on an ipad and switch from portrait to landscape your subpixel order changes.
Hinting was necessary back before good antialiasing existed but I don't think many renderers use it now. This technique renders each glyph differently according to its subpixel position - completely the opposite approach to hinting, which snaps features of the glyph to integer positions on the pixel grid.
This is really neat. Though in practice if you are using this kind of thing I'm pretty sure you will want to combine it with atlasing (i.e. construct an atlas of glyphs via render to texture). That's because it's a waste of time to rerasterize glyphs in the FS every frame (which appears to be 200+ lines) instead of caching the results and reducing the per-frame work to a 5-line FS that just blits. In most apps pans are more common than zooms and the same glyphs are used over and over, so this usually ends up being a win. Using atlases also reduces the number of raster operations because the same glyphs tend to appear on the page repeatedly, and by using atlases you rasterize each glyph only once. Even more importantly, though, atlasing gives you the ability to reuse a blitting fragment shader to render other content (images, figures, etc.) as well as glyphs from the atlas in order to reduce state changes.
That was my first thought. This is brilliant, but the FS is way overworked, making this less practical and going back to the original solution (because it's fast).
It's a trade off between speed and variability, which for games (and most apps) speed trump variability (especially because of consistent design). But given generating a texture atlas is a "1 time task"; and recompiling the curves in the GPU is per frame in the paper, the next best thing I guess is off loading the glyph atlas generator to the GPU, not CPU, since the GPU maybe faster.
Wait. I guess you did not read the article at all? It is all about atlasing.
There are issues with atlas of glyphs; so rather than putting the whole glyphs in atlas, he slices the glyphs in a grid, the grid being defined by the intersection of the underlying Bezier curves. And then he puts the grid pieces in atlas.
> Wait. I guess you did not read the article at all? It is all about atlasing.
Yes, I understand the technique. Perhaps I should be more clear about my terminology. I'm using "atlas" in the narrow sense of "actual pieces of bitmap data stored in a texture". My point is that this technique should not be a substitute for atlasing in this sense, but could well be combined with it.
I don't know why you are being downvoted, that is what is happening and I am not convinced that straight textures would be better. We already know that SDF textures are better than straight texture atlases anyway. The technique is obviously practical in some respect (since it renders a pdf faster than anything else I've seen) and the quality of the text is very sharp, even close up.
Super sharp lines when zoomed in may not seem like a big deal, but as you increase resolution you will reap the benefits since the same size text will be sharper.
> We already know that SDF textures are better than straight texture atlases anyway.
This is isn't true. SDF textures can be "better" for "some" applications (like games that want to display stencils mostly
at arbitrary zoom levels without losing a lot of quality) but they're also visibly lower-quality compared to glyphs rendered by freetype at each one of these zoom levels.
So, for an application that simply displays lots of text while infrequently changing the glyph size, SDF textures are not a good solution.
You are essentially saying that an SDF texture isn't as good as a freetype glyph rendered at the resolution it is being displayed at. If you don't have to change the zoom of what you are looking at you don't have to use a texture at all, you can just use the pixels from the freetype vector render directly.
The FS is something like 200 lines (verified in the WebGL debugger). A blitting shader is about 5. If fragment shading load doesn't matter, then of course you don't need straight textures. But if you want to reduce the load on the GPU and do the efficient thing, without any quality loss, it's better to use atlases.
> We already know that SDF textures are better than straight texture atlases anyway.
Even that depends. SDFs are slow to construct and, because they have a special FS, you have to pay attention to the number of state changes. They do have the advantage (unlike this technique) of having a very simple FS--they can even be used in some fixed function hardware!--making it practical to rerasterize glyphs every frame.
> The technique is obviously practical in some respect (since it renders a pdf faster than anything else I've seen)
That says a lot about how existing PDF renderers are slow and nothing about what is the fastest thing to do on the GPU.
There's also the higher level—the thing web browser rendering engines do, where you generate a texture for each "tile" (single-layer viewport rectangle) of likely-to-be-static text, and carry those around until you have to zoom or the text changes.
I'm not sure if you need the medium-level (font atlasing) if you have both lower-level (bezier-curve atlasing), and higher-level (pre-composited "tiles.")
This is one of the reasons I really like WebGL, actually: for GUI elements, you can usually skip rendering them within the game altogether, instead using plain HTML+CSS controls positioned above the <canvas>, where they'll be treated as separate "tiles" that aren't dirtied by whatever your game is doing. Imagine: relatively-static fully-RGBA-translucent stuff sitting on top of your animating viewport, composited into a final image each frame, "for free."
The font atlas is definitely still worth it. For example: your comment, which fits in a single browser tile, is made up of 735 drawn glyphs. There are 50 distinct glyphs used (not counting italics because I'm lazy).
Assuming it takes twice as long to draw a glyph into an atlas but half as long to draw it once it's in the atlas, an atlas that starts completely empty will speed up text drawing for your comment by 66% (735 divided by 50 * 2 + (735 - 50) * 0.5, which is 442.5). And this underestimates the savings by a lot (drawing glyphs from an atlas is much more than two times as fast, and some glyphs are probably in the atlas already).
The question is about amortized time-cost, though.
• Most frames won't involve zooming or text-change—and therefore will be handled at "zero" time-cost by re-using the pre-composited tile.
• The frames that do involve text-change (but not zoom) can benefit from font-atlasing.
• The frames that involve discrete zoom, where text goes from one integral font size (represented in the atlas) to another integral font size (also in the atlas) can also benefit from font-atlasing.
• The frames that involve continuous zoom—such as in this demo—can't benefit from font-atlasing, because the frames where the font is e.g. 7.037773pt large can't be served from the atlas.
Keep in mind that the only reason we do discrete zoom-levels in PDF readers, text editors, etc. is that font-atlasing is "the optimization we have", and it calls for discrete zoom levels. When the UX affordance is for continuous zoom—think about pinch-to-zoom in Mobile Safari, or about zooming into Google Earth—font-atlasing doesn't and can't work. But this technique can.
Now, in games in particular, "continuous zoom" is a pretty rare edge-case. GUI elements don't tend to get zoomed into[1].
And "in-VR" elements, like street signs in an FPS, aren't usually designed as a bunch of geometry[2], but rather as a very high-detail flat texture that maybe gets mipmapped from far away. "Zooming" by approaching a street sign doesn't make the fonts on it re-render themselves; there are no fonts on it, they're just baked into an image.
---
[1] Well, maybe; I could see a case for continuous zoom of the fonts—and other GUI geometry—plastered over units in RTS games. "Pan up" with units selected, they stay selected but their unit names+stats get smaller and smaller and eventually get obscured by draw-distance and atmosphere effects, just as if they were actually "there" in the scene instead of part of the GUI. (Makes me think of the windows in the TV show ReBoot.)
[2] This does make me imagine a Blade Runner-esque game full of holographic signs, though. Those could definitely benefit from this approach.)
Font atlasing can and does work during pinch-to-zoom: that was the point I was trying to make. You have to generate a new atlas each time you paint, but drawing multiple "e" glyphs from an atlas is much cheaper than rasterizing each "e" individually.
Huh; I almost don't think of that as font-atlasing. If you know you're continuously zooming—and therefore are going to throw out the atlas at the end of the frame—it's almost more like a "glyphs rendered this frame" cache. Which changes things a bit, because you don't need to preload the entirety of the "font"—which might be a unicode font!—into the atlas, but only what you're rendering.
I have no argument against that kind of font-atlasing as a process; it's fairly more stateless, more like a secret cache inside the black box of the glyph-rendering kernel.
Consider that each "e" may be slightly differently aligned to the pixel grid though. So using an atlas is lower quality than drawing each "e" from scratch exactly as it aligns to the screen.
> There's also the higher level—the thing web browser rendering engines do, where you generate a texture for each "tile" (single-layer viewport rectangle) of likely-to-be-static text, and carry those around until you have to zoom or the text changes.
I think this is actually a harmful approach in most cases--all it does is reduce the number of vertices and slightly reduces overdraw in exchange for a lot more memory consumption and a lot more draw calls. Overdraw can be reduced in other ways, such as sorting front to back and taking advantage of early Z for opaque content. Web pages and PDFs are so totally not bound on either ROPs or VS, and are very sensitive to draw call count, that caching tiles to minimize overdraw is almost never worth it.
But this is somewhat off topic.
> I'm not sure if you need the medium-level (font atlasing) if you have both lower-level (bezier-curve atlasing), and higher-level (pre-composited "tiles.")
You definitely do. You're sacrificing a lot of performance for the rendering of each tile if you don't do that.
I don't think we're imagining the same things here. Think less '3D landscape with text floating in it for some reason' (e.g. SecondLife), and more '3D landscape with labels applied to it' (e.g. Google Earth). Nothing from the "VR" layer obscures the "labels" layer; the labels float above everything, as if part of the GUI, but following the same scale as the things they label.
> PDFs
I'm actually unsure why a (non-interactive) PDF page, displayed at 100% zoom, can't be baked into a single extremely-high-resolution tile—applied as a texture to a single GL rectangle with plain-old lanzcos downsampling. That'd work for the overwhelming majority of the frames.
You'd only need to re-render the tile when you zoom in enough that it'd look bad; effectively, you could think of the PDF page as coming from a mipmapped texture, but where you only have the current size and will just-in-time rerender when the zoom-factor changes. And then, when the zoom factor makes the PDF not fit on the screen, the "tile" would be the screen size—so you'd re-render sibling tiles when the user moves.
Or, in other words: PDFs can be rendered exactly the way the image tiles on a mapping website are rendered, can't they? Doesn't that approach win over re-rendering all the text from a font-atlas every frame? And if not, why do browser rendering engines use tiles? (This isn't a rhetorical question to make my point; I'm not a graphics dev and I honestly don't see why.)
> I'm actually unsure why a (non-interactive) PDF page, displayed at 100% zoom, can't be baked into a single extremely-high-resolution tile—applied as a texture to a single GL rectangle with plain-old lanzcos downsampling. That'd work for the overwhelming majority of the frames.
Sure, but that uses a lot of memory, and initial pageload is going to be really slow. Especially when you consider you need to generate mipmaps if it's really high resolution, or else blow out your GPU's L1/L2 cache in FS execution.
> Doesn't that approach win over re-rendering all the text from a font-atlas every frame?
Not significantly, in my experience. Assume, in the simple case, that you're just rendering a bunch of text with no overlap. In that case, the only thing that tiles buy you over rerendering all the text from an atlas is decreased vertex count. You're still touching the same number of pixels in the FS and ROP units either way: you just do so with more vertices in the texture atlas case and fewer vertices in the tiling case. Now consider that creating and maintaining the tiles has costs over maintaining the glyph atlas (which you have to do either way): you have the memory of the tiles, the overhead of creating and switching FBOs, lots of little textures to keep around (which in naive implementations results in tons of state changes), and extra draw calls to render the tiles after rendering the content.
> why do browser rendering engines use tiles?
Mostly because it was an easy way to fit GPU-accelerated panning and zooming into the existing, originally CPU-based, rendering architectures that browser engines used (and still largely use). Tiling was popularized by Mobile Safari on the iPhone in 2007 as an easy way to avoid repainting the entire page on the CPU every frame when the user performed touch gestures. I don't think it's necessarily a globally optimal decision.
This latter bit is exactly what modern 2D display drivers already do, or at least what the NVIDIA binary X driver does. It keeps a glyph cache in video memory and just blits from it to the extent that it can. It's still counting on the software to do the rasterization though, and I don't think there's a way to do the rasterizing on the GPU and then feed that data straight into the glyph cache.
The demo runs very poorly on my 15" MBP (late 2013, Intel Iris Pro). I'm not sure if it's just the sheer number of glyphs being rendered at once, or whether it's something specific that this GPU is having trouble with.
Some related links for those interested in WebGL text rendering:
i too, am interested in the sources. Viewing the source on the page is an OK option too i guess, but it'd be nice to have it as a library that i can examine (and potentially use!)
Whole PC freezes (except the cursor) with GeForce GT 610 in Chrome (Linux) on the demo. Can't even switch to another tty to kill the process. The PC is average otherwise (AMD FX 8320 with 8GB of RAM).
Nice work. The subdivision into cells and encoding into GPU textures is strongly reminiscent of the 2008 paper "Random-Access Rendering of General Vector Graphics" by Nehab and Hoppe [1].
Rotating the line samples is a rather interesting idea. Seems like that would converge to the equivalent of a convolution with a radial tent filter.
Why don't GPUs have some hardware in them to make rendering text simpler? It seems like a pretty common function that every game has. Have you ever seen a game without any text in it?
I really like this idea, but bezier curves are quite computationally difficult to use. There are many representations of these kind of continuous shapes which are much easier for this kind of processing.
For example I would try converting the bezier representation into an implicit HRBF representation (here is a good explaination in 3D http://rodolphe-vaillant.fr/?e=12). This representation should be much easier to process on the GPU - checking how much the point is inside/outside the glyph should only be one matrix multiplication which would make the computation in the actual shader really really fast.
The method seems to assume the geometric complexity of a glyph is relatively small. I wonder if it will work with CJK glyphs. Possibly the number of tiles per glyph would need to be greater; I'm not sure how badly a couple of doublings would impact performance.
90 comments
[ 11.9 ms ] story [ 266 ms ] threadEdit: I may have spoken slightly too soon, not since the technique isn't awesome, but because this isn't yet usable without an atlas generation tool that I don't know how to build.
As far as performance goes, this is rendering a 124-page PDF (I know, just the glyphs...) utterly without lag using the GPU, on a 16 Gbyte quad i7 box that can barely render a couple of pages under OS X Preview without significant pauses. Impressive and Incredible.
For the curious, the Lumia error appears to stem from this: https://groups.google.com/forum/embed/#!topic/angleproject/-...
It's a bit like saying: wow this rocket propelled car is fast. Are we surprised it is faster ? No. Are we like: "wow we could go faster !"? Yes
To be fair, I don't understand why we don't have a lib, or even this stuff build into OpenGL/Direct3D. We are doing raster work CPU's. It's relatively slow.
Nvidia has it's nvPathRendering extension to OpenGL https://developer.nvidia.com/nv-path-rendering
I'm not aware of a fully portable equivalent to either of these two libs.
It's not that it's that amazing that it's possible on a GPU, but that it's amazing to see the difference.
Of course, the font renderers have gotten smarter, and of course, screen resolutions have increased, but the difficulties inherent in font renderering are all at the small scale, where you want text to remain crisp even when using only a few pixels.
In that case, caching very expensive rendering to improve performance makes complete sense. Nice Impression summary and screenshot here. http://www.iconbar.com/forums/viewthread.php?threadid=10197 Bear in mind running on a 4 MHz ARM3 I would think.
Now, with a sufficiently high-resolution display and a performant pixel shader, all the stupid caching and hinting logic can be thrown away, and effort can go instead into optimizing this technique, with no loss of generality, when rendering to arbitrary structures in the page.
Just about everything in computing is a form of caching. It's rarely stupid, so throwing it out is short-sighted.
With the CPU, yes, but not on the GPU. The point is that we've generally been limited to doing this serially, which ties into this:
> and no, caching rendered glyphs does not eat huge amounts of memory
It does the moment your user starts to zoom rapidly in and out like in this demo and you want smooth zooming without rendering artefacts, potentially with multiple fonts, and potentially dealing with large font sizes and multi-colour fonts.
I'm an old enough fart myself to know that at least the earliest ARMs with RiscOS (as well as the Amiga's) had little enough memory/cpu bandwidth to make fast, smooth zooming of lots of text a challenge even with pre-cached font-bitmaps. And it took a long time before most systems would have enough memory to be able to treat this that casually.
> Of course, the font renderers have gotten smarter, and of course, screen resolutions have increased, but the difficulties inherent in font renderering are all at the small scale, where you want text to remain crisp even when using only a few pixels.
I've written terminal emulator code, and I've written GUI toolkit code several times, and text rendering speed on CPUs is still something that needs a lot of optimization to avoid it being a bottleneck. At small sizes you obviously can easily afford to cache, but then you end up having many more small blits - the overhead is still painful but for different reasons. E.g. intelligently blitting the existing window content to minimize re-rendering tends to be a major speedup vs. re-rendering every glyph even with fixed width bitmap fonts.
Well yes, that'll wipe out the cache hit rate, but when has the use case of 'zooming rapidly in and out of text' ever been a particular point on contention for any application? Even the original iphone's web zoom was good enough, a smooth zoom with temporarily blurry text until you pick a good zoom point. Seems like a very obscure niche functionality to optimise, one where no-one apparently cares about right now.
I'm an old enough fart myself to know that at least the earliest ARMs with RiscOS (as well as the Amiga's) had little enough memory/cpu bandwidth to make fast, smooth zooming of lots of text a challenge even with pre-cached font-bitmaps.
True enough! I can't even think of any app that had smooth zooming... most apps had a zoom factor (e.g. x1, x2, etc) that would work perfectly with a font cache. The main crunch point was the simple, intelligent, partial rendering of text as you pan around a page.
At small sizes you obviously can easily afford to cache, but then you end up having many more small blits
Going back to my nostalgia for RiscOS, I wrote a dumb anti-aliased font renderer that would draw fonts dot-by-dot rather than by blitting a 2d transparent bitmap to the screen, it turned out to be massively faster than even a simple cached bitmap font renderer, even at larger sizes. Definitely not a good approach today with higher-res screens though... (and the storage format was massively wasteful at larger sizes)
Terminal text rendering is a bone of contention for me, as it's frustrating how some modern-day terminals can be utterly appalling at doing their job (e.g. typing 'dmesg' and having to wait several seconds for it to complete), so I'm all in favour of anyone who takes an interest in making it fast. Most terminals unfortunately seem to try and redraw every new line to the screen, rather than realising that they've got another 10000 lines to print and perhaps the user might be content with a (say) 30fps update interval...
I've been looking at different text rendering methods for a VR engine. I wonder if this technique works in 3D space? If so I'll give it a shot, benchmark it against SDF :)
http://digitalnativestudios.com/forum/index.php?topic=470.ms...
Some discussion about text in VR:
https://www.reddit.com/r/oculusdev/comments/34d8nm/reading_i...
More about Pantomime:
http://pantomimecorp.com
This technique treats a pixel as a unit circle, coloring it according to its length of intersection, along multiple axes, with a glyph simply described by 3 point beziers, oriented to keep 'inside' on the right.
A point fully inside a glyph has a trivial intersection, and is fully shaded.
Points on the edge of a glyph intersect a bezier, with shading of the pixel being proportionate to the amount of intersection, giving anti-aliasing.
The product I work on in my day job has similar code to handle rendering filled polygons, with similar constraints on widdershins/anti-widdershins to turn off filling for 'holes'. The 'magic' of this technique seems to be applying that technique in a shader, on a per-pixel basis.
There's no triangulation that I can see (compare with the elaborate schemes Loop-Blinn paper above), just a decomposition of glyphs to these ordered collections of 3 point bezier curves, and some fiddling to get it all saved in a texture.
But even 1x damages are pretty high. Read it carefully and do something sufficiently different and avoid infringement in the first place.
I remember reading a 2007 SIGGRAPH paper about something similar Valve did in HL2 (or maybe it was Orange Box). Either way, it was a similar method of rendering vector textures via distance fields. Here it is: http://www.valvesoftware.com/publications/2007/SIGGRAPH2007_...
It's a little odd to me that they describe distance fields as having an unavoidable problem with sharp corners, though. The Valve paper on signed distance fields for text demonstrates a simple workaround that fixes sharp corners...
https://lambdacube3d.wordpress.com/2014/11/12/playing-around...
Beyond the obvious benefits of SDF fonts, it also has excellent and extensive layout, formatting and other useful features, like automatically scaling the text to fit in a given area.
The source code for the shaders and formatter and ugui integration is included, but not the atlas generator (although you might ask the developer if you need it -- he's engaged and helpful, and provides good support and quick fixes for Unity updates). It includes desktop shaders with many fancy features (outlines, drop shadows, beveling, glow, bump mapping, a surface shader that reacts to lighting and throws shadows into the world), and simplified optimized shaders for mobile.
I've used it (and read the source code to see how it works), and can testify that it works great on Unity's desktop, iOS, Android and WebGL backends. The code itself and the way it offloads so much work to the GPU is very beautiful and elegant.
TextMesh Pro - Unite 14 Demo: https://www.youtube.com/watch?v=q3ROZmdu65o
Here's a benchmark that compares 5000 crisp TextMesh Pro objects rendering at 70 FPS, versus 5000 fuzzy Unity text objects rendering at 42 FPS: https://www.youtube.com/watch?v=rdc8UkxuSZc
Here's the same benchmark compiled with Unity's WebGL backend, with the same number of 2500 static and 2500 dynamic text objects. Nowhere near the 70 FPS of native code of course, but not bad for WebGL and so many objects -- zoom in with the mouse wheel to see the text up close: http://donhopkins.com/home/TextMeshProBenchmark/
TextMesh Pro on the Unity Asset Store: https://www.assetstore.unity3d.com/en/#!/content/17662
Digital Native Studios home page: http://digitalnativestudios.com/
Performance. Even if an inefficient painting algorithm manages to paint at 60 FPS (which is very much not the case in, for example, common Web apps), improving its performance can free up the CPU to do more things and can reduce power consumption.
Think about it. If your GPU is sitting there idle, why not use it and free up the CPU to do other things? Today, most apps that aren't games are woefully underutilizing the GPU. Programs ideally should be utilizing all the silicon in your device while working so the work can be done faster and the hardware can enter low-power mode quicker.
For games specifically I'd prefer more work done on CPU and free up the GPU, provided the communication overhead isn't too much and latency doesn't become a problem.
Low power is less of a problem than thermal limiting for compute intensive work. I see about 90% variation in test times on my laptop for unit tests, for example.
So overall I see more application for this in apps than games, and only then when the tradeoffs are right. It's not a simple call.
Also, I care quite a bit about power draw because my customers on laptops and mobile are very sensitive to it. If my game is a battery hog, it doesn't get played as long or as often.
Plus, every implementation of subpixel hinting I've seen (and I've seen them all, on all three major desktop platforms), produces absolutely disgusting colour fringing.
Just lay off the ClearType(tm) kool-aid and do normal anti-aliasing. We don't do colour-fringing anti-aliasing for any other geometric primitive, I see no reason to implement a font-specific approach that only works if the user is colour-blind.
I can see questioning the utility for current 2d applications but this is really critical speed when the large amount of translation and scaling of many pages in a 3d environment is the expectation.
The css3d "portal" solution for webgl doesn't seem like it can get us very far at all...
I'd use this today in my webvr/gl project if it was source complete.
The CPU takes a long time to rasterize fonts. It only seems fast because the glyphs are cached. If you need text drawn in so many different sizes, like in that demo, the cache is not as effective.
Hinting was necessary back before good antialiasing existed but I don't think many renderers use it now. This technique renders each glyph differently according to its subpixel position - completely the opposite approach to hinting, which snaps features of the glyph to integer positions on the pixel grid.
http://dl.acm.org/citation.cfm?id=1111433
http://docdro.id/ynU32mg (Paper)
http://docdro.id/t6XiE97 (Slides)
It's a trade off between speed and variability, which for games (and most apps) speed trump variability (especially because of consistent design). But given generating a texture atlas is a "1 time task"; and recompiling the curves in the GPU is per frame in the paper, the next best thing I guess is off loading the glyph atlas generator to the GPU, not CPU, since the GPU maybe faster.
It's really awesome research!
There are issues with atlas of glyphs; so rather than putting the whole glyphs in atlas, he slices the glyphs in a grid, the grid being defined by the intersection of the underlying Bezier curves. And then he puts the grid pieces in atlas.
Yes, I understand the technique. Perhaps I should be more clear about my terminology. I'm using "atlas" in the narrow sense of "actual pieces of bitmap data stored in a texture". My point is that this technique should not be a substitute for atlasing in this sense, but could well be combined with it.
Super sharp lines when zoomed in may not seem like a big deal, but as you increase resolution you will reap the benefits since the same size text will be sharper.
This is isn't true. SDF textures can be "better" for "some" applications (like games that want to display stencils mostly at arbitrary zoom levels without losing a lot of quality) but they're also visibly lower-quality compared to glyphs rendered by freetype at each one of these zoom levels.
So, for an application that simply displays lots of text while infrequently changing the glyph size, SDF textures are not a good solution.
> We already know that SDF textures are better than straight texture atlases anyway.
Even that depends. SDFs are slow to construct and, because they have a special FS, you have to pay attention to the number of state changes. They do have the advantage (unlike this technique) of having a very simple FS--they can even be used in some fixed function hardware!--making it practical to rerasterize glyphs every frame.
> The technique is obviously practical in some respect (since it renders a pdf faster than anything else I've seen)
That says a lot about how existing PDF renderers are slow and nothing about what is the fastest thing to do on the GPU.
I'm not sure if you need the medium-level (font atlasing) if you have both lower-level (bezier-curve atlasing), and higher-level (pre-composited "tiles.")
This is one of the reasons I really like WebGL, actually: for GUI elements, you can usually skip rendering them within the game altogether, instead using plain HTML+CSS controls positioned above the <canvas>, where they'll be treated as separate "tiles" that aren't dirtied by whatever your game is doing. Imagine: relatively-static fully-RGBA-translucent stuff sitting on top of your animating viewport, composited into a final image each frame, "for free."
Assuming it takes twice as long to draw a glyph into an atlas but half as long to draw it once it's in the atlas, an atlas that starts completely empty will speed up text drawing for your comment by 66% (735 divided by 50 * 2 + (735 - 50) * 0.5, which is 442.5). And this underestimates the savings by a lot (drawing glyphs from an atlas is much more than two times as fast, and some glyphs are probably in the atlas already).
• Most frames won't involve zooming or text-change—and therefore will be handled at "zero" time-cost by re-using the pre-composited tile.
• The frames that do involve text-change (but not zoom) can benefit from font-atlasing.
• The frames that involve discrete zoom, where text goes from one integral font size (represented in the atlas) to another integral font size (also in the atlas) can also benefit from font-atlasing.
• The frames that involve continuous zoom—such as in this demo—can't benefit from font-atlasing, because the frames where the font is e.g. 7.037773pt large can't be served from the atlas.
Keep in mind that the only reason we do discrete zoom-levels in PDF readers, text editors, etc. is that font-atlasing is "the optimization we have", and it calls for discrete zoom levels. When the UX affordance is for continuous zoom—think about pinch-to-zoom in Mobile Safari, or about zooming into Google Earth—font-atlasing doesn't and can't work. But this technique can.
Now, in games in particular, "continuous zoom" is a pretty rare edge-case. GUI elements don't tend to get zoomed into[1].
And "in-VR" elements, like street signs in an FPS, aren't usually designed as a bunch of geometry[2], but rather as a very high-detail flat texture that maybe gets mipmapped from far away. "Zooming" by approaching a street sign doesn't make the fonts on it re-render themselves; there are no fonts on it, they're just baked into an image.
---
[1] Well, maybe; I could see a case for continuous zoom of the fonts—and other GUI geometry—plastered over units in RTS games. "Pan up" with units selected, they stay selected but their unit names+stats get smaller and smaller and eventually get obscured by draw-distance and atmosphere effects, just as if they were actually "there" in the scene instead of part of the GUI. (Makes me think of the windows in the TV show ReBoot.)
[2] This does make me imagine a Blade Runner-esque game full of holographic signs, though. Those could definitely benefit from this approach.)
I have no argument against that kind of font-atlasing as a process; it's fairly more stateless, more like a secret cache inside the black box of the glyph-rendering kernel.
I think this is actually a harmful approach in most cases--all it does is reduce the number of vertices and slightly reduces overdraw in exchange for a lot more memory consumption and a lot more draw calls. Overdraw can be reduced in other ways, such as sorting front to back and taking advantage of early Z for opaque content. Web pages and PDFs are so totally not bound on either ROPs or VS, and are very sensitive to draw call count, that caching tiles to minimize overdraw is almost never worth it.
But this is somewhat off topic.
> I'm not sure if you need the medium-level (font atlasing) if you have both lower-level (bezier-curve atlasing), and higher-level (pre-composited "tiles.")
You definitely do. You're sacrificing a lot of performance for the rendering of each tile if you don't do that.
I don't think we're imagining the same things here. Think less '3D landscape with text floating in it for some reason' (e.g. SecondLife), and more '3D landscape with labels applied to it' (e.g. Google Earth). Nothing from the "VR" layer obscures the "labels" layer; the labels float above everything, as if part of the GUI, but following the same scale as the things they label.
> PDFs
I'm actually unsure why a (non-interactive) PDF page, displayed at 100% zoom, can't be baked into a single extremely-high-resolution tile—applied as a texture to a single GL rectangle with plain-old lanzcos downsampling. That'd work for the overwhelming majority of the frames.
You'd only need to re-render the tile when you zoom in enough that it'd look bad; effectively, you could think of the PDF page as coming from a mipmapped texture, but where you only have the current size and will just-in-time rerender when the zoom-factor changes. And then, when the zoom factor makes the PDF not fit on the screen, the "tile" would be the screen size—so you'd re-render sibling tiles when the user moves.
Or, in other words: PDFs can be rendered exactly the way the image tiles on a mapping website are rendered, can't they? Doesn't that approach win over re-rendering all the text from a font-atlas every frame? And if not, why do browser rendering engines use tiles? (This isn't a rhetorical question to make my point; I'm not a graphics dev and I honestly don't see why.)
Sure, but that uses a lot of memory, and initial pageload is going to be really slow. Especially when you consider you need to generate mipmaps if it's really high resolution, or else blow out your GPU's L1/L2 cache in FS execution.
> Doesn't that approach win over re-rendering all the text from a font-atlas every frame?
Not significantly, in my experience. Assume, in the simple case, that you're just rendering a bunch of text with no overlap. In that case, the only thing that tiles buy you over rerendering all the text from an atlas is decreased vertex count. You're still touching the same number of pixels in the FS and ROP units either way: you just do so with more vertices in the texture atlas case and fewer vertices in the tiling case. Now consider that creating and maintaining the tiles has costs over maintaining the glyph atlas (which you have to do either way): you have the memory of the tiles, the overhead of creating and switching FBOs, lots of little textures to keep around (which in naive implementations results in tons of state changes), and extra draw calls to render the tiles after rendering the content.
> why do browser rendering engines use tiles?
Mostly because it was an easy way to fit GPU-accelerated panning and zooming into the existing, originally CPU-based, rendering architectures that browser engines used (and still largely use). Tiling was popularized by Mobile Safari on the iPhone in 2007 as an easy way to avoid repainting the entire page on the CPU every frame when the user performed touch gestures. I don't think it's necessarily a globally optimal decision.
The demo runs very poorly on my 15" MBP (late 2013, Intel Iris Pro). I'm not sure if it's just the sheer number of glyphs being rendered at once, or whether it's something specific that this GPU is having trouble with.
Some related links for those interested in WebGL text rendering:
1 - https://github.com/Jam3/three-bmfont-text
2 - http://mattdesl.svbtle.com/material-design-on-the-gpu
3 - https://github.com/mattdesl/text-modules
http://w3.impa.br/~diego/projects/GanEtAl14/
Rotating the line samples is a rather interesting idea. Seems like that would converge to the equivalent of a convolution with a radial tent filter.
[1] http://w3.impa.br/~diego/publications/NehHop08.pdf
haha, that's clever ^^ EDIT: seriously though, those are some Paint skills, I didn't even notice until I saw the "source" text below it.
https://goo.gl/photos/3EXv2t4YZRxmyNPe8
For example I would try converting the bezier representation into an implicit HRBF representation (here is a good explaination in 3D http://rodolphe-vaillant.fr/?e=12). This representation should be much easier to process on the GPU - checking how much the point is inside/outside the glyph should only be one matrix multiplication which would make the computation in the actual shader really really fast.