70 comments

[ 3.3 ms ] story [ 119 ms ] thread
I've never fully understood just what exactly the problem is?

Many apps ranging from vlc to Totem seem to hardware accelerate just fine, all without seeming to know or care which specific hardware I'm using.

What makes it so much more difficult for FF?

DRM probably.
YouTube etc. don't have DRM in the video they serve, as far as I know.

Hardware accelerated decoding of DRM'ed video is a completely different story, and the only browser I have seen do that on my machine even on Windows 10 is MS Edge. But there's a lot of non-DRM'ed video on the web, and getting it decoded in hardware on Linux isn't a given either.

Accelerated video decoding really wants to decode into a hardware texture and then render directly from that texture, and copy/move the data around as little as possible; ideally, any further processing happens entirely on the GPU.

Most video players have a defined pipeline from video decode to display; they may have some things they can do to the video before displaying it, but they can handle accelerated video.

Among other things, HTML5 video allows JavaScript/WASM code to get at individual frames and do further processing prior to display. It's not as simple as "decode video and render it".

I see two parallel processes here: "decode video and render it" and interpret JavaScript/WASM at every frame.
The problem is that code runs between decode and render, and wherever possible the video data needs to stay on the GPU.
All modern browsers have done GPU compositing for a while now, so if anything GPU decode to a texture is the nicer path anyway.

As for HTML5 video allowing JS to get individual pixels: no one made any performance guarantees. The expectation is if you use these APIs (don't!) that you will incur at minimum a full framebuffer copy into a CPU shadow buffer, and likely hog your AVX-esque units that have to decode from some GPU-specific format into raster or do colorspace stuff.

Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case. Your hardware video decoder is likely separate silicon that prefers to decode into its own YUV-happy format, and that format is in turn more or less only guaranteed to be understood by the hardware compositor - the GPU might not be able to share it. So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming. This is well understood by the people that build Android and iOS, not so much web developers.

I think the expectation is that WebGL is used to manipulate video frames.
> Note that in this thread we equate hardware video decoding with GPU, but on mobile or mobile-derived hardware this is not at all the case.

It is the same for desktop hardware. The encode/decode block is extra silicon on the graphics card or extra block on the iGPU, not GPU itself. There have been some experiments with hybrid codecs - i.e. using shaders to for decoding, for example VP9 on pre-Raven Ridge AMD hardware (windows only) - but it was only stop gap solution and the next generation had proper decoder and encoder.

> So if you start trying to do anything with video beyond displaying it in it's own plane, you can very easily force the hardware into a path that will be slow and energy consuming

Mobile chips have their own bunch of problems; if you have too many planes (where too many is defined by the specific chip), you are getting into not enough memory bandwidth territory.

Maybe this is just unwarranted speculation, but I remember a period many years ago (maybe 10+?) when Firefox was very unstable when hardware acceleration was enabled, even on Windows, although it depended on what exactly your hardware was. Eventually the quality of the drivers on Windows made hardware acceleration less unstable on Windows, but that same progress did not happen on Linux (or was happening much slower) so for a while Firefox gave up on Linux hardware acceleration.

Anyways, I think it is a case where they were burned before trying to compensate for different levels of driver quality for hardware acceleration and so they are more skittish now and want to proceed slowly.

Of course, I have no sources to cite and this is based on half-remembered blogposts from a decade ago, so take this with a grain of salt.

You can read about the specific reasons that driver versions were blocked on the Mozilla Wiki https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drive... Most are due to the drivers causing crashes or security issues in Firefox.

I think a broader answer to your question is that most apps accept that if a driver causes a crash or security problem, it's not their problem and it's up to the user (and driver developer) to remediate it. Firefox has chosen to attempt software fallback to avoid the issues. There are obviously many trade-offs for good and ill with this approach.

It still seems, at least based on anecdata and a hunch, that decode acceleration through VA-API works more often in VLC than in Firefox.

VLC on Linux systematically hardware decodes H.264 on my aging integrated GPU, but I haven't got it to work on Firefox ever.

(Most web video nowadays is probably VP9 or something, which my GPU doesn't decode, unless you coerce YouTube et al. to serve H.264 using a browser extension, but that's another story; even if I do that, it still doesn't get decoded in hardware.)

I'm sure the browser developers have reasons for being conservative about enabling VA-API acceleration, ranging from possible issue with their codebase to buggy drivers, but it does seem like there are way more problems than e.g. in VLC.

I think legacy contributed most.

On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel. Here’s a proof, I did it alone spending a month or so part time, and resource use is lower than VLC’s despite my code is in .NET: https://github.com/Const-me/Vrmac/tree/master/VrmacVideo

However, that wasn’t always the case. Until recently software had to fallback to software decoders, and implement other workarounds. Software developers are reluctant to throw away old code and do something completely different.

1. They spent much time on that old code (sunk cost fallacy).

2. Old code usually works more often than it doesn’t (don’t fix things which aren’t broken).

3. Massive refactor of large software is very expensive in general. Moving stuff from CPU to GPU is going to cause large changes in many parts. You can’t usually afford to copy uncompressed video frames from VRAM back to system RAM and call it a day, that would be too much bandwidth. Instead, you have to change a lot of other code to handle GPU textures instead of system RAM images.

> On modern Linux, it’s possible, and not even terribly hard, to implement hardware-accelerated video playback without any libraries or frameworks, by directly consuming V4L2 API exposed by the kernel

Do you mean V4L2 M2M decoding? That's only supported on a very small subset of devices, mostly ARM devices. Other devices might use VA-API (Intel), VDPAU (NVIDIA proprietary), or something from the OpenMAX standard (some AMD drivers).

M2M is far from easy to handle in a lot of circumstances, though it depends on the kinds of devices you use. I don't really know why you're going into a screed about "sunk cost fallacy".

> Other devices might use VA-API (Intel), VDPAU (NVIDIA proprietary), or something from the OpenMAX standard (some AMD drivers).

Nothing prevents Firefox from consuming these things. The majority of time I’ve spent on that library was not on V4L2, it was containers parsing (they are incredibly complicated these days, as I found out in the process), bitstream and parameter sets parsing, also audio codecs shenanigans: https://github.com/Const-me/DtsDecoder

> M2M is far from easy to handle in a lot of circumstances

I know, and I never said it was trivially simple. But it’s not terribly complicated either.

I have implemented that alone, for fun, spending a month part time. My implementation plays all h264 video files on my hardrives (more than 1TB of them). Because of that, I would expect better from a company with 9-digit annual revenue. BTW I had no prior experience with Linux multimedia, albeit I did a lot of related things on Windows, mostly with Media Foundation.

> why you're going into a screed about "sunk cost fallacy"

I have no idea how Firefox is developed. However, I’ve been developing other software for living for couple decades now, worked with many codebases including very old ones, and I think I know a few things about software development in general. That was just my educated guess.

BTW, you can strip down ffmpeg build to a pretty barebones level, so that it only includes what you need.
True. But I didn’t want to violate their license, nor publish my code under GPL.

There were other considerations as well. For example, if I would have forked or re-built ffmpeg, I would then have to support my version. That’s something I try to avoid when I can. I think OS maintainers are doing much better job updating these third-party packages than I can possibly do myself, at least for an unpaid hobby project.

Very cool, thanks for sharing! I have some pretty old hardware so lower resource use is always good, will be trying your player out!
Thank you for sharing your project!

Very cool to see a combination of low-level Linux kernel APIs and .NET.

True hacker spirit right there!

Thanks! BTW I used more than V4L2.

ALSA from libasound.so.2, there: https://github.com/Const-me/Vrmac/blob/master/VrmacVideo/IO/... and the rest of libasound.*.cs files in that folder (parts of the same static class). Audio output is harder than necessary on all OSes I developer for, including Windows. From the internets I got an impression that at least on RPi, ALSA PCM is more reliable than the rest of them. Also doesn’t depend on any desktop components, important because the library can also be used without any desktop environment, rendering on a physical display.

Message queues from librt.so https://github.com/Const-me/Vrmac/blob/master/VrmacVideo/IO/... .NET has equivalent stuff already. I needed poll() to get notified about multiple things not just queued messages, as both V4L2 and ALSA expose poll()-compatible file handles in their respective APIs.

The problem is compositing with the rest of the web page. In VLC you don't have much complexity on top of or around the video (a few buttons, sometimes subtitles or a volume overlay), whereas around a video in a browser you have THE INTERWEBS (insert scream emoji here), which themselves also go through hardware acceleration.

Martin Stransky (the implementer of the feature) explains this in one of the related bugzilla bugs, if you want details.

They used to have playback via gstreamer (and before that mplayer plugin). First they castrated it for political reasons (you could only use certain codecs, officially so that codecs wouldn't proliferate and website authors wouldn't start to depend on them) and then they removed it altogether. Now it uses a bundled ffmpeg. I think this switch is one reason it took so long.

Maybe a controversial opinion, but I think a browser should not be in the business of delivering video codecs. The operating system (distribution) should have one set of codecs that is tested, optimized, and accelerated, and all apps should be able to use them.

That is how it works on most desktop platforms with their video frameworks.

Naturally Linux desktop being as it is, this solution will never work.

You mean "naturally, since many of these codecs are completely or partially proprietary and/or not publically documented, this solution will not work on platforms that focus on non-proprietary open standards"?
Idealism has a price, but then there is the NIH syndrome on top of that.
This is just one of the times where you’d just have to read the source. No documentation usually means I start diving into the source code.
That attitude/capability is closely related to a job interview question I tend to use. "Could you tell me about a time you delved into the source code of some software product or library to fix a problem you faced?"

(My pet theory: I think often it's just the attitude that is missing. So much can be accompished with the right attitude.)

It's impossible to get anything done otherwise, and some cursory investigation is usually a lot faster than emailing some overworked maintainer with nothing but "plz help".
I ran into similar issues with driver blocklists on both Linux and Windows. There are workarounds, such as setting MOZ_GFX_SPOOF_* environment variables, but even the suggestions on the Mozilla Wiki at https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drive... no longer work, because the spoofed driver version (8.15.10.2302) was blocked in https://bugzilla.mozilla.org/843273 It's a mess.

I can understand the desire to avoid using drivers which cause a poor user experience. I don't envy the task of trying to implement and maintain a system to achieve that. I do wish it wasn't so painful and confusing for users when the behavior doesn't match our expectations.

IDK why this post is so negative. The post says it's the first release with that feature, and that things are still improving. I'm happy that it's being worked on at all and maybe a couple more releases down the road it's available to me as well. Mozilla/Firefox contributors are adding a feature here in the open but it turn out it's very complicated to add it.
Right, and my feeling is that any feature that requires you to mess around with about:config is inherently an alpha/beta feature. Expecting it to work properly on every setup is expecting too much; once it's enabled by default, then you should expect it to work out of the box, unless your hardware/OS/driver combo has been explicitly blocked due to issues.
Hardware accelerated video decoding is a feature that is and has been critical for battery life on laptops for years.

ffmpeg has supported this for forever and yet firefox doesn't. It's bizarre.

Do you believe enabling hardware video decoding in Firefox was a trivial effort?
What Firefox 78-80 brought was hardware video decoding under Linux, not hardware video decoding in general.

Under Windows and MacOS, it has been implemented years ago. Under Linux, it wasn't. Linux has API for video acceleration and players like VLC, mpv or Kodi have been supporting it for years, as well as ffmpeg, just browsers ignored it.

I'm not going to belittle the effort, quite opposite, but it is telling when this feature has been implemented by Redhat employee and not Mozilla.

Should have been, after all Flash was doing it 10 years ago.

Since Flash plugins were killed, Linux has been a mess for hardware accelerated video on the Web.

Not killing gstreamer way back when would've helped.
What if:

Mozilla is a spooky company. They covardly pretend to be pro-freedom, but in reality they have a shady agreement with Google to look like an alternative, but not better.

This is well known pattern. For example what if Nvidia has agreement with Microsoft to provide mediocre experience to average Desktop Linux user.

Then explain the ~five year period where the bulk of their revenue came from Yahoo. Just a ridiculous idea.
From what I can gather, they're using an Intel card, which I don't think supports VA-API on MPEG-4 videos. Further, I can't tell which distro they use but I'm not sure when the Intel work was added. Their distro provided version would likely iron out some of these issues.
That Intel 8700K definitely supports VA-API with H.264, HEVC and VP9, decoding and encoding. An extra driver is (intel-vaapi-driver) is needed, it is not distributed and installed together with Mesa as AMD driver is.
>That Intel 8700K definitely supports VA-API with H.264, HEVC and VP9, decoding and encoding

It does not support MPEG-4, or it's disabled by default. I don't know how widespread that format is, but I believe it was developed with streaming in mind.

If you mean MPEG 4 ASP (i.e. part 2, as opposed to H.264, which is also MPEG-4, but part 10) I don't believe any Intel hardware did ever support it. Today, it is obsolete.
(comment deleted)
What Firefox really needs is a little icon (preferably displayed prominently in the toolbar) that appears when videos are not being decoded in hardware. I'm sure they've lost users who just think Firefox is "super laggy with YouTube" or that "Firefox really eats battery". An indication that something is wrong and the browser knows about it prevents the assumption that the browser is just buggy/slow/etc; the word-of-mouth changes from "Firefox sucks" to "Firefox told me it couldn't hardware accelerate my videos or something?" Still not positive, but potential users will hear that Firefox might not work with their hardware, rather than that Firefox is buggy, slow, and unreliable.

And I know technical users would appreciate it. I still don't know what is and isn't getting hardware video decode, or whether I actually need a particular about:config flag or system package, and I have tried hard to figure out.

You can go icon blind pretty easily. That being said I know my computer doesn't have VP8/9 hardware support so I installed the h264ify plugin for Firefox so YouTube serves me H.264 videos instead. Maybe Mozilla needs to detect hardware support and offer to install plugins like that or give a setup screen (newer video formats with more features vs old video formats with better battery/less fans)
The default assumption today and in the past on Linux is that there is zero hardware accelerated video display in any browser.
Which is why macOS, Windows and ChromeOS are the platforms for anyone who care about video hardware acceleration without messing with configurations.
ChromeOS is Linux.
I mean, sure, at a kernel level. It has its own GUI rendering stack (Freon) rather than the usual X11/Wayland of "normal" Linux, and it's Freon that has their acceleration support.
Not really, it is an implementation detail not exposed to users.

Even Crostini relies on containers and virtualization to expose a second kernel, WSL2 style.

I have a GTX 1080 which performs wonderfully under Linux (Ubuntu) but I still have to go into Firefox and Chrome browser flags to force hardware accelerated GPU rendering on. Once that's enabled they work as well as Windows. New users may benefit if browsers prompted users to try forcing hardware acceleration and have the ability to easily disable it, or do so automatically if it detects crashes, e.g. where location, microphone and video settings can be toggled.
Nitpick: Browsers on Linux have been using hardware video display acceleration since the Netscape 0.9 / Mosaic days, but video codec acceleration has been spotty.
I did manage to get it working but it has a ton of issues with Youtube vp9 decoding despite Chromium's years-old implementation working fine.

This is complex stuff. I spent a day last week trying to figure out why hevc vaapi patches for OBS were failing with mkv containers but not mp4 ones. Its just really ugly C apis all the way down.

Yes, the UX is raw for now; hopefully Mozilla will make it simpler to enable after a few versions of testing and confirming it doesn't cause crashes for half the linux userbase. It makes sense to do little early advertising of such a feature, and to initially hide it behind flags & env. vars only within reach of "power users", because these fresh-out-of-the-oven bits are a complex assembly with heavy crash potential, depending on your hardware and drivers up-to-date-ness. As the implementer, you want your initial users to know what they are doing, and you want to maximize the chances they're technically proficient enough to gather the mountain of gpu/driver/environment debug info you'll need for a bug report to be actionable.

In the meantime, to save you rummaging around bugzilla, follow https://wiki.archlinux.org/index.php/Firefox#Hardware_video_... , it's the most constantly {up-to-date, concise, reliable} source you'll find. You don't have to be using Arch, the information contained in this section applies to any Linux/Firefox distro.

Note that Firefox isn't the only browser to struggle with it; see https://wiki.archlinux.org/index.php/Chromium#Hardware_video... for the similarly non-obvious instructions in Chromium (and by the way, currently it's plain impossible in Chrome).

Props to Martin Stransky from RedHat who implemented this for both Wayland and Xorg during the last months.

do what chrome does, detect crash and fallback. what are sandboxes even for if you can't crash once in a while without whining too much about it...
I doubt things are that simple.

See my before-last paragraph above: no, regarding hardware video acceleration, even Chrome doesn't do what you describe. What you describe stands for non-video content crashes, which isn't what's being discussed here. Could the technique you mention be applied to video gpu crashes? I personally don't know and I'd lean towards "nope" (else Mozilla & Google engineers wouldn't be so exceedingly cautious about it). If anyone is knowledgeable enough to precise / contradict this, please chime in.

At any rate, even Chrome is built with zero hardware video acceleration support on Linux (hardware acceleration yes, hardware video acceleration no). Chromium does offer the option, just as hidden behind flags as Firefox, and until last month you had to build it yourself with a VAAPI patch left unmerged for years by Chromium devs.

That sounds nice in theory, but only works if you can (reliably) detect the cause of a crash. This is one of those things that sound much easier than it actually is.
It's not possible go get VA-API to work in FF 88 on X11 or Wayland because VA-API was broken in Firefox 79. The fix is in FF 81.
It seems consistent to what I experimented on Arch LTS/Wayland since FF 79. I have very frequent video player crashes when enabled (like on YT).
Not sure I follow, did you possibly mean:

It's not possible to get VA-API to work in FF 80 on X11 or Wayland because VA-API has been broken since FF 79. The fix will be in FF 81.

Just guessing, I have not looked into the topic except in this thread. What would be the bugzilla id for that?

Replace the '88' with '80'.

VA-API hardware video acceleration was broken in Firefox 79 and won't be fixed until Firefox 81. Therefore, it's not possible to make VA-API work, under either Wayland or X11, in Firefox 80.

See: https://bugzilla.mozilla.org/show_bug.cgi?id=1656436

So to put it cynically: In FF 80 the situation is crystal clear, not at all confusing as the original article claims: It doesn't work.

In FF 81 it will be confusing to the non-experienced again: You need to fiddle with config and environment variables. Well, unless a new bug comes up...

For me, on Arch LTS/wayland/gnome Firefox's video hardware acceleration became buggy since Firefoxy 79 (I followed the Arch wiki recommendations). I had to disable the gfx renderer. And cpu consumption is only marginally better than Chromium. When enabled, cpu consumption is very low <10% watching 720p YT videos, between 10-20% for 1080p but the video player crashes very often.
Yeah, since the anti-Flash movement has won, Linux is even less interesting for anyone wanting an out of the box experience regarding video hardware acceleration for Web browsers.
What would be a poor man's reliable test to see whether acceleration is actually in use or not? If you could just reliably switch it on and off the difference should be obvious in top, possibly by switching to threads view by pressing H (instead of the processes default). But if you cannot, there is nothing to compare.
Go to about:support and look in the Graphics section.
not at my computer now, but if I remember right, that shows you whether HW acceleration could in theory work for some video or not at all. The second case is a clear one. But in the first case you still don't know whether it is actually used for a certain video or not. I understand there can be many parameters preventing usage.