> In addition, given the high industry demand for AV1 codec support, an AV1 decode extension release is imminent, with an AV1 encode extension development also underway.
That's good.
By the way, mpv already supports Vulkan video decoding:
On Windows Vulkan video would work. But not on macOS (that mpv thread mentions it).
On Linux it should be a better option than VAAPI once drivers will support it. Since Firefox is using ffmpeg anyway, it probably shouldn't be too hard to make it an option.
VAAPI is also using the GPU. The fixed-function video decoders are on the GPUs, not the CPU. That just happens to be using the iGPU on Intel so you don't need a discreet GPU necessarily, but this is also missing if you have a CPU-only sku like the -f suffix parts
The only difference is the vulkan spec is _in theory_ cross-platform. However, since MacOS doesn't support Vulkan and MoltenVK doesn't implement this extension (and probably won't), and this extension also isn't implemented on Android, then your scope of actual availability is quite pathetic. So it probably doesn't make sense to move off of VAAPI regardless unless you really want Linux & Windows to share a codepath, even though you have something OS-specific anyways for MacOS/iOS & Android.
It's unhelpful to bundle this into the "GPU" at large and is only true for desktop systems anyway. On mobile, video decoding, GPU and display pipeline are typically different hardware blocks often made by different companies.
I might just be bad at Google-fu, but my understanding is that -F processors don’t have Quick Sync, which seems to imply that the iGPU is required in some capacity.
Vulkan Video Extensions use the same physical hardware as VAAPI, but with a little bit less of forcing you to do codec-specific nonsense. It should be better in that it's much easier to get a zero-copy Vulkan texture from the decoded frame.
The GPU has also the fixed function video encode like integrated CPUs. Both VAAPI and Vulkan Video address the same hardware. Differences are in the API itself (with Vulkan being more explicit) and potentially in which vendors (or OSS driver developers) support which API in what degree.
That's right. "Software decoding" means the decoding algorithms run on the CPU. (As opposed to "hardware decoding" which typically means the work is done by some fixed-function video units on e.g. a GPU)
I think not being tied to VAAPI reduces the chain of development (like you don't need to wait for Intel to add something first since they are the primary developers of VAAPI in general).
Beware on Windows, out-of-the-box Microsoft will ship e.g. nvidia drivers that have DirectX 12, but not Vulkan. This can lead to confusing user situations ('DirectX 12 games work fine.. but not Vulkan') and probably means DXVA would be a better choice if targeting Windows
In Windows 10 and above, windows update provides a default graphics driver for each IHV, which comes from the manufacturer, has DirectX support, but doesn't have Vulkan.
Often nvidia users will try to debug it, and discover 'the vulkan checkbox is off but I have the latest driver version!' in gpuZ[0][1] - but it happens across all IHVs, AMD, Intel, etc. I was told by someone in the industry the default Microsoft drivers across all IHVs indeed have DirectX, but not Vulkan.
To make matters more annoying, Windows Update often _overwrites_ the user's driver which they installed from the manufacturer directly, leading to Vulkan becoming broken overnight[2][3]
Easy to find multiple instances of this happening[4][5][6], I became aware of it because of someone asking on /r/vulkan a few weeks ago.
The Vulkan implementation is still there. You just have to install the redistributable. (see https://vulkan.lunarg.com/sdk/home - Runtime)
There's no separate Vulkan-less version of the drivers AFAIK - it's just that the installation program of the manufacturer doesn't run so that there's no opportunity to install the Vulkan runtime.
Thanks for the info, cool to know you can install that separately.
My original point remains: out of the box, many Windows users find they have the 'latest drivers' for their graphics card, find that DirectX 12 is working fine on their system, while simultaneously finding Vulkan does not.
For games, it's fine to require users to fix their Vulkan drivers. But if you're making a video player and want to leverage hardware decoding.. I think this is an aspect you should be concerned about and aware of.
But why would Firefox go out of its way to use Vulkan on Windows or macOS when it doesn’t have to because it can just use the native OS functionality and let the OS worry about implementation details?
> Now we need OBS and Firefox supporting Vulkan video too.
Why? They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos. Why use Vulkan video extensions? It seems like it just complicates things since MacOS/iOS and Android are unlikely to get these and Vulkan on Windows' is a second-class citizen as well. Not to mention the feature & API availability is lagging very far behind native APIs that they're already using.
It's not a bad extension necessarily, seems like it'd be a great option for games that want to integrate some cutscenes. But if you have an existing solution, it also doesn't seem valuable to migrate to this, either.
Firefox still has no support for hardware video decode on Linux. There's all sorts of shims for it, but I've never managed to get them functioning. Maybe Vulkan video decode could be the one agnostic API to rule them all this time, especially if it's more cross-platform than the current situation.
How can I confirm whether Firefox is actually using hardware decode for videos on my machine for a given codec? Is there a magic keyword I should search for within the about:support page, or is it recorded somewhere else? Mine shows for instance "VP9_HW_DECODE default available", does this means it's using hardware decode for VP9, or merely that it might be possible for my Firefox version?
We've put a dedicated section called Media in about:support, it has decoding capabilities and other things such as audio IO informations.
If you find that it is not accurate, e.g. by cross-checking via other means, please open a ticket at https://bugzilla.mozilla.org/enter_bug.cgi, component "Audio/Video".
> We've put a dedicated section called Media in about:support, it has decoding capabilities
I see, I hadn't found it when searching for the codec name because it only said something like "Information not available. Try again after playing a video." After opening a random YouTube video and refreshing about:support, it did show the hardware decoding information, and it was as I had expected given the hardware on this computer.
Whilst Firefox may support hardware video decoding, Mesa since March 2022 disables patent encumbered codecs by default[1], and distributions such as Fedora and OpenSuse do not explicitly enable these patent encumbered codecs to avoid possible legal problems. Even Gentoo (built from source code by the user) requires the user to explicitly enable a USE flag (proprietary-codes) to use patent encumbered codecs.[2]
The thought process is that AMD, NVIDIA, Intel and the likes are not providing a patent license with their hardware.[3] They are instead just supplying part of an overall system that together with operating system kernel, display manager software, video player software, etc allows the decoding and encoding of patent encumbered video files. Open source software projects and distributions are concerned they'd be found to be infringing patents by enabling a complete solution out-of-the-box. Hence they put some hurdles in place so that a user has to go out of their way to separately piece together the various parts to form a complete system capable of encoding and decoding patent encumbered codecs.
edit: To clarify, if your Intel or AMD GPU/APU supports a patent-free codec such as AV1 (most GPUs/APUs available for sale?), Firefox on a standard Linux distribution will use hardware video decoding out of the box by default for the patent-free codec. The issue is really one of whether you're sourcing content from a provider that uses a good choice of codec like AV1. The good news is that patent trolls are doing a good job of pushing laggard content providers down this path.[4]
Which is of course a bullshit reason because Linux distributions don't ship the patented hardware decoders (because they are hardwared) so the user already has to assemble the complete system.
> a provider that uses a good choice of codec like AV1
Much more limited HW support and slow as molasses encoders. Not exactly a good choice for most.
My bad, that's what I get for commenting before bed. I meant to tack on an "on Nvidia" in there. AMD and Intel are well supported with VAAPI, but I've never managed to get any of the NV-to-VAAPI wrappers and shims working.
I think a very simple reason is I'd rather rely on Vulkan driver for faster iteration, than on VAAPI. It's historically moving much slower since it first needs Intel to pull weight, then AMD to implement their side for example.
> They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos
One vendor specific API, not "various OS' native APIs".
Firefox currently supports hardware video decoding with Intel's vendor specific VA-API only on Linux, which is not supported by NVIDIA. (A third-party VA-API to NVDEC translation layer for Linux does exist on GitHub, nvidia-vaapi-driver, but it's not yet reliable as the officially supported VDPAU or NVDEC, and is not included in official linux package repositories.)
Intel has VA-API, AMD has AMF, and NVIDIA has VDPAU which is being replaced by NVDEC/NVENC.
The idea behind Vulkan Video Extensions is to have a vendor independent and cross-platform video API.
> No, Firefox currently supports hardware video decoding with Intel's vendor specific VA-API only, which is not supported by NVIDIA.
Intel is behind VA-API originally, but I don't think it's fair to say it's a vendor specific API anymore. It's supported by the open source drivers for GPUs from all 3 vendors. It's just that the open source drivers for Nvidia cards are not very practical and the proprietary drivers only support vdpau and nvdec/nvenc
You can make the same argument for VDPAU. AMD officially supports it, and there is an unofficial translation layer with limited capabilities for some Intel GPUs. Is VDPAU not a vendor specific API anymore then?
Intel, AMD and NVIDIA have their own vendor-specific video APIs, and even when they provide official support for the API of another vendor, it tends to expose a limited subset of the full functionality (like the list of available codecs and encoding features).
You are free to call these vendor specific APIs for what they are or something else, but the reality has been that there is no single video API officially supported by Intel, AMD and NVIDIA. This changed with Vulkan Video.
But Vulkan Video isn't just about desktop: mobile devices, Raspberry Pi, etc. are expected to get on board with it eventually, just like they did with Vulkan.
> It's supported by the open source drivers for GPUs from all 3 vendors.
Which 3 vendors are you referring to? Intel, AMD, and who?
> It's just that the open source drivers for Nvidia cards are not very practical and the proprietary drivers only support vdpau and nvdec/nvenc
Why are you bringing up open source drivers, and what is not practical? Both official open source drivers (open-gpu-kernel-module) and unofficial open source drivers (nouveau, through binary firmware) support VDPAU. However, NVIDIA's drivers (open source or binary) does not support VA-API.
What of it? So does nvidia-vaapi-driver. They're third party projects, that's very different from "supported by the vendor", and doesn't change the fact that NVIDIA as a vendor offers no support for VA-API.
> Which 3 vendors are you referring to? Intel, AMD, and who?
I either missed some of the other text in your post or it was added after I started to reply.
> Both official open source drivers (open-gpu-kernel-module)
This is not remotely close to being a complete graphics driver. Most of a GPU driver on Linux is in userspace and there is no official open source user space component.
> Why are you bringing up open source drivers, and what is not practical?
nouveau has never been practical for serious use due to poor performance and mediocre hardware support (as you noted). open-gpu-kernel-module is only practical when paired with a proprietary userspace driver.
Anyway, my original point in all this is that describing VA-API as an Intel vendor specific API is unfair given it has been well supported on AMD GPUs for a long time now and on nouveau it's supported as well as VDPAU (i.e. not very well as you note). I did not intend to imply that it was universal. I didn't even intend to imply that VDPAU is a vendor specific API (though as a decode-only API it's not really a complete replacement).
Intel tried to make va-api the standard for hardware encode and decode on Linux, Nvidia tried to make VDPAU the standard for hardware decode on Linux. Neither was entirely successful. By contrast, NVENC/NVDEC, AMF and the Intel Media SDK (and whatever they replaced this with) never had such ambitions.
> One vendor specific API, not "various OS' native APIs".
Incorrect. Firefox uses Windows Media Foundation, which is cross-vendor, on Windows. It uses MediaCodec on Android which is again cross-vendor. Presumably it uses whatever iOS' equivalent is as well.
It only uses VA-API on a single OS, Linux, and that's probably more a reflection on the media qualities (or lack thereof) of Linux as a whole. Maybe Vulkan video extensions will be the savior on Linux. Or maybe it won't because it won't be anyone's focus of investment since it's largely a Linux-only problem in the first place.
I'm always annoyed how any Linux media player or encoder needs to bring its own entire media operating system, down to each individual nut and bolt.
On Windows there's Windows Media Foundation and DirectShow that centrally manage everything and also support the "individual nut and bolt" approach. Android has its own central thing (MediaCodec?) that must be used. MacOS and iOS presumably have their own central manager (Quicktime?) too.
But Linux? It doesn't serve as an operating system for media. It's tremendously inconvenient as an admin/user rather than an evangelist.
You don't need to implement every nut and bolt in the application. Lot's of useful things can do the heavy lifting (Pipewire, ffmpeg, libplacebo, Mesa and so on). Linux isn't after calling it all using some uniform "DirectFoo" naming scheme, but tools are there.
Comparison is also invalid. Linux as a whole (not the kernel but OS experience) isn't controlled by some Big Brother who decides what and how it's done single mindedly. So such kind of composite result is somewhat expected.
Those who use it appreciate their efforts. You aren't using it, why are you even complaining especially with complete nonsense comments. Anti Linux shilling should be getting old.
> Android, ChromeOS and WebOS, have replaced most of my needs
Great! I really hope you're happy with that setup! It's your personal computer and by all means, do what works well for you. In the end that's always a personal thing that's different for everyone. Who am I to judge how you use your computer?
But maybe ... stop complaining about Linux desktop then? If you don't like it? This must be like the 3rd time I've seen these types of single-line dismissive "Linux will never win the desktop"-comment from you in the last few days. Just one line, little or no context, or explanation, and IMHO also zero value, and an entire discussion derailed.
This is just becoming disruptive. You don't need to say anything you know. Personally, I rather dislike a number of things, but you don't see me complaining about it with one-liners every chance I get – and when I do say something, at least I make sure it's something of some substance, when I feel it actually contributes. And I sure as hell don't go around complaining people are "children" for disagreeing.
Yeah on UNIX side, NeXTSTEP, Irix, Solaris had their own thing, as graphical workstation UNIXes, and were great.
Ideally that kind of thing would be part of GNOME, or KDE, but then there are those that rather keep using twm like experience, making GNU/Linux really only good for headless experiences, at least the UNIX/POSIX part is always there.
And when someone installs some obscure or outdated and vulnerable codec on these systems, it's then automatically exposed to all sorts of applications to exploit. Maybe Windows sandboxes that these days(?) It was definitely a problem in the past.
No perfect solutions here; both "system-wide codecs" and "every application brings their own codecs" have their own up- and downsides.
Besides, with ffmpeg and gstreamer the system-wide codecs paradigm also works on Linux.
This is one of those "it's different but it doesn't really matter much" type of things. Most people "just" install vlc or mpv or whatnot and things will "just work" for them, not really different from Windows. That it's technically slightly different is almost entirely transparent to the user.
What is "incorrect"? The full sentence that you conveniently chose to cut in the middle before quoting (apparently to fit into some pessimistic forecast about the significance of Linux desktop) reads
> Firefox currently supports hardware video decoding with Intel's vendor specific VA-API only on Linux, which is not supported by NVIDIA.
(emphasis added)
You further wrote:
> Firefox uses Windows Media Foundation, which is cross-vendor, on Windows. It uses MediaCodec on Android which is again cross-vendor.
And? None of those APIs are cross-platform. Vulkan Video will eventually allow developers (including Firefox developers) to write a single code path for video to cover a wide range of platforms and vendors (likely with the exception of walled gardens like Apple-land, although someone might find a way to support like via a wrapper like MoltenVk for Vulkan).
> The full sentence that you conveniently chose to cut in the middle before quoting (apparently to fit into some pessimistic forecast about the significance of Linux desktop) reads
What are you talking about? They didn't quote that sentence at all, and didn't cut in the middle of the sentence they quoted.
> And? None of those APIs are cross-platform.
Your original objection, the thing that got quoted, was about whether things are cross-vendor. That question is completely unrelated to whether things are cross-platform.
> You did say that, but it's not the part of your post they were responding to.
So if someone criticizes a portion of your statement which is already countered by your original full statement, you're not allowed to remind your full statement. What kind of logic is that?
My original post says the point of Vulkan Video is it will be cross-platform and cross-vendor. And gives one example of cross-vendor side of things on Linux.
Someone criticizes me by essentially saying "you are incorrect, that's only on Linux. Windows, Android and iOS have their own video APIs...". This "correction" is incorrect because I already said on Linux, and it goes on to actually reinforce the post that he is responding to by highlighting cross-platform side, which also is in the post he is responding to.
So, if you look at the full conversation, the criticism is self-contradictory. This is what I'm pointing out, but you are implying I'm not allowed to do that.
I disagree. When you fragment a statement in a way that changes its meaning and make a straw man out of it, people are justified in responding to it.
> So if someone criticizes a portion of your statement which is already countered by your original full statement, you're not allowed to remind your full statement. What kind of logic is that?
The other stuff in your comment did not "counter" what they said. You made statements about cross-vendor and cross-platform. They chose to only respond to one of those statements. That's not incorrect.
> This "correction" is incorrect because I already said on Linux
The first part of your comment specifically said "not "various OS' native APIs"". That goes beyond Linux. The later part of your comment was about Linux in particular, but your introduction was an overall statement that wasn't true.
> When you fragment a statement in a way that changes its meaning
They didn't. You misspoke and they didn't know what you actually meant.
And from your other post: > Obviously, I meant to say statement, not sentence, but I can't edit it anymore.
That was not obvious. They quoted an entire paragraph, and the subsequent paragraph does not change its meaning the way you're claiming it does.
What about systems where there is now hardware support for AV1? How would Vulkan implement that?
Also, maybe someone knows more about this, but I thought that most software like Firefox, MPV and VLC are using dav1d for AV1 decoding as it seems to be the most performant AV1 decoder (in Software) out there at the moment.
So to clarify, this still leverages the fixed function hardware video encoders present alongside many GPUs from eg. Intel/NVIDIA/AMD, and would not (for example) be possible to use on just a GPU itself (without said encoding hardware)?
It is for using GPU video related ASICs (same as VAAPI let's say). If you want to use shaders for some video filters, scaling and such - you don't need those extensions.
mpv for example can use Vulkan for both at the same time.
Interestingly, quite a bit of video decode isn't as parallelizable as you might think and isn't a great fit for GPUs. For instance the initial Huffman decoding of the stream is essentially an intrinsically sequential process.
Modern video codecs use motion prediction algorithms to encode the information between key frames which depends on the entire frame and the ones immediately preceding it.
You can encode the independent groups of pictures (the key frame and all the subsequent predictive frames until the next key frame) in parallel but at least with 4k video you hit memory bandwidth limitations quite quickly.
Also, to get best compression you don't want to do key frames at even intervals, but rather when it pays off (ie large changes between two frames, like a scene change).
The point was rather that it makes parallel encoding more difficult. With a fixed interval it is trivial, you just split the incoming stream based on the fixed rate, and let each thread work on separate intervals.
With adaptive key frame placement you don't know the intervals up front, and they might have wildly different lengths. Some might be hundreds of frames, some might be just a few.
I guess because information of pixels in a video are not independent in either time or space, while parallelization relies on the independence of either temporal or spatial axis.
What does that mean? A video is inherently a series of frames. And using that sequential nature is critical to achieving good compression.
However, most formats allow for some parallelism at the "macroblock" level - you can usually decode all 16x16 pixels simultaneously. To some extent you can decode macroblocks separately, but "intra prediction" requires you to have the ones above and to the left available.
It means that e.g. in H265 you can split the frame into tiles that are much larger than macroblocks: it's like the video is a set of n videos that are then composed to build a complete frame. This allows both encoding and decoding to work in parallel even inside a single frame. In addition, in VR 360 context, it allows decoding only frames the viewer can see.
It reduces the compression efficiency, but only a little, because the tiles will be quite large anyway, such as 6 tiles per frame (but can be more for 360 video applications). It can be limited by the number of decoding instances a piece of hardware can have concurrently.
> 6 tiles per frame (but can be more for 360 video applications)
Seems like for VR video, if it’s split up longitudinally you would only have to decode the tiles that are in the direction the user is looking at any given time, and just let the other bits pass by undecoded.
AV1 also has "tiles". This is basically slicing each frame into N separate frames and encoding each mostly separately. You lose a little efficiency (as you can't exploit cross-tile redundancy very well or at all) but unlock a lot of parallelism on both encode and decode.
During the video card shortage I was kind of lamenting that straight encoder cards just don't exist anymore. To get the best encoder acceleration you were stuff fighting with gamers and crytominers over the same video cards even if you didn't need the majority of the hardware for your task.
109 comments
[ 4.8 ms ] story [ 167 ms ] threadThat's good.
By the way, mpv already supports Vulkan video decoding:
https://github.com/mpv-player/mpv/issues/11739
Now we need OBS and Firefox supporting Vulkan video too.
On Linux it should be a better option than VAAPI once drivers will support it. Since Firefox is using ffmpeg anyway, it probably shouldn't be too hard to make it an option.
Decoding h264/h265 in fixed (CPU) hardware is much more efficient than on the GPU.
Don’t get me wrong, its still miles and miles better than software decode.
Intel CPUs without iGPU, like the F-series, do not support QSV.
The only difference is the vulkan spec is _in theory_ cross-platform. However, since MacOS doesn't support Vulkan and MoltenVK doesn't implement this extension (and probably won't), and this extension also isn't implemented on Android, then your scope of actual availability is quite pathetic. So it probably doesn't make sense to move off of VAAPI regardless unless you really want Linux & Windows to share a codepath, even though you have something OS-specific anyways for MacOS/iOS & Android.
Since Khronos appears to have decided to require manual DPB management, this cannot be implemented in HW on macOS by MoltenVK.
not a thing. at worst you get a situation where the ICD loader isn't installed
Often nvidia users will try to debug it, and discover 'the vulkan checkbox is off but I have the latest driver version!' in gpuZ[0][1] - but it happens across all IHVs, AMD, Intel, etc. I was told by someone in the industry the default Microsoft drivers across all IHVs indeed have DirectX, but not Vulkan.
To make matters more annoying, Windows Update often _overwrites_ the user's driver which they installed from the manufacturer directly, leading to Vulkan becoming broken overnight[2][3]
Easy to find multiple instances of this happening[4][5][6], I became aware of it because of someone asking on /r/vulkan a few weeks ago.
[0] https://i.imgur.com/qj3Rn9p.png https://rog-forum.asus.com/t5/rog-strix-series/nvidia-driver...
[1] https://forums.x-plane.org/index.php?/forums/topic/228597-vu...
[2] https://www.reddit.com/r/Amd/comments/1145ffp/windows_update...
[3] https://answers.microsoft.com/en-us/windows/forum/all/window...
[4] https://i.imgur.com/PXZRxYh.png
[5] https://www.reddit.com/r/pcmasterrace/comments/qhfesc/vulkan...
[6] https://www.reddit.com/r/techsupport/comments/110bytq/vulkan...
The Vulkan implementation is still there. You just have to install the redistributable. (see https://vulkan.lunarg.com/sdk/home - Runtime)
There's no separate Vulkan-less version of the drivers AFAIK - it's just that the installation program of the manufacturer doesn't run so that there's no opportunity to install the Vulkan runtime.
My original point remains: out of the box, many Windows users find they have the 'latest drivers' for their graphics card, find that DirectX 12 is working fine on their system, while simultaneously finding Vulkan does not.
For games, it's fine to require users to fix their Vulkan drivers. But if you're making a video player and want to leverage hardware decoding.. I think this is an aspect you should be concerned about and aware of.
As pointed out elsewhere, it's not possible on macOS either way. Apple are too stuck up not supporting Vulkan and it won't work with MoltenVK.
Why? They have perfectly great existing hardware decoder offloading APIs via the various OS' native APIs for videos. Why use Vulkan video extensions? It seems like it just complicates things since MacOS/iOS and Android are unlikely to get these and Vulkan on Windows' is a second-class citizen as well. Not to mention the feature & API availability is lagging very far behind native APIs that they're already using.
It's not a bad extension necessarily, seems like it'd be a great option for games that want to integrate some cutscenes. But if you have an existing solution, it also doesn't seem valuable to migrate to this, either.
That's not true at all. See:
https://www.omgubuntu.co.uk/2023/07/firefox-115-intel-gpu-vi... https://www.phoronix.com/news/Mozilla-Firefox-115 https://www.omglinux.com/firefox-hardware-acceleration-raspb...
Something like this (assuming your GPU index is 0):
You should see there: If GPU accelerated video is being played.I think you might need to set this flag to true in Firefox's about:config (or it might be not needed anymore):
If you find that it is not accurate, e.g. by cross-checking via other means, please open a ticket at https://bugzilla.mozilla.org/enter_bug.cgi, component "Audio/Video".
I see, I hadn't found it when searching for the codec name because it only said something like "Information not available. Try again after playing a video." After opening a random YouTube video and refreshing about:support, it did show the hardware decoding information, and it was as I had expected given the hardware on this computer.
The thought process is that AMD, NVIDIA, Intel and the likes are not providing a patent license with their hardware.[3] They are instead just supplying part of an overall system that together with operating system kernel, display manager software, video player software, etc allows the decoding and encoding of patent encumbered video files. Open source software projects and distributions are concerned they'd be found to be infringing patents by enabling a complete solution out-of-the-box. Hence they put some hurdles in place so that a user has to go out of their way to separately piece together the various parts to form a complete system capable of encoding and decoding patent encumbered codecs.
edit: To clarify, if your Intel or AMD GPU/APU supports a patent-free codec such as AV1 (most GPUs/APUs available for sale?), Firefox on a standard Linux distribution will use hardware video decoding out of the box by default for the patent-free codec. The issue is really one of whether you're sourcing content from a provider that uses a good choice of codec like AV1. The good news is that patent trolls are doing a good job of pushing laggard content providers down this path.[4]
[1] https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/15...
[2] https://github.com/gentoo/gentoo/commit/1265a159743d7f07185a...
[3] https://lists.fedoraproject.org/archives/list/devel@lists.fe...
[4] https://news.ycombinator.com/item?id=38249527
Hardware video decode is working for FF.
Not working for Chrome though, sadly.
> a provider that uses a good choice of codec like AV1
Much more limited HW support and slow as molasses encoders. Not exactly a good choice for most.
One vendor specific API, not "various OS' native APIs".
Firefox currently supports hardware video decoding with Intel's vendor specific VA-API only on Linux, which is not supported by NVIDIA. (A third-party VA-API to NVDEC translation layer for Linux does exist on GitHub, nvidia-vaapi-driver, but it's not yet reliable as the officially supported VDPAU or NVDEC, and is not included in official linux package repositories.)
Intel has VA-API, AMD has AMF, and NVIDIA has VDPAU which is being replaced by NVDEC/NVENC.
The idea behind Vulkan Video Extensions is to have a vendor independent and cross-platform video API.
Intel is behind VA-API originally, but I don't think it's fair to say it's a vendor specific API anymore. It's supported by the open source drivers for GPUs from all 3 vendors. It's just that the open source drivers for Nvidia cards are not very practical and the proprietary drivers only support vdpau and nvdec/nvenc
Intel, AMD and NVIDIA have their own vendor-specific video APIs, and even when they provide official support for the API of another vendor, it tends to expose a limited subset of the full functionality (like the list of available codecs and encoding features).
You are free to call these vendor specific APIs for what they are or something else, but the reality has been that there is no single video API officially supported by Intel, AMD and NVIDIA. This changed with Vulkan Video.
But Vulkan Video isn't just about desktop: mobile devices, Raspberry Pi, etc. are expected to get on board with it eventually, just like they did with Vulkan.
> It's supported by the open source drivers for GPUs from all 3 vendors.
Which 3 vendors are you referring to? Intel, AMD, and who?
> It's just that the open source drivers for Nvidia cards are not very practical and the proprietary drivers only support vdpau and nvdec/nvenc
Why are you bringing up open source drivers, and what is not practical? Both official open source drivers (open-gpu-kernel-module) and unofficial open source drivers (nouveau, through binary firmware) support VDPAU. However, NVIDIA's drivers (open source or binary) does not support VA-API.
By the way, nouveau's support is currently limited and not useful: https://nouveau.freedesktop.org/VideoAcceleration.html see Video engine support status table, only old GPUs and no H.265 or AV1 support.
It was an answer to this question specifically
> Which 3 vendors are you referring to? Intel, AMD, and who?
I either missed some of the other text in your post or it was added after I started to reply.
> Both official open source drivers (open-gpu-kernel-module)
This is not remotely close to being a complete graphics driver. Most of a GPU driver on Linux is in userspace and there is no official open source user space component.
> Why are you bringing up open source drivers, and what is not practical?
nouveau has never been practical for serious use due to poor performance and mediocre hardware support (as you noted). open-gpu-kernel-module is only practical when paired with a proprietary userspace driver.
Anyway, my original point in all this is that describing VA-API as an Intel vendor specific API is unfair given it has been well supported on AMD GPUs for a long time now and on nouveau it's supported as well as VDPAU (i.e. not very well as you note). I did not intend to imply that it was universal. I didn't even intend to imply that VDPAU is a vendor specific API (though as a decode-only API it's not really a complete replacement).
Intel tried to make va-api the standard for hardware encode and decode on Linux, Nvidia tried to make VDPAU the standard for hardware decode on Linux. Neither was entirely successful. By contrast, NVENC/NVDEC, AMF and the Intel Media SDK (and whatever they replaced this with) never had such ambitions.
Incorrect. Firefox uses Windows Media Foundation, which is cross-vendor, on Windows. It uses MediaCodec on Android which is again cross-vendor. Presumably it uses whatever iOS' equivalent is as well.
It only uses VA-API on a single OS, Linux, and that's probably more a reflection on the media qualities (or lack thereof) of Linux as a whole. Maybe Vulkan video extensions will be the savior on Linux. Or maybe it won't because it won't be anyone's focus of investment since it's largely a Linux-only problem in the first place.
On Windows there's Windows Media Foundation and DirectShow that centrally manage everything and also support the "individual nut and bolt" approach. Android has its own central thing (MediaCodec?) that must be used. MacOS and iOS presumably have their own central manager (Quicktime?) too.
But Linux? It doesn't serve as an operating system for media. It's tremendously inconvenient as an admin/user rather than an evangelist.
Comparison is also invalid. Linux as a whole (not the kernel but OS experience) isn't controlled by some Big Brother who decides what and how it's done single mindedly. So such kind of composite result is somewhat expected.
Luckly Android, ChromeOS and WebOS as proper Linux distributions have replaced most of it.
ChromeOS hasn't displaced Linux on school laptops, there wasn't any.
LG WebOS hasn't displaced "Linux", it competes with Google TV (formerly Android TV)
So here goes a children explanation.
> Unfortunely I still have to, from time to time.
I, pjmlp, still have to use GNU/Linux desktop from time to time.
> Luckly Android, ChromeOS and WebOS as proper Linux distributions have replaced most of it.
Android, ChromeOS and WebOS, have replaced most of my needs, pjmlp, for GNU/Linux in the desktop and similar devices.
Great! I really hope you're happy with that setup! It's your personal computer and by all means, do what works well for you. In the end that's always a personal thing that's different for everyone. Who am I to judge how you use your computer?
But maybe ... stop complaining about Linux desktop then? If you don't like it? This must be like the 3rd time I've seen these types of single-line dismissive "Linux will never win the desktop"-comment from you in the last few days. Just one line, little or no context, or explanation, and IMHO also zero value, and an entire discussion derailed.
This is just becoming disruptive. You don't need to say anything you know. Personally, I rather dislike a number of things, but you don't see me complaining about it with one-liners every chance I get – and when I do say something, at least I make sure it's something of some substance, when I feel it actually contributes. And I sure as hell don't go around complaining people are "children" for disagreeing.
Ideally that kind of thing would be part of GNOME, or KDE, but then there are those that rather keep using twm like experience, making GNU/Linux really only good for headless experiences, at least the UNIX/POSIX part is always there.
Yeah, binary software will have to ship its own copy of ffmpeg... This isn't unique to media codecs though.
No perfect solutions here; both "system-wide codecs" and "every application brings their own codecs" have their own up- and downsides.
Besides, with ffmpeg and gstreamer the system-wide codecs paradigm also works on Linux.
This is one of those "it's different but it doesn't really matter much" type of things. Most people "just" install vlc or mpv or whatnot and things will "just work" for them, not really different from Windows. That it's technically slightly different is almost entirely transparent to the user.
> Firefox currently supports hardware video decoding with Intel's vendor specific VA-API only on Linux, which is not supported by NVIDIA.
(emphasis added)
You further wrote:
> Firefox uses Windows Media Foundation, which is cross-vendor, on Windows. It uses MediaCodec on Android which is again cross-vendor.
And? None of those APIs are cross-platform. Vulkan Video will eventually allow developers (including Firefox developers) to write a single code path for video to cover a wide range of platforms and vendors (likely with the exception of walled gardens like Apple-land, although someone might find a way to support like via a wrapper like MoltenVk for Vulkan).
What are you talking about? They didn't quote that sentence at all, and didn't cut in the middle of the sentence they quoted.
> And? None of those APIs are cross-platform.
Your original objection, the thing that got quoted, was about whether things are cross-vendor. That question is completely unrelated to whether things are cross-platform.
> The idea behind Vulkan Video Extensions is to have a vendor independent and cross-platform video API.
(emphasis added)
So if someone criticizes a portion of your statement which is already countered by your original full statement, you're not allowed to remind your full statement. What kind of logic is that?
My original post says the point of Vulkan Video is it will be cross-platform and cross-vendor. And gives one example of cross-vendor side of things on Linux.
Someone criticizes me by essentially saying "you are incorrect, that's only on Linux. Windows, Android and iOS have their own video APIs...". This "correction" is incorrect because I already said on Linux, and it goes on to actually reinforce the post that he is responding to by highlighting cross-platform side, which also is in the post he is responding to.
So, if you look at the full conversation, the criticism is self-contradictory. This is what I'm pointing out, but you are implying I'm not allowed to do that.
I disagree. When you fragment a statement in a way that changes its meaning and make a straw man out of it, people are justified in responding to it.
The other stuff in your comment did not "counter" what they said. You made statements about cross-vendor and cross-platform. They chose to only respond to one of those statements. That's not incorrect.
> This "correction" is incorrect because I already said on Linux
The first part of your comment specifically said "not "various OS' native APIs"". That goes beyond Linux. The later part of your comment was about Linux in particular, but your introduction was an overall statement that wasn't true.
> When you fragment a statement in a way that changes its meaning
They didn't. You misspoke and they didn't know what you actually meant.
And from your other post: > Obviously, I meant to say statement, not sentence, but I can't edit it anymore.
That was not obvious. They quoted an entire paragraph, and the subsequent paragraph does not change its meaning the way you're claiming it does.
Obviously, I meant to say statement, not sentence, but I can't edit it anymore.
It is also intended as a multi-platform abstraction.
This makes it a no-go as a platform API. The open drivers for AMD use VA-API.
they are also plentiful, vendor specific and unstable.
a generic solution would be well appreciated
Also, maybe someone knows more about this, but I thought that most software like Firefox, MPV and VLC are using dav1d for AV1 decoding as it seems to be the most performant AV1 decoder (in Software) out there at the moment.
See the post. They are working on AV1 extensions.
mpv for example can use Vulkan for both at the same time.
The primary choices are space savings and power efficiency. Dedicated hardware and serial decoding/encoding often win out as a result.
You can encode the independent groups of pictures (the key frame and all the subsequent predictive frames until the next key frame) in parallel but at least with 4k video you hit memory bandwidth limitations quite quickly.
With adaptive key frame placement you don't know the intervals up front, and they might have wildly different lengths. Some might be hundreds of frames, some might be just a few.
It's the non-linear-algebra stuff that is usually problematic.
However, most formats allow for some parallelism at the "macroblock" level - you can usually decode all 16x16 pixels simultaneously. To some extent you can decode macroblocks separately, but "intra prediction" requires you to have the ones above and to the left available.
It reduces the compression efficiency, but only a little, because the tiles will be quite large anyway, such as 6 tiles per frame (but can be more for 360 video applications). It can be limited by the number of decoding instances a piece of hardware can have concurrently.
Seems like for VR video, if it’s split up longitudinally you would only have to decode the tiles that are in the direction the user is looking at any given time, and just let the other bits pass by undecoded.
how to exceed the character limit on your super wide screen with just one function name