What changed? I see the infrastructure has been upgraded, this seems like a big release, etc. I guess there was a recent influx of contributors? A corporate donation? Something else?
Impressed anytime I have to use it (even if I have to study its man page again or use an LLM to construct the right incantation or use a GUI that just builds the incantation based on visual options). Becoming an indispensable transcoding multitool.
I think building some processing off of Vulkan 1.3 was the right move. (Aside, I also just noticed yesterday that Asahi Linux on Mac supports that standard as well.)
This seemed to be interesting to users of this site. tl;dr they added support for whisper, an OpenAI model for speech-to-text, which should allow autogeneration of captions via ffmpeg
Why would they be tied to this release number when they can build themselves at their own schedule?
> Note that these releases are intended for distributors and system integrators. Users that wish to compile from source themselves are strongly encouraged to consider using the development branch
LLMs have really made ffmpeg implementations easy-- the command line options are so expansive and obscure it's so nice to just tell it what you want and have it spit out a crazy ffmpeg command.
Has anyone made a good GUI frontend for accessing the various features of FFMPEG? Sometimes you just want to remux a video without doing any transcoding, or join several video and audio streams together (same codecs).
Joining videos together sounds easy, but there's tons of ways it can go wrong! You've got time bases to consider, start offsets, frame/overscan crops, fps differences (constant vs variable), etc. And even though your videos might both be h264, one might be encoded with B frames and open GOP, and the other not, and that might cause playback issues in certain circumstances. Similarly, both could be AAC audio, but one is 48kHz sample rate, the other 44.1kHz.
Someone else mentioned Lossless-Cut program, which is pretty good. It has a merge feature that has a compatibility checker ability that can detect a few issues. But I find transcoding the separate videos to MPEG-TS before joining them can get around many problems. If you fire up a RAM-Disk, it's a fast task.
Nice! Anyone have any idea how and when this will affect downstream projects like yt-dlp, jellyfin, etc? Especially with regard to support for HW-acceleration?
Happy to hear that they've introduced video encoders and decoders based on compute shaders. The only video codecs widely supported in hardware are H.264, H.265 and AV1, so cross-platform acceleration for other codecs will be very nice to have, even if it's less efficient than fixed-function hardware. The new ProRes encoder already looks useful for a project I'm working on.
> Only codecs specifically designed for parallelised decoding can be implemented in such a way, with more mainstream codecs not being planned for support.
It makes sense that most video codecs aren't amenable to compute shader decoding. You need tens of thousands of threads to keep a GPU busy, and you'll struggle to get that much parallelism when you have data dependencies between frames and between tiles in the same frame.
Kostya did a lot of the RV60/RMHD reverse engineering work for NihAV back in 2018! His blog also talks about the GPL violations from Real.
The old RV40 had some small advantages over H264. At low bitrates, RV40 always seemed to blur instead of block, so it got used a lot for anime content. CPU-only decoding was also more lightweight than even the most optimized H264 decoder (CoreAVC with the inloop deblocking disabled to save even more CPU).
ffmpeg is a treasure to the open source and audio technology communities. The tool cuts right through all kinds of proprietary and arcane roadblocks presented by various codecs and formats and it's clear a tremendous amount of work goes into keeping it all working. The CLI is of course quite opaque and the documentation for various features is often terse, but it's still the only tool on any platform anywhere that will always get you what you need for video and audio processing without ever running up against some kind of commercial paywall.
If there's anything that needs audio/video automation, I've always turned to FFmpeg, it's such a crucial and indispensible tool and so many online video tools use it and are generally a UI wrapper around this wonderful tool. TIL - there's FFmpeg.Wasm also [0].
In Jan 2024, I had used it to extract frames of 1993 anime movie in 15 minutes video segments, upscaled it using Real-ESRGAN-ncnn-vulkan [1] then recombining the output frames for final 4K upscaled anime [2]. FWIW, if I had built a UI on this workflow it could've become a tool similar to Topaz AI which is quite popular these days.
I tried the exact same steps you did with the exact same movie but with Topaz AI and got very bad results which made me abondon the project. I'd be greatful if you could share the upscaled movie.
The Vulkan compute shader implementations are cool...particularly for FFv1 and ProRes RAW. Given that these bypass fixed-function hardware decoders entirely, I'm curious about the memory bandwidth implications. FFv1's context-adaptive arithmetic coding seems inherently sequential, yet they're achieving "very significant speedups."
Are they using wavefront/subgroup operations to parallelize the range decoder across multiple symbols simultaneously? Or exploiting the slice-level parallelism with each workgroup handling independent slices? The arithmetic coding dependency chain has traditionally been the bottleneck for GPU acceleration of these codecs.
I'd love to hear from anyone who's profiled the compute shader implementation - particularly interested in the occupancy vs. bandwidth tradeoff they've chosen for the entropy decoding stage.
43 comments
[ 5.1 ms ] story [ 69.8 ms ] threadSecondly, just curious: any insiders here?
What changed? I see the infrastructure has been upgraded, this seems like a big release, etc. I guess there was a recent influx of contributors? A corporate donation? Something else?
I think building some processing off of Vulkan 1.3 was the right move. (Aside, I also just noticed yesterday that Asahi Linux on Mac supports that standard as well.)
This seemed to be interesting to users of this site. tl;dr they added support for whisper, an OpenAI model for speech-to-text, which should allow autogeneration of captions via ffmpeg
[0] - https://xkcd.com/2347/
> Note that these releases are intended for distributors and system integrators. Users that wish to compile from source themselves are strongly encouraged to consider using the development branch
https://youtu.be/9kaIXkImCAM?si=b_vzB4o87ArcYNfq
Someone else mentioned Lossless-Cut program, which is pretty good. It has a merge feature that has a compatibility checker ability that can detect a few issues. But I find transcoding the separate videos to MPEG-TS before joining them can get around many problems. If you fire up a RAM-Disk, it's a fast task.
> Only codecs specifically designed for parallelised decoding can be implemented in such a way, with more mainstream codecs not being planned for support.
It makes sense that most video codecs aren't amenable to compute shader decoding. You need tens of thousands of threads to keep a GPU busy, and you'll struggle to get that much parallelism when you have data dependencies between frames and between tiles in the same frame.
I wonder whether encoders might have more flexibility than decoders. Using compute shaders to encode something like VP9 (https://blogs.gnome.org/rbultje/2016/12/13/overview-of-the-v...) would be an interesting challenge.
The old RV40 had some small advantages over H264. At low bitrates, RV40 always seemed to blur instead of block, so it got used a lot for anime content. CPU-only decoding was also more lightweight than even the most optimized H264 decoder (CoreAVC with the inloop deblocking disabled to save even more CPU).
If there's anything that needs audio/video automation, I've always turned to FFmpeg, it's such a crucial and indispensible tool and so many online video tools use it and are generally a UI wrapper around this wonderful tool. TIL - there's FFmpeg.Wasm also [0].
In Jan 2024, I had used it to extract frames of 1993 anime movie in 15 minutes video segments, upscaled it using Real-ESRGAN-ncnn-vulkan [1] then recombining the output frames for final 4K upscaled anime [2]. FWIW, if I had built a UI on this workflow it could've become a tool similar to Topaz AI which is quite popular these days.
[0]: https://github.com/ffmpegwasm/ffmpeg.wasm
[1]: https://github.com/xinntao/Real-ESRGAN-ncnn-vulkan
[2]: https://files.horizon.pics/3f6a47d0-429f-4024-a5e0-e85ceb0f6...
Are they using wavefront/subgroup operations to parallelize the range decoder across multiple symbols simultaneously? Or exploiting the slice-level parallelism with each workgroup handling independent slices? The arithmetic coding dependency chain has traditionally been the bottleneck for GPU acceleration of these codecs.
I'd love to hear from anyone who's profiled the compute shader implementation - particularly interested in the occupancy vs. bandwidth tradeoff they've chosen for the entropy decoding stage.