I don’t think we’re far off a “final” video codec, in the sense that jpg and mp3 and zip are effectively “final” image and sound and data codecs. Sure, there are newer codecs with higher compression foot a given quality level, but at some point it’s good enough to not bother upgrading, especially when the new version costs exponentially more for negligible gain.
…maybe THIS one will be The Final Codec… doodoodoo dooo
Re: mp3, Opus is a pretty big improvement. In my experience I've found that Opus is pretty reliably transparent at 96kbit/s, whereas LAME (which really squeezes mp3 for all it's got) needs ~130kbit/s
I think encode costs are higher (although I'm struggling to find any actual data) but in my experience this is totally negligible in practice.
Of course, don't go converting all your mp3s to Opus, but if you're running a music streaming server or whatever, Opus is pretty sweet... and probably close to "as good as it gets" until/unless we figure out some crazy new tricks.
> jpg and mp3 and zip are effectively “final” image and sound and data codecs
In popular consciousness, maybe. When people think image they thing jpg. But most websites you use aren't delivering images as jpg. Most sites care about saving money on bandwidth and their site loading fast, and because of that they tend to use webp and avif, which are much smaller for the same quality.
Yes, "neural" or "learned" video codecs are considered the future. They already have competitive compression ratios, but last thing I heard was that they are still much slower than conventional codecs.
Intel has supported native hardware decoder for AV1 since Tiger Lake. That’s quite old. You could buy a used NUC with Tiger Lake. Even when new a NUC with 1115G4 or 1135G7 isn’t expensive. I found one with damaged WiFi for $20. https://www.ebay.com/itm/366767313411 You can buy it now.
I went to the link to buy a computer (a NUC11TNKi3) with this capability for $20, but all it would let me do is place a bid on it; I cannot buy it now.
Lots of media devices that don't have AV1 decoders tend to have issues decoding high resolution AV1 in real time so you either get pauses to buffer or dropped frames.
Cheaper to get a long fiber optic HDMI cable and connect your TV to your PC GPU. That's what I do. You get the best uncompressed video (typ 4K 120Hz or 8K 60Hz), zero lag so it can be used for stuff other than video, and don't have to maintain another box
"The H.264 video, 300 frames long is 175KB. A single frame of that video in PNG is 1015KB. It looks like we're storing 300 times the amount of data in the video. But the file size is a fifth. So H.264 would seem to be 1500x as efficient as PNG. How is this even possible? All right, what's the trick?"
Mostly the trick is they're comparing lossy to lossless compression which doesn't even make any sense.
Actually I had the same thought, but this is obviously an article for people who don't know the difference between lossless and lossy compression. Ok, you might argue that it's not a good fit for HN, but going by the many times it seems to have been posted here, some people disagree.
This reminds me that I still haven't watched `part 2` of a very nice video from 3Blue1Brown - "Compression is Intelligence". YouTube suggestions, definitely failed me with this one.
The main reason to use H.264 in 2026 is legacy device support. Some devices cannot play H.265, but H.264 is playable almost anywhere, as a 20+ year old codec.
I was surprised that Twitter only lets you upload H.264 still, in 2026.
Twitter's video story is not their strongest. I used to run an account that posted a lot of video content, and their compression quality was so mixed that I ended up writing a little mobile app that would transcode videos to Twitter's advertised optimal format (720p 30fps H.264 5Mbps + AAC LC) before uploading them.
Even then, it was a crapshoot whether or not the video would look like a pixel collage, and I'd need to delete the Tweet, tweak the video a bit, and try again.
For personal use, if those are your only two choices, then go with H.265 for higher efficiency, or H.264 for wider compatibility. If other codecs are an option, go with AV1 for higher efficiency. It’s currently the best and already pretty widely supported.
Patents aside, h265 from my reading is almost universally better than h264. My understanding which could be wrong is that for a given quality h265 will always be smaller, and particularly for >1080p it's recommended. Hardware support has been around since at least the Shield days (2019) so that shouldn't be an issue. Only use h264 if you are needing to hw-decode on really old hardware.
AV1 is the future/present: Fully open standard, better than h265. Only downside is hardware support: You need hardware made in the last few years.
With JPEG XL finally being allowed into mainstream web browsers, we’ll hopefully see more native platform support and be able to slowly phase every older codec out, since JPEG XL does very well at more or less everything (except for animation, compared to video codecs).
I hope so, but I disagree with the size of the "except": JXL performs better than APNG and GIF, and still supports transparency and lossless encode. Video codecs struggle here. So even with animation it still depends!
There's also a case for AVIF in some use cases (for this image it performs poorly). But if I had to choose one, I'd choose JXL.
I really like the idea that compression is intelligence, there are so many tricks an interesting ideas related to it. A bunch of them are described in the article, but you can imagine a long tail of many more inscrutable tricks chained together to get really incredible shrinkage in data.
Hutter prize (I think mentioned in a 3b1b video somewhere else in this thread) is a great example. Given this was written before advent of LLM, I would love to see someone throw openAI/Anthropic levels of compute at a similar problem to see what the swarm can produce.
The newest submissions to Hutter Prize made crazy jumps by just replacing the LSTM with a transformer and training it.
35'000$ for a prompt and some training is a nice chunk of cash.
Next steps are replacing the encoders and all the other human-biased compression parts by just raw data and training, probably another 20-30% is up for grabs.
I remember reading an article on here by one of the founders of MPEG about why he was leaving MPEG after several decades, and part of it was that modern AI is indistinguishable from compression and he wanted to explore new compression algorithms using the recent explosion of knowledge
H.264 High Profile is close to Patent Free. [1] I often wonder how much more efficiency could we get if we were to increase block size outside of spec.
One uncommonly used feature of H.264 is support for lossless video. It's a bit hard to use due to tooling (not easy to discover and select the feature), and is sometimes incompatible with web browsers, but you can get actual lossless video using a mainstream video codec. You can verify that it really is lossless, and doesn't have any conversion step where it's quantized to 8-bit YUV.
"x264vfw" is one tool which supports this format, even inside of an AVI file. So older software that uses Video For Windows can create those files.
I've attempted to get researchers to use lossless H.264 to encode CT scans. They typically use directories full of tiffs. Even with intra-slice compression (i.e. tiff compression) they are pretty huge. But H.264 will let you recover the original images to the individual bit, and save a huge amount of space thanks to inter-slice similarities.
I’d imagine the main drawback of using H.264 here is that you’d still have to decode to get the frames back. This is similar to how even though many consumer grade cameras will output natively to h.264, if you want to edit the videos from it you’d usually have to convert it to prores or something first (the editor would usually do this automatically behind the scenes)
I’m not sure if lossless h.264 is I only or not, but if it uses inter-frame compression then it’s difficult to randomly access arbitrary frames. For any arbitrary frame you’d have to find the nearest key frame, then interpolate from there based on the inter-frame compression metadata.
There are some interesting advances in neural video encoders and decoders - I suspect it will look a lot like the neural models in DLSS 5 for compression -> decompression/reconstitution. There have been some lossless text-compression neural models built that seem to surpass even high level compression algorithms so with time those could get there.
Had the pleasure of working with Jason Garrett-Glaser (better known as DarkShikari) circa 2011 on a patch for x264 to better utilize more cores (slice-threaded lookahead). At the time he was working with Sony to optimize their PlayStation game streaming service, and this had the potential to lower encoding latency (critical to game streaming) even on same-core systems by spreading the frame over more cores to do more parallel work, and doing more intelligent scheduling and convergence after that tile-based work was reconverged into a frame. At the time we were using servers with 48 physical cores to do transcoding, a bit of a rarity in a single NUMA space back then and there were no instances to test this on so I flamegraph traced a few encodes at very high thread parallel counts and send back the instrumented profiles (from a custom compiled version) showing the core work being done and he used that to optimize the motion estimation code which worked on the first shot to perfectly level out cpu usage and increase encoding speed across ~24 cores (was previously topping out around 8 and running one core hot iirc). I got a look into how exotic and complex this project was, particularly in terms how low level and advanced things like motion estimation, quantizing and perceptual compression were... it was and still is sort of magic how many smart techniques are baked into something that we barely pay attention to. They were, even at that time, working on x265.
Very cool. Took a similar approach to optimizing openutv by looking at flamegraphs. UTV is a fork of openrv which is an older release of the commercial RV. Like many things it uses ffmpeg and also native readers and writers for certain formats. Allowed me to get it to 8k@60fps buttery smooth playback with hardware acceleration. If you are still in the video/imaging space would love your feedback on UTV. openutv.com
Using multiple CPUs for encoding goes way back. The early (1994) C-Cube MPEG-1 and MPEG-2 real-time encoders for DirecTV were built with 8 chips running in parallel (out of necessity since they were only 60 MHz processors) to encode 704x480. Each chip encoded a slice of the image and an auxiliary bus was used to send reconstruction video between chips so that motion vectors could cross chip image boundaries.
Here's a picture of the VME bus development board. It was a two board sandwich to support 10 chips in parallel (only one board pictured).
The first real-time HDTV encoders for terrestrial digital television (ATSC) were built by Harris using an IBM encoder chip. It took 6 chips to encode 1920x1080. The IBM chips didn't have an auxiliary bus, so you could see motion vectors limited to chip image boundaries (in this case, 6 tiles arranged like so).
Yeah at this time intel was just releasing their quick sync tech and a few vendors were putting 4 intel cpus on a pcie board and using it for video encoding! Matrox also had a considerable piece of this business doing video compression and editing/compositing using bespoke processors and boards.
Despite being more "power efficient" - both nvidia and intel's hardware solutions left a lot to be desired in terms of final quality for the same bitrate, compared to x264 that seemed to just put out substantially better quality video for the same bitrate even at the same h264 profile levels. There are still lots of x264 design elements, some even verbatim ported, into modern av1 and hevc encoders.
I'm surprised that 10 years later, the article still teaches an incorrect conceptual understanding of "quantization".
The context heavily implies that quantization is "discarding information" generally (e.g. clipping, masking). The wikipedia article he links in the footnote disagrees, and so do I:
> quantization is the process of mapping input values from a large set (often a continuous set) to output values in a (countable) smaller set, often with a finite number of elements.
Quantization is a function over values, or the image itself. Clipping and masking, by contrast, are functions over the space/frequency dimensions, i.e. the pixel indices ("preimage"). A rule like "apply a circular mask" and "drop the first quadrant" do not depend on the values!
The most extreme example of quantization is a threshold filter. Regardless of location, output "1" if the pixel is brighter than X, or "0" otherwise.
Codecs like H.264 and JPEG use spatially-varying quantization rules to effectively discard more information at high frequencies, so his explanation of H.264 is spot on, but a non-expert encountering quantization for the first time would probably come away with the wrong idea. For an article explicitly targeted to non-experts that goes out of its way to introduce the term, it's a pretty important oversight.
Consider me confused. Isn’t quantizing a value, e.g. reducing the number of bits in an int, a form of discarding information from that value? Isn’t your comparison just about discarding information on another level (discarding entire values)?
I think the GP's point is, if you don't know what quantization is, the article gives the impression it consists of discarding some of the values, which is where the compression comes from. This is a good "0-th order" intuition, but quantization is a softening of this; instead of fully retaining or fully discarding a value, there is a sliding scale of how much information to discard. If you already know what quantization is, it's easy to see how discarding is a special case, but the article gives the impression discarding = quantization.
Exactly correct. Quantizing is like rounding. This is the same notion as in quantum mechanics: the values aren’t any real number (a huge input space), but rather integer multiples of some constant (a small output space). Quantization is “rounding” to the nearest allowed output.
To build on the example used in the article, you could quantize an image by dropping the lower half of the bits of every pixel value. This discards half the information by mapping the larger input image space (e.g. 8 bits per pixel) to a smaller output image space (e.g. 4 bits per pixel), effectively scaling/rounding the original values.
The fact that higher frequencies can be quantized more intensely with relatively little image quality loss is a completely separate idea from quantization itself. In this context, high frequency masking as described in the article could be accurately described as a spatially dependent quantization to zero-bit output values. But the quantization part is the mapping, not the shape of the mask, and I think the article is misleading about the distinction.
In real codecs, there usually isn’t a hard boundary like a mask, but instead an empirical table of how many bits to keep based on the perceptual value of the information at each frequency.
I’m not saying that quantization doesn’t lose information. Instead, I’m saying that every step to lose information is not quantization. Functions on the preimage (clipping, masking) are not quantization.
I remember reading this article when it was new, and I left fully agreeing, but more broadly: modern video compression is magic.
Like, it's baffling to me how much we can compression high-quality video nowadays. HEVC and AV1 and VP9 are all also very cool and they still sort of break my brain. I understand how they work but it doesn't feel like it should compress as well as it does.
I've done AV1 encodes where I was convinced the conversion must have failed in some way for it to be that small, scrubbed through it and it all looks fine, watched through the entire thing to confirm that wasn't the case, just waiting for the shoe to drop, but then to find out, no, it's really just that good.
Well, that was disappointing. Everything in that article is generic to any video compression codec, even MPEG. I thought this would discuss why H.264 compresses so much more than predecessors, but not a word about that. Should have been titled "video compression is magic".
I've shared this article many times. I work in space and the vast majority of the time, new people insist that all images should be raw. After all, you go to great expense to get somewhere in the name of science, you don't want to discard any information. And that's true in some cases but typically not for a whole mission or even a whole image if you can crop a specific part. Usually 100 or 1000 images instead of one is more valuable even if the compression is lossy.
That's just image compression though. Very few space-grade cameras support video compression. In a lot of cases it works well even though you think you're after a series of still images.
It’s mostly mission dependent as you mentioned, if the value of some images are different than others, compressing several pictures and selecting the best to get in raw is a good strategy, and you can also crop them to get interesting areas as mentioned.
It can also depends on the downlink bandwidth available and power/energy budget, so the optimal strategy will differ.
It can be misleading at times though, for example some interesting feature may happen to be just a compression artifact. A cloud looking like some strange human-made object for example, happened to us one time.
I wish the video presented had been one with, say, 5x higher bitrate, as it would have sufficed to make the point without presenting a video with very glaring compression artifacts.
79 comments
[ 0.18 ms ] story [ 26.1 ms ] threadhttps://engineering.fb.com/2026/06/22/video-engineering/adop...
And soon AV2
https://av2.aomedia.org/
Otherwise we will have this:
https://xkcd.com/927/
…maybe THIS one will be The Final Codec… doodoodoo dooo
I think encode costs are higher (although I'm struggling to find any actual data) but in my experience this is totally negligible in practice.
Of course, don't go converting all your mp3s to Opus, but if you're running a music streaming server or whatever, Opus is pretty sweet... and probably close to "as good as it gets" until/unless we figure out some crazy new tricks.
In popular consciousness, maybe. When people think image they thing jpg. But most websites you use aren't delivering images as jpg. Most sites care about saving money on bandwidth and their site loading fast, and because of that they tend to use webp and avif, which are much smaller for the same quality.
And then we'll get AI-designed hardware codecs for them.
It looks like they normally sell for around $200: https://www.ebay.com/sch/i.html?_nkw=NUC11TNKi3&LH_Sold=1
https://news.ycombinator.com/item?id=16796127
https://news.ycombinator.com/item?id=12871403
https://news.ycombinator.com/item?id=30710574
https://news.ycombinator.com/item?id=19997813
https://news.ycombinator.com/item?id=40681306
Mostly the trick is they're comparing lossy to lossless compression which doesn't even make any sense.
Should I use H.264 or H.265?
I was surprised that Twitter only lets you upload H.264 still, in 2026.
Even then, it was a crapshoot whether or not the video would look like a pixel collage, and I'd need to delete the Tweet, tweak the video a bit, and try again.
How are you generating video?
How are you playing it back?
AV1 is the future/present: Fully open standard, better than h265. Only downside is hardware support: You need hardware made in the last few years.
PNG: 1039993 bytes - baseline
optimized*: 541539
WebP: 398974 - lossless mode, of course.
JPEG XL: 310778 - lossless, quick, and easy.
*after monumental compute effort; still probably some room for improvement but the effort is ridiculous.
There's also a case for AVIF in some use cases (for this image it performs poorly). But if I had to choose one, I'd choose JXL.
Hutter prize (I think mentioned in a 3b1b video somewhere else in this thread) is a great example. Given this was written before advent of LLM, I would love to see someone throw openAI/Anthropic levels of compute at a similar problem to see what the swarm can produce.
https://arxiv.org/html/2404.09937v1
35'000$ for a prompt and some training is a nice chunk of cash.
Next steps are replacing the encoders and all the other human-biased compression parts by just raw data and training, probably another 20-30% is up for grabs.
[1] https://meta.wikimedia.org/wiki/Have_the_patents_for_H.264_M...
"x264vfw" is one tool which supports this format, even inside of an AVI file. So older software that uses Video For Windows can create those files.
Is it similar to lossless audio were there's not a ton of optimization you can do?
They are still using directories of tiff, though.
I’m not sure if lossless h.264 is I only or not, but if it uses inter-frame compression then it’s difficult to randomly access arbitrary frames. For any arbitrary frame you’d have to find the nearest key frame, then interpolate from there based on the inter-frame compression metadata.
the days where the OS matters for video playback are long gone
https://news.ycombinator.com/item?id=38099633
Here's a picture of the VME bus development board. It was a two board sandwich to support 10 chips in parallel (only one board pictured).
https://www.w6rz.net/scorpio.jpg
The first real-time HDTV encoders for terrestrial digital television (ATSC) were built by Harris using an IBM encoder chip. It took 6 chips to encode 1920x1080. The IBM chips didn't have an auxiliary bus, so you could see motion vectors limited to chip image boundaries (in this case, 6 tiles arranged like so).
Despite being more "power efficient" - both nvidia and intel's hardware solutions left a lot to be desired in terms of final quality for the same bitrate, compared to x264 that seemed to just put out substantially better quality video for the same bitrate even at the same h264 profile levels. There are still lots of x264 design elements, some even verbatim ported, into modern av1 and hevc encoders.
The context heavily implies that quantization is "discarding information" generally (e.g. clipping, masking). The wikipedia article he links in the footnote disagrees, and so do I:
> quantization is the process of mapping input values from a large set (often a continuous set) to output values in a (countable) smaller set, often with a finite number of elements.
Quantization is a function over values, or the image itself. Clipping and masking, by contrast, are functions over the space/frequency dimensions, i.e. the pixel indices ("preimage"). A rule like "apply a circular mask" and "drop the first quadrant" do not depend on the values!
The most extreme example of quantization is a threshold filter. Regardless of location, output "1" if the pixel is brighter than X, or "0" otherwise.
Codecs like H.264 and JPEG use spatially-varying quantization rules to effectively discard more information at high frequencies, so his explanation of H.264 is spot on, but a non-expert encountering quantization for the first time would probably come away with the wrong idea. For an article explicitly targeted to non-experts that goes out of its way to introduce the term, it's a pretty important oversight.
To build on the example used in the article, you could quantize an image by dropping the lower half of the bits of every pixel value. This discards half the information by mapping the larger input image space (e.g. 8 bits per pixel) to a smaller output image space (e.g. 4 bits per pixel), effectively scaling/rounding the original values.
The fact that higher frequencies can be quantized more intensely with relatively little image quality loss is a completely separate idea from quantization itself. In this context, high frequency masking as described in the article could be accurately described as a spatially dependent quantization to zero-bit output values. But the quantization part is the mapping, not the shape of the mask, and I think the article is misleading about the distinction.
In real codecs, there usually isn’t a hard boundary like a mask, but instead an empirical table of how many bits to keep based on the perceptual value of the information at each frequency.
Like, it's baffling to me how much we can compression high-quality video nowadays. HEVC and AV1 and VP9 are all also very cool and they still sort of break my brain. I understand how they work but it doesn't feel like it should compress as well as it does.
That's just image compression though. Very few space-grade cameras support video compression. In a lot of cases it works well even though you think you're after a series of still images.
It can also depends on the downlink bandwidth available and power/energy budget, so the optimal strategy will differ.
It can be misleading at times though, for example some interesting feature may happen to be just a compression artifact. A cloud looking like some strange human-made object for example, happened to us one time.