At a very (very!) abstract level AI is compression and compression is AI.
There's been rumblings about how Google's AI edge would lead to jumps in video and image coding but nothing big seems to have happened yet, though lots of micro-optimisations appear to be ML based.
Some suggested that the AV1 throw a bunch of tools at it spec design was built for AI to use.
Possibly VMAF is the biggest ML impact on codecs so far.
A full AI video codec redesign could yield some powerful benefits (especially on the generative side), but there are solid gains even on the existing codecs.
Shot or scene based detection can be used to place your I, P, and B frames in better locations for example.
For those that don't know, an I frame is a keyframe, a still image. P and B frame are predictions off of other frames (such as a keyframe). Frames within the same shot or scene are likely to be similar to one another, so having awareness of that contextual information can help you place for example a keyframe at precisely the start of that shot or scene, and predict everything in that shot or scene off of that.
I-frame placement isn't hard; it's very obvious when a scene changes or when an I-frame would be profitable. You generally need to have them far more often than you'd like anyway, as otherwise you can't seek in videos.
The main advantage of ML for video compression is that people accept giant ML models, so you can hide a lot of common "features" in there. The disadvantage is that classical codecs are macroblock-based for a reason; you can't just process a whole frame at a time, nobody has the memory bandwidth for that.
> At a very (very!) abstract level AI is compression and compression is AI.
There's actually a pretty nice read from one of the key figures in data compression field - Matt Mahoney: "Text Compression as a Test for Artificial Intelligence" [1]. Some interesting reads are also on his personal website, though a bit outdated now: https://mattmahoney.net/dc/rationale.html
AI = compression + goal. Though strictly speaking I think compression is only required in partially-observable and/or stochastic worlds where the information content of the world (far) exceeds the actor's model capacity.
The thing about video compression is that it's already very compute intensive. Most devices have dedicated hardware cores for encoding and decoding in order to save power. They work exclusively with integer values, chiefly unsigned 8 and 10 bit values with sequential and block memory transactions.
Humans not only design for compression quality, we also really focus on cost. And we design direct to what is cheap in hardware: integer adds, shifts, binary logic.
I do believe a different paradigm of ML is necessary for this particular application. The neural primitives used need to be more closely related to transistors than to matrix multiplications.
Matrix math units are a smart use of silicon for many ML applications, true.
But Von Neumann compute isn't going away. Neither are fixed function modules for shading, ray tracing and massively parallel GPGPU type applications. Also not video coding.
My point was that if training could emit an efficient video codec as a network of logic gates, rather than as a dense array of neural network weights, it might just produce something practical for video playback on mobile devices and video processing at hyperscale.
Since modern chips from cell phone application processors to datacenter GPUs all have significantly invested real estate to accelerate matrix operations, it only makes sense to take advantage of it, whether it is optimal for the task at hand.
That is exactly the same path that led us to GPGPU, which in turn derived to using GPUs to accelerate neural nets. In every step of the way it was about repurposing existing hardware that was designed for something else and was thus suboptimal.
Most videos are only going to be viewed a tiny number of times, often just on the devices they were taken on. It’s probably not worth spending too much effort on compressing these.
A small number of videos are going to be viewed millions of times. For these few videos it’s worth the compute to do the best possible compression.
There’s no reason the same video formats and decoding hardware can’t be used for both. The high compute version can use an ML model to figure out how to optimize the bandwidth budget across every scene while the cheap version uses some easy heuristics.
This is a common incorrect assumption. You have to balance ratio with speed, otherwise you end up with something that spends more time decompressing than actually downloading, as is literally the case today with XZ. this is why alternatives like Zstandard are getting popular. 9 times faster decompressing for 10% worse ratio.
I just realized there's inevitably going to be dedicated hardware to help speed things up with ai related tasks. The GPU helps significantly but I'd assume something even more suitable could be made.
> Yet, despite a decade-long effort, full neural-video compression—using deep learning—has not beat the best configurations of conventional codec standards in normal conditions. Reviews from third parties show that when benchmarked with conventional distortion metrics as well as human opinion scores, conventional video encoders still outperform neural-network compression, especially when conventional encoders are enhanced with AI tools.
I don't know how much better AI can do in practice.
If you sit down and really think through how an intraframe compression technique works - something like JPEG or MPEG - You may come to the conclusion that we are already playing with the fringes of our reality.
Consider The DCT and quantization. When a piece of information is transformed by The DCT, we talk about "energy compaction". That is to say, we took a block of information and determined how to represent it as a perfect (lossless) arrangement of amplitudes at given frequencies. This results in most of the important information (in human terms) condensing around the DC component in the top-left corner. We then use something explicitly engineered (i.e. 'trained') on the human visual system to quantize these figures into levels the computer can tolerate (incurring loss). Zig-zag encoding is practically an emergent mathematical mandate at this point...
Then, take a long look at how we are doing interframe compression. Forward & backward difference frames, frames that are differences of differences, predicted vectors of information, etc. A lot of this is very obvious, but there are tradeoffs to be made that perhaps a human should be in charge of - I.e. subjective measures of latency in video streaming relative to group-of-picture size.
Consider this - How could an AI system improve on techniques that optimize for the Human Visual System? A very large part of our multimedia compression today comes from strategically sacrificing information that is subjectively worthless to humans.
All that said, I am sure you can find some low-hanging fruit. For example, if you go and ask ChatGPT to make JPEG work better, it may helpfully suggest swapping Huffman for arithmetic coding. Not exactly a groundbreaking discovery in my book, but it would make a big difference.
Kind of tangential but: JPEG has always had arithmetic encoding in the spec. No one implemented it due to patents (even after the patents expired about a decade ago).
> How could an AI system improve on techniques that optimize for the Human Visual System
One way it may believed that AI may improve upon the traditional methods you mention is the assumption that AI can learn a tighter optimisation for the human system (than humans can handcraft)
>Consider this - How could an AI system improve on techniques that optimize for the Human Visual System? A very large part of our multimedia compression today comes from strategically sacrificing information that is subjectively worthless to humans.
Exactly that. AI could recognize what is important in a frame and use less compression there, while using pattern generation for high entropy parts which are not important.
We truly don't need AI to tell us what is important or not. We already know this, objectively. It's mathematically defined as the visible spectrum of the a available information, adjusted with relative high/low thresholds for compression level.
I think when they said important, they meant distinct or unique. For example, Vin Diesel driving muscle cars is the "important" part of the fast and furious movies, the city/plot/most of the supporting characters are details that could be simplified and reconstructed. An algorithm that generated FF movies could mostly hallucinate silly reasons for Vin Diesel to drive cars and shoot guns, with little real coherency or consistency and I bet the fanbase would love it.
In the context of video, that means not spending bits on things like wall textures, plants, buildings, crowds, details of misc objects, etc.
Mathematical information density is not the same as important. A picture of white noise is mathematically very dense in information, but only median an standard deviation are important to us.
>I don't know how much better AI can do in practice.
Coming soon, you show AI the script and the poster of the movie you want to compress, and then it just makes shit up on the fly when you hit play. There is no compression! The movie will be unique to each viewer as the algo will take a seed from each playback device and create the music, images, dialog all on the fly.
At some point it'll become like the "Be kind, rewind" movie where an entire movie can be compressed to a text description of each scene then and entirely regenerated on demand by AI.
I remember reading about fractal-based image compression in the 1990’s. IIRC, there was some product out there that used it (I vaguely remember product ads with a paint can in them) but I don’t think it ever took off.
I wonder if coupling an AI to help find self-similar portions of the image could revive this idea?
I got super into the concept of fractal compression when I was young, and then became pretty disillusioned at how primitive the techniques turned out to be. IIRC it wasn’t much more than “divide the image into, say 16x16 blocks, then search for 8x8 blocks which are similar to a 50% scaled version of a large block, and store the delta”. Young me was hoping for some cool math, but it was more like brute force motion compensation within a single image.
Somehow I got a copy of James Gleick's book Chaos and it had a big impact on me. I was kind of obsessed with chaos and fractals for a while. Right around that time a whole bunch of similar pop sci books came out and I read a bunch of them. Steven Levy's Artificial Life, a Mondo 2000 collection, John Koza's Genetic Programming, and Cliff Stoll's The Cuckoo's Egg all stand out in my memory.
I’m working on research for a journal article on this right now. Basically, get a bunch of videos you want compressed, use an auto encoder and just store the very small latent space variables instead of the video. Keep the decoder sitting around that was trained on those videos, and badaboom you can take a 512x512 movie and store it in 16x16x3 or another similarly sized latent space. However, this only works for fairly consistent video. Right now it works best for constantly monitoring video systems of physical phenomena. I.e. if you have multiple cameras pointed at the ocean to watch waves, you’re going to end up training a latent space that describes the wave physics. These small latent spaces are also the key to removing the black box. You can grab the latent spaces and connect the fields to physical quantities. For example, the wave reconstruction latent space has examples of velocity or wave height (variables needed to mathematically construct a wave field). Cool stuff!
The trouble is at some point it isn't really video compression, it's video synthesis from wave velocity data.
I mean, it's fine for stuff that doesn't really matter like video conferencing or TV/films, but I think you'd be mad to use this for CCTV or science or whatever.
It feels like if you’re willing to make computation >> storage, which is likely the bad optimization right now, you can use generative AI trained on the original video and key frames chosen to optimize size : fidelity and let the AI fill in the frames dynamically.
I'm going to draw a line with AI at what I call the "eyes on forehead" problem. With Ai Image rendering, there is the emergent trend of AI images being like a horror film, with faceless and extra-features on faces.
When AI is using a LLM dataset to fill in the gaps, it becomes a problem, because it's guessing what should be there when the "natural" noise would be better depicting nothing. Predictive vectors such as SVG use smooth images. i think the rasterization of a dataset turns single pixels into quasi-intelligible datapoints, but when treated indendently by a dumb AI, ends up rendering eye balls on a forehead. This is your problem, not mine. Please fix it.
That or you're in a lucid dream. I once read the way to test if you're in a lucid dream, is to look around for an alarm clock. If it tells the wrong time, it's probably a lucid dream. This conversation can't be a lucid dream unless it's the script of Waking Life. ;)
That Xerox bug is interesting- I once used a Parallel port cable to print to an Apple Laser Printer from the 90s, until Windows 7 no longer supported the driver. I don't remember how many megabytes were transferred for large print jobs, but it seemed to take a while. I thought- perhaps it would be better if it sent the data in low-byte, text only code, so that the printer-side can reconstruct text using hard-coded translation the as opposed rendering all the bitmap- and while some data was xferred as a JPEG, perhaps it was compressed, and thus altered.
I have yet to see a true AI ( or ML ) based Video Codec, or some potential Generative Video Codec like what Google did with Audio For Voice. And if it cant be done for Audio I doubt it will work on Video given the complexity. The current working prototype of H.267 ECM 8.0 aims to squeeze another 40% to 50% reduction in bitrate from H.266 / VVC Codec. There will likely be some ML usage on the Encoder side, but in terms of codec the exploration path hasn't provided much evidence how it will fundamentally change the codec efficiency.
47 comments
[ 3.7 ms ] story [ 85.1 ms ] threadThere's been rumblings about how Google's AI edge would lead to jumps in video and image coding but nothing big seems to have happened yet, though lots of micro-optimisations appear to be ML based.
Some suggested that the AV1 throw a bunch of tools at it spec design was built for AI to use.
Possibly VMAF is the biggest ML impact on codecs so far.
Shot or scene based detection can be used to place your I, P, and B frames in better locations for example.
For those that don't know, an I frame is a keyframe, a still image. P and B frame are predictions off of other frames (such as a keyframe). Frames within the same shot or scene are likely to be similar to one another, so having awareness of that contextual information can help you place for example a keyframe at precisely the start of that shot or scene, and predict everything in that shot or scene off of that.
The main advantage of ML for video compression is that people accept giant ML models, so you can hide a lot of common "features" in there. The disadvantage is that classical codecs are macroblock-based for a reason; you can't just process a whole frame at a time, nobody has the memory bandwidth for that.
I think it's not so abstract at all. The relationship feels fundamental.
Encoder, decoder, codec. Compression, entropy, lossiness, ...
There's been some cool stuff in the speech area: https://ai.googleblog.com/2021/02/lyra-new-very-low-bitrate-...
Also from Microsoft: https://techcommunity.microsoft.com/t5/microsoft-teams-blog/...
There's actually a pretty nice read from one of the key figures in data compression field - Matt Mahoney: "Text Compression as a Test for Artificial Intelligence" [1]. Some interesting reads are also on his personal website, though a bit outdated now: https://mattmahoney.net/dc/rationale.html
[1] https://static.aminer.org/pdf/PDF/000/014/009/text_compressi...
Humans not only design for compression quality, we also really focus on cost. And we design direct to what is cheap in hardware: integer adds, shifts, binary logic.
I do believe a different paradigm of ML is necessary for this particular application. The neural primitives used need to be more closely related to transistors than to matrix multiplications.
The gravitas will shift to enabling matrix math rather than algorithms being designed for special hardware.
Matrices are the new paradigm.
My point was that if training could emit an efficient video codec as a network of logic gates, rather than as a dense array of neural network weights, it might just produce something practical for video playback on mobile devices and video processing at hyperscale.
That is exactly the same path that led us to GPGPU, which in turn derived to using GPUs to accelerate neural nets. In every step of the way it was about repurposing existing hardware that was designed for something else and was thus suboptimal.
A small number of videos are going to be viewed millions of times. For these few videos it’s worth the compute to do the best possible compression.
There’s no reason the same video formats and decoding hardware can’t be used for both. The high compute version can use an ML model to figure out how to optimize the bandwidth budget across every scene while the cheap version uses some easy heuristics.
This is a common incorrect assumption. You have to balance ratio with speed, otherwise you end up with something that spends more time decompressing than actually downloading, as is literally the case today with XZ. this is why alternatives like Zstandard are getting popular. 9 times faster decompressing for 10% worse ratio.
https://machinelearning.apple.com/research/neural-engine-tra...
I don't know how much better AI can do in practice.
If you sit down and really think through how an intraframe compression technique works - something like JPEG or MPEG - You may come to the conclusion that we are already playing with the fringes of our reality.
Consider The DCT and quantization. When a piece of information is transformed by The DCT, we talk about "energy compaction". That is to say, we took a block of information and determined how to represent it as a perfect (lossless) arrangement of amplitudes at given frequencies. This results in most of the important information (in human terms) condensing around the DC component in the top-left corner. We then use something explicitly engineered (i.e. 'trained') on the human visual system to quantize these figures into levels the computer can tolerate (incurring loss). Zig-zag encoding is practically an emergent mathematical mandate at this point...
Then, take a long look at how we are doing interframe compression. Forward & backward difference frames, frames that are differences of differences, predicted vectors of information, etc. A lot of this is very obvious, but there are tradeoffs to be made that perhaps a human should be in charge of - I.e. subjective measures of latency in video streaming relative to group-of-picture size.
Consider this - How could an AI system improve on techniques that optimize for the Human Visual System? A very large part of our multimedia compression today comes from strategically sacrificing information that is subjectively worthless to humans.
All that said, I am sure you can find some low-hanging fruit. For example, if you go and ask ChatGPT to make JPEG work better, it may helpfully suggest swapping Huffman for arithmetic coding. Not exactly a groundbreaking discovery in my book, but it would make a big difference.
https://en.wikipedia.org/wiki/Discrete_cosine_transform
Roughly 10% improvement I believe.
One way it may believed that AI may improve upon the traditional methods you mention is the assumption that AI can learn a tighter optimisation for the human system (than humans can handcraft)
Exactly that. AI could recognize what is important in a frame and use less compression there, while using pattern generation for high entropy parts which are not important.
In the context of video, that means not spending bits on things like wall textures, plants, buildings, crowds, details of misc objects, etc.
Coming soon, you show AI the script and the poster of the movie you want to compress, and then it just makes shit up on the fly when you hit play. There is no compression! The movie will be unique to each viewer as the algo will take a seed from each playback device and create the music, images, dialog all on the fly.
A 2 hour movie in <1MB of storage
I wonder if coupling an AI to help find self-similar portions of the image could revive this idea?
not sure about actual implementations
I got super into the concept of fractal compression when I was young, and then became pretty disillusioned at how primitive the techniques turned out to be. IIRC it wasn’t much more than “divide the image into, say 16x16 blocks, then search for 8x8 blocks which are similar to a 50% scaled version of a large block, and store the delta”. Young me was hoping for some cool math, but it was more like brute force motion compensation within a single image.
I mean, it's fine for stuff that doesn't really matter like video conferencing or TV/films, but I think you'd be mad to use this for CCTV or science or whatever.
When AI is using a LLM dataset to fill in the gaps, it becomes a problem, because it's guessing what should be there when the "natural" noise would be better depicting nothing. Predictive vectors such as SVG use smooth images. i think the rasterization of a dataset turns single pixels into quasi-intelligible datapoints, but when treated indendently by a dumb AI, ends up rendering eye balls on a forehead. This is your problem, not mine. Please fix it.
That Xerox bug is interesting- I once used a Parallel port cable to print to an Apple Laser Printer from the 90s, until Windows 7 no longer supported the driver. I don't remember how many megabytes were transferred for large print jobs, but it seemed to take a while. I thought- perhaps it would be better if it sent the data in low-byte, text only code, so that the printer-side can reconstruct text using hard-coded translation the as opposed rendering all the bitmap- and while some data was xferred as a JPEG, perhaps it was compressed, and thus altered.