229 comments

[ 3.5 ms ] story [ 189 ms ] thread
(comment deleted)
Thank you! I hope you now share my feeling of awe over this technical wonder.
Excellent! :)

It would be really cool to further extend it showing actually how the various tiles are encoded and between frames, something along the lines of: http://jvns.ca/blog/2013/10/24/day-16-gzip-plus-poetry-equal...

H.265 can even do deltas between blocks in the same frame, IIRC, and is excellent for still image compression too.

H.265 is my next post :)
So if H.264 is magic, what is H.265 then? :)
Excuse me, i would like my account back.
> discard information which will contain the information with high frequency components. Now if you convert back to your regular x-y coordinates, you'll find that the resulting image looks similar to the original but has lost some of the fine details.

I would expect also the edges in the image to become more blurred, as edges correspond to high-frequency content. However, this only seems to be slightly the case in the example images.

You're right! The images that I am using are zoomed cropped sections of a much larger image of the entire Apple home page.
This is probably because in the sample image you have clean vertical edges. It's pretty easy to represent these edges with a waveform.
You can see exactly that with the speaker grill and the text (This type of transformation is notoriously bad at compressing images of text, and is why you shouldn't use jpg for pictures of text)

In this context, the edges of, say, the macbook are not "high frequency" content, since they only feature one change (low to high luminosity) in a given block rather than several (high-low-high-low-high) like for the grill.

You should have a look at the Fourier transform of a step-function. It has high frequency components.
I thought I'll learn something special about H.264, but all information here is high level and generic.

For example if you replace H.264 with a much older technology like mpeg-1 (from 1993) every sentence stays correct, except this:

"It is the result of 30+ years of work" :)

Sure, but also keep in mind that the technology hasn't changed much over time. Even HEVC, which causes extreme gains in compression on high-res video with minimal loss in quality, is still mostly the same algorithm as H.264 but with larger blocks, slightly more flexible coding units rather than frame-wide interpolation changes, and 35 rather than 9 directions recognized for predictions.
"This post will give insight into some of the details at a high level - I hope to not bore you too much with the intricacies."

Did you miss the third paragraph?

As someone who knew nothing about it before, I found it lived up to it's goal.

I think they just mean it shouldn't be "H.264 is magic", it should just be "video compression is magic" or some such. That irked me a little bit too.
I was a bit disappointed in this article for the same reason: this is a great primer for people new to MPEG video compression, but it doesn't have anything to do with H.264.

I was hoping the author would write about H.264 specifically, for instance, how it was basically the "dumping ground" of all the little tweaks and improvements that were pulled out of MPEG-4 for one reason or another (usually because they were too computationally expensive), and why, as a result, it has thousands of different combinations of features that are extremely complicated to support, which is why it had to be grouped into "profiles" (e.g., Baseline, Main, High): http://blog.mediacoderhq.com/h264-profiles-and-levels/

I was also hoping that he would at least touch on the features that make H.264 unique from previous MPEG standards, like in-loop deblocking, CABAC Entropy Coding, etc..

Again, it's fine as an introduction to video encoding, but there's nothing in here specific to H.264.

The lossy transform is important, but I think what's actually most important in video compression is getting rid of redundancy --- H.264 actually has a lossless mode in which that transform is not used, and it still compresses rather well (especially for noiseless scenes like a screencast.) You can see the difference if you compare with something like MJPEG which is essentially every frame independently encoded as a JPEG.

The key idea is to encode differences; even in an I-frame, macroblocks can be encoded as differences from previous macroblocks, and with various filterings applied: https://www.vcodex.com/h264avc-intra-precition/ This reduces the spatial redundancies within a frame, and motion compensation reduces the temporaral redundancies between frames.

You can sometimes see this when seeking through video that doesn't contain many I-frames, as all the decoder can do is try to decode and apply differences to the last full frame; if that isn't the actual preceding frame, you will see the blocks move around and change in odd ways to create sometimes rather amusing effects, until it reaches the next I-frame. The first example I found on the Internet shows this clearly, likely resulting from jumping immediately into the middle of a file: http://i.imgur.com/G4tbmTo.png That frame contains only the differences from the previous one.

As someone who has written a JPEG decoder just for fun and learning purposes, I'm probably going to try a video decoder next; although I think starting from something simpler like H.261 and working upwards from there would be much easier than starting immediately with H.264. The principles are not all that different, but the number of modes/configurations the newer standards have --- essentially for the purpose of eliminating more redundancies from the output --- can be overwhelming. H.261 only supports two frame sizes, no B-frames, and no intra-prediction. It's certainly a fascinating area to explore if you're interested in video and compression in general.

This is really interesting and the imgur picture you linked (with your explanation) explains it really clearly!

But when seeking, why wouldn't any local media playback seek backwards and reconstruct the full frame? It's not like the partial frame after seeking is useful - I'd rather wait 2 seconds while it scrambles (i mean "hurries up") to show me a proper seek, wouldn't everyone?

What was your Internet search for finding that imgur frame? What is this effect called?

>why wouldn't any local media playback seek backwards and reconstruct the full frame?

Most codecs/players do. VLC used to be criticized for being different in that regard. One possible advantage is istantaneous seeking, as there's no need to decode all the needed frames (which could amount to several seconds of video) between the nearest I-frames[1] (the complete reference pictures) and the desired one.

[1]: plural, because prediction can also be bidirectional in time

The use of incomplete video frame data for artistic purposes is called "datamoshing".

I try to use VLC when I can because it offers intuitive playlist support, but for high-resolution H.264 and friends I usually have to switch to Media Player Classic.

VLC is willing to let my entire screen look like a blob of grey alien shit for 10 seconds instead of just taking a moment to reconstruct frames.

And its hardware acceleration for newer codecs is balls. Sucks because otherwise, it's right up there with f2k for me.

>VLC is willing to let my entire screen look like a blob of grey alien shit for 10 seconds instead of just taking a moment to reconstruct frames.

Yes, this is what I was talking about, and yes, specifically for VLC. Plus it's not like playback is so taxing that all cores are pegged at 100% during playback. When I seek, VLC should get off its ass and scramble to come up with the correct full frame then. I'll wait.

I recently bought a camera that has 4k video recording. VLC just gives up playing the video. Even Windows Media Player can handle it. No idea what's going on, but I was really surprised and disappointed with VLC.
See if you can cut a small segment and submit it as a sample to ffmpeg. Hell, see if ffprobe and ffmpeg can play it. Happy to help, if you've got enough upstream bandwidth.
Sure. I'll give those a try tonight. I assume if it works in `ffplay` directly, there's no need to submit it?
I stopped using VLC when I found mpv [0]. I really like it because it exposes everything from the CLI, so once you're familiarized with the flags you're interested in using, it's easy to play anything. For everyday usage it "just works" too, as expected of any video player.

[0] https://mpv.io/

I'll check it out and let you know what I think. Thanks~
How does it compare to mplayer? My biggest complaint about mplayer is it still doesn't play VFR videos well.
I've tried it.

* Sane defaults (encodings and fonts, scaletempo for audio)

* instantaneous play of next and previous videos

* navigation in random playlist actually works

* Easy always on top key binding

* Most mplayer key bindings work

I'll definitely keep on trying it for a while.

Does it include all the codecs by default? I think this was a major reason VLC succeeded the way it did. With all other players (BPlayer anyone?) you needed to find and install tons of codecs while in VLC it just worked.
It has played everything I've thrown at it so far...
man... that's a big manual :)

I think I can find some use for this in certain situations. Still lacks a good playlist building schema.

Isn't the other advantage that VLC can play incomplete movie files? Any other players I have tried 'crash' on incomplete torrents, when VLC just fails until it finds the next I frame.
"datamoshing" is a term I've heard for people deliberately removing I-frames, so P-frames are applied to the wrong base image.
In a course I taught (2010) on music visualizations that's the term I used.

The example I used in the lecture where datamoshing came up was the music video for Charlift's "Evident Utensil"[1]; I always thought this was a neat example.

[1]: https://www.youtube.com/watch?v=mvqakws0CeU

(comment deleted)
(comment deleted)
> MJPEG which is essentially every frame independently encoded as a JPEG.

"essentially" makes it sound like it isn't precisely true. MJPEG is literally just a stream of JPEG images. The framing of the stream varies a bit, but many implementations are just literal JPEG images bundled one after the other into a MIME "multipart/x-mixed-replace" message.

A real fun read. Had an assignment a couple of weeks ago where we used the most k most significant singular values of matrices (from picture of Marilyn M.) to compress the image. H.264 is on a whole other level, though ;)
Now the Question is - Manson or Monroe - and which one would be easier to compress? ;)
I love how you can edit photos from people to correct some skin imperfections without loosing the touch that the image is real (and not that blurred, plastic look) when you decompose it in wavelets and just edit some frequencies.

Don't know in photoshop, but in Gimp there's a plugin called "wavelet decomposer" that does that.

Well, that's the most awesome thing I've seen in a long time! Thanks for sharing.
(comment deleted)
The part about entropy encoding only seems explain run-length encoding (RLE). Isn't the interesting aspect of making use of entropy in compression rather to represent rarer events with longer longer code strings?

The fair coin flip is also an example of a process that cannot be compressed well at all because (1) the probably of the same event happening in a row is not as high as for unfair coins (RLE is minimally effective) and (2) the uniform distribution has maximal entropy, so there is no advantage in using different code lengths to represent the events. (Since the process has a binary outcome, there is also nothing to gain in terms of code lengths for unfair coins.)

This was a good and interesting read. Is h.264 an open standard ?
Doesn't look like it; https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC

> H.264 is protected by patents owned by various parties. A license covering most (but not all) patents essential to H.264 is administered by patent pool MPEG LA.[2] Commercial use of patented H.264 technologies requires the payment of royalties to MPEG LA and other patent owners. MPEG LA has allowed the free use of H.264 technologies for streaming internet video that is free to end users, and Cisco Systems pays royalties to MPEG LA on behalf of the users of binaries for its open source H.264 encoder.

It is an open standard. Anyone can purchase and implement it, and it was developed by ISO. The technologies are not royalty free in the US. Don't conflate the two. *

Edit: I emphasize this mainly because the terms have a specific meaning in standards jargon but also because it places the blame for software patent abuses on the wrong parties (the standards developers rather than the lawyers and legislators).

blame for software patent abuses on the wrong parties (the standards developers)

Uh, anyone familiar with the MPEG process will assure you that the companies involved love (let me restate that: PREFER) to bring in technology on which they own the patents so they get a good cut of the resulting patent pool.

Sometimes this is even done even though it technically makes no sense. Best example: hybrid filter-bank in MP3.

The process also provides no protection or discouragement from patents from semi-involved industry partners appearing later on, etc.

This difference in approach is a stark contrast to the IETF, which is why Opus work, and future AV1 work are happening under the IETF rather than the MPEG groups.

OK, so it is open, but not free ? Is it available for academic purposes free of cost ?
No, not at all. There's even restrictions on distributing H264/AVC video files themselves.
As the grandparent comment says, it is free for non-commercial use.

Of course, this also only applies in countries which enforce software patents.

I wonder if across a lot of videos, the frequency domain representations look similar and if instead of masking in a circle we could mask with other (pre-determined) shapes to keep more information (this would require decoders to know them, of course). Or maybe this article is too high-level and it's not possible to "shape" the frequencies.
It's certainly possible to use any arbitrary shape. The way it really works is that there is a quantization matrix - which essentially is a configurable mask for your frequency domain signal.

Yes, I've dumbed it down in the article to a simple circle to illustrate the point.

Anyone who likes this would probably also enjoy the Daala technology demos at https://xiph.org/daala/ for a little taste of some newer, and more experimental, techniques in video compression.
Note that Daala has been discontinued in favor of AV1: https://en.wikipedia.org/wiki/AOMedia_Video_1

Previously Daala was presented as a candidate for NETVC but apparently this didn't go anywhere? https://en.wikipedia.org/wiki/NETVC

The demos are still neat, and some of the ideas are being used in AV1.
I'm not sure if there's been an official announcement, but I had assumed that AV1 was going to be adopted/ratified as NETVC so that it's got a standards body rubber stamp, as well as the practical adoption/support from browser vendors, GPU manufacturers, streaming sites etc.
Daala continues to be research platform for new ideas. New techniques are a lot easier to prototype in Daala than in more mature code bases.
What are directions for the future? Could neural networks become practically useful for video compression? [1]

[1] http://cs.stanford.edu/people/eroberts/courses/soco/projects...

Perhaps for intra-prediction. I wouldn't hold my breathe.
(comment deleted)
Suppose I have a table of 8-digit numbers that I need to add and subtract for various reasons. Do I A: have a child, train them how to read numbers, add, and subtract, and then have the child do it or B: use a calculator purpose built to add and subtract numbers?

Neural nets are always expensive to train. You'd better be getting something from them that you can't get some other way.

Yes, you don't need the machinery of learning when you already have an algorithm you're happy with. Adding a table of numbers, I don't think anyone hopes to do much better than we already do with our circuits and computer architectures.

With video compression, I think most would agree that there might be better architectures/algorithms that we haven't stumbled upon yet. Whether specifically "neural networks" will be the shape of a better architecture, I don't know. But almost surely some meta-algorithm that can try out tons of different parameters/data-pipeline-topologies for something that vaguely resembles h.264 might find something better than h.264.

Neural nets are expensive to train. But so is designing h.264.

Damn, lost me during the frequency part.
Sometimes it's just easier to learn the math. (I am not kidding.)
H.265 gets you twice the resolution for the same bandwidth, or the same resolution for half the bandwidth.
No, it doesn't. Though that may have been the goal, HEVC has only thus far achieved an improvement of around 25%, not 50%.
In my anecdotal experience, h265 gets me 50-60% improvements in file sizes at the same quality for fairly low quality targets and the gains drop off rather quickly as you increase the quality. For videos where you don't care about the quality all that much, it's superb.
H.265 gets you half the file size for ten times more in royalty fees, or saving 50% of bandwidth for 1000% more in royalty.
Do you have a reference for that?

I was under the impression that the first 100,000 units are free, and then 20c per unit afterwards to a max of $25m.

H264 drops to 10c per unit after 5m units, to a max of $6.5m.

You need to be shipping 125 million units annually to hit the full $25m.

Yes it's more, but it's not quite ten times. And notably if the chip maker pays the royalties, then the content creators don't need to (though that was excepted indefinitely with H264).

Parts regurgitated from a quick google for reference [1]

[1] http://www.theregister.co.uk/2014/10/03/hevc_patent_terms_th...

HEVC got an additional licensing pool in HEVC Advance that demanded significantly greater license fees on top of MPEG LA's.

Said group's demands are basically the reason Netflix started considering VP9.

It is actually more then 10x. The annual cap for H.264 royalty fees is 6.5M from MPEG-LA. For H.265 it is 25M from MPEG-LA, AND 50M from HEVC-Advance. That is a total of 75M. And like others have pointed out there are Technicolor patents fees not included.

So it depends how these royalty works in details. If only the chip manufacture are paying, Mediatek, Qualcomm, Samsung, Intel, AMD, Nvidia, Apple. That is at least 10 players paying maximum. And if you consider small players, the total contribution of Royalty fees to HEVC is 1 Billion / Year. ONE BILLION!! In the life time of a Video Codec that typically run at least a decade, these patents are 10 Billions.

Do you think that is a fair price, i think everyone should decide for their selves.

It also uses the same techniques and principles.
Wow, now tell me how H.265 works!
Related, how h265 works: http://forum.doom9.org/showthread.php?t=167081

This is a great overview and the techniques are similar to those of h264.

I found it invaluable to get up to speed when I had to do some work on the screen content coding extensions of hevc in Argon Streams. They are a set of bit streams to verify hevc and vp9, take a look, it is a very innovative technique:

http://www.argondesign.com/products/argon-streams-hevc/ http://www.argondesign.com/products/argon-streams-vp9/

Heh, happy to see doom9 still alive and kicking. They were the n°1 resource in the early days of mainstream video compression.
It's not really alive and kicking. The forum is still active but the rest of the site hasn't been touched since 2008.
Very well explained. But I could have understood it all without the bro-approach to the reader. You see where I am going with this? Get it? See where I am going? Ok!
Maybe I'm in the minority here but I think it adds a bit of color to an otherwise dry topic to write about.
I remember loving this style when I was a novice, e.g. Beej's networking tutorial. Not a big fan anymore, either, but certainly valuable for (part of) the target audience, I think.
(comment deleted)
Today you need tricks like that to get Twitter-generation people to read even half of that amount of text.
Articles like this are what makes HN great, and not all those repeated links to the visual studio 1.7.1.1.0.1.pre02-12323-beta3 changelog.
"See how the compressed one does not show the holes in the speaker grills in the MacBook Pro? If you don't zoom in, you would even notice the difference. "

Ehm, what?! The image on the right looks really bad and the missing holes was the first thing I noticed. No zooming needed.

And that's exactly my problem with the majority of online video (iTunes store, Netflix, HBO etc). Even when it's called "HD", there are compression artefacts and gradient banding everywhere.

I understand there must be compromises due to bandwidth, but I don't agree on how much that compromise currently is.

>No zooming needed

Isn't the images above the text a zoomeed version?

>Here is a close-up of the original...

It indeed is, as long as we accept the first screenshot as the normal scale.
I took it to mean that we had to zoom to see that the holes were gone in the compressed version.
There's another false claim like that a bit below. I can only assume that the author is close to legally blind, or uses a VGA-res display to watch the page.
Of course while reading the article you are going to be very conscious of detail and image quality because that is the subject matter of the post.

However if that MacBook Pro image was placed on the side of an article where the primary content was the text you were reading, you'd glance at the image and your brain would fill in the details for you. You probably wouldn't notice the difference in that context.

For most use cases, there likely is very little functional difference between the two images. At least, that was how I understood it.

The first thing I noticed was the ringing, which is an artifact of low-pass filtering so it's a nice opportunity to go into problems with that kind of filtering. Other than that I think it was an ok teaser that gives an idea of how compression is done and what the trade-offs are.
I find the 480p setting on my plex server at home actually looks better than most of the 1080p HD streams on the internet.

Although to be fair, I suspect that a lot of times what I'm looking at are mpeg videos that have been recompressed a half dozen or more times with different encoders. Each encoder having prioritized different metrics. So, the the quality gets worse until it doesn't really matter how good the compression algorithm is. Each new re-compression is basically spending 3/4 of its bits maintaining the compression artifacts from the previous two passes.

Yep, me too - more like, if I was blind, I wouldn't notice the difference. Which is why the bitrate is always the first thing I look at when sourcing video.
But given other settings with even h.264 vs. h.265 and the source content, that isn't always a valid metric either.

I mean for fast action scenes, I rarely notice the difference between 720p and 1080p at 10ft away... but different encoding and sources, not just size alone can make significant differences.

Ugh. Comparing the file size difference between a lossless PNG and a LOSSY H.264 video of a STATIC PAGE is absurd. Calling it "300 times the amount of data," when it's a STATIC IMAGE is insulting in the extreme. It really doesn't matter if the rest of the article has insights, because you lost me already.
It's a very high level example of the concept, which admittedly is absurd, but drives home the point for people who might not be familiar with the scope of the numbers involved.
He clarifies right after that he got to those numbers because he used a lossless vs lossy encoder. Really should've kept reading
"right after that".

No he didn't explain "right after that." He rambled on and on, and even after all of that, he STILL doesn't bring up JPG.

It's an inherently stupid comparison to make. You can't polish a turd.

Thanks for the feedback! Sorry if I was unclear. The comparison with PNG is very intentional to illustrate the vast difference in the compression efficiencies involved. I do state the difference clearly here though:

> This concept of throwing away bits you don't need to save space is called lossy compression. H.264 is a lossy codec - it throws away less important bits and only keeps the important bits.

> PNG is a lossless codec. It means that nothing is thrown away. Bit for bit, the original source image can be recovered from a PNG encoded image.

So what's the final car weight? It looks like you stopped at the Chroma subsampling section..
6.5 Ounces! or 0.4 lbs. Thanks for the feedback! I added the final weight into the conclusion.
"Uncompressed" video has been stored, even before computers, as YUV (or some variant), not RGB as this author states. It makes me doubt everything else he says.
Ya'll wanna get the most out of your H.264 animu rips? Check out Kawaii Codec Pack, it's based on MPC and completely changed my mind about frame interpolation. http://haruhichan.com/forum/showthread.php?7545-KCP-Kawaii-C...
a) offtopic

b) Leave codec packs in 2000 where they belong. They are a great malware vector and also good at messing with settings they shouldn't.

>KCP utilizes the following components: MPC-HC - A robust DirectShow media player. madVR - High quality gpu assisted video renderer. Included as an alternative to EVR-CP. xy-vsfilter / XySubFilter(future) - Superior subtitle renderer. LAV-Filters - A package with the fastest and most actively developed DirectShow Media Splitter and Decoders. (Optional) ReClock - Addresses the problem of audio judder by adapting media for smooth playback OR utilized for bit perfect audio.

I'm actually using MPC-HC and AC3Filter to deal with some files where I couldn't hear the centre channel on VLC (on stereo speakers). Everything else isn't really needed.

oh crap it's the topic police. I use it specifically for madVR and interpolating frames for high-quality low FPS anime. It looks really great. The best I've found for this particular purpose. Be nice.
Can someone explain how the frequency domain stuff works? I've never really understood that, and the article just waves it away with saying it's like converting from binary to hex.
It's the Fourier transform basically. There were even some past links on HN that explained it nicely so you might check those first

(Though for images it's in 2D, not 1D which is more commonly done)

It's a bad analogy. Binary and hex are just different formats for representing the same number. Spatial domain and frequency domain are different views of a complex data set. In the spatial domain, you are looking at the intensity of different points of the image. In the frequency domain, you are looking at the frequencies of intensity changes in patterns in the image.

A good way to develop an intuition for the fourier space is to look at simple images and their DFT transforms: http://web.cs.wpi.edu/~emmanuel/courses/cs545/S14/slides/lec... (3/4 of the way through the slide deck).

This analysis of a "bell pepper" image and its transform is also helpful: https://books.google.com/books?id=6TOUgytafmQC&pg=PA116&lpg=....

As for why you want to do this: throwing away bits in the spatial domain eliminates distinctions between similar intensities, making things look blocky. In the frequency domain, however, you can throw away high-frequency information, which tends to soften patterns like the speaker grills in the MBP image that the human eye isn't that sensitive to to begin with.

> Spatial domain and frequency domain are different views of a complex data set.

Or in this case, a real data set.

In the most basic terms, not even talking about frequency, the mechanics of this is that one series of numbers (pixel values, audio samples, etc.) is replaced, according to some recipe or another, with a different series of numbers from which the original can be recovered (using a similar "inverse" recipe). The benefit of doing this comes from the discovery that this new series has more redundancy in it and can be compressed more efficiently than the original, and even if some of the data are thrown away at this point, the purpose of which is to make compression even more effective, the original can still be recovered with high fidelity.
Basically we can represent any signal as an infinite sum of sinusoids. If you know about Taylor expansion of a function, then you know that the first order term is the most important, then the second and so on. Same principle with the sinusoids. So if we remove the sinusoids with very high frequency we remove the terms with least information.
Well done. The only thing that could make this better is an interactive model/app for me to play around with. The frequency spectrum can probably be used while retouching images as well.

A video on youtube led me to Joofa Mac Photoshop FFT/Inverse FFT plugins [1] which was worth a try. I was unable to register it, as have others. Then I came across ImageJ [2], which is a really great tool (with FFT/IFFT).

Edit: if anyone checks out ImageJ, there's a bundled app called Fiji [3] that makes installation easier and has all the plugins.

If anyone has other apps/plugins to consider, please comment.

[1] http://www.djjoofa.com/download

[2] https://imagej.nih.gov/ij/download.html

[3] http://fiji.sc/

I published a set of utilities that I developed for playing and to help myself learn about frequency analysis here, you might find them interesting:

https://github.com/0x09/dspfun