If you out your videos in a Quicktime .MOV file, you can put the intro in a separate file too, which can be placed anywhere (I think relative paths work too), you can even put a URL in there to load it from a server somewhere.
Alternately, you could decompress the intro frames - that might get ZFS dedup working, if the length of the file header is the same with all episodes.
I'm guessing it's because they have to do much more work: western rip group just rip and upload, fansubs have to rip, translate (or find a translation), integrate subbing, recompress and upload. And they'll be going through each episode multiple times to QC timings, subs readability and check that subs doesn't prevent seeing anything of interest, at least for quality groups.
As a result they're always on the hunt for options which could help them do their stuff and:
* they'll find out about other possibly interesting features at the same time as they're poring over release notes
* because they might have to re-export their rips multiple times during QC, they can toy around extensively with features & settings (especially on early exports when they'll most likely have typos & al and will have to redo the export anyway)
tl;dr: they have plenty of opportunities to find and try out bleeding-edge features.
Also, they target much more tech-savvy people who run the latest media player with the latest codec on the latest OS. There is much more chance that the evolutions in the material is readable by consumers; there is less backward-compatibility to support.
Indeed: they were early adopters of many things: Ogg, Vorbis, Matroska (and the advanced features like linked ordered chapters which is designed for the use-case in OP's scenario), h.264 High10 Profile, advanced animated subtitles formats...
I think the long answer has something to do with what they do benefitting from it more than other things, and the short answer is "because they're geeks"; so they perhaps care less about slow hardware adoption of things like 10-bit than other encoders.
They also don't have releasing standards to adhere to. As was mentioned earlier, most fansub viewers are watching on their computers, which can cope with newer codecs, subtitle formats and container features with ease, where-as western TV rips are targeted at viewers using lowest common denominator hardware (and encoded according to a set of standards agreed between groups).
Changes to those standards can cause significant upheaval, as happened when the standard def TV groups multilaterally agreed to switch from XviD in AVI to h.264 in MP4 container.
AIUI, the tertiary anime groups, who take h.264/MKV/ASS releases and re-encode as MP4 with burned in subtitles, cater to those using lower spec hardware but who still want high def video.
I think that Anime release groups are also a lot less "regulated" than US TV rip scene. For animes, you can just ask your friends to help you translate an uncovered anime in your country, you get the "raw" from japan and you sub. For TV, they have rules as complicated as this: http://scenerules.irc.gs/n.html?id=2011_TV_X264_u2.nfo if they don't follow them, they're excluded from the scene.
TV shows may have the same intro sequence, but the overlaid credits are always a bit different.
Hence any compression applied will always produce different results between episodes of a show. This would make any de-duplication extremely difficult.
I think it's a bit of a stretch to say the opening credits are "always a bit different". I can think of quite a few shows display the episode directors and any guest stars after the intro (ie during the first scene of the main episode). I presume they do this to save editing time and cost changing the intro - though there's also the convenient side effect of still having the credited names appear on screen even when viewers fast-forward through the intro.
I have to admit that I had not checked that, but I think Dexter's intro is really the same and the changing cast lists appear during the first scenes afterwards. But for most shows you're right, that would be another reason why it could never work.
I think this was a pretty dumb experiment, and the outcome was to be expected.
There will always be some sort of noise, pixels aligned differently etc., in a production like a tv series, and expecting the encoded output to be identical/matching on a blocklevel is pretty naive to say the least.
I did some experimenting using ZFS deduplication on MPEG2 files, where I encoded hundreds of MPEG2 dvd-sized videos, where 90% of the material was identical (the last 10% was affected by applying different watermarking techniques to the footage), and got some decent deduplication ratio (x1.2:1 or so).. But ZFS deduplication is expensive in memory/SSD, and it was definitely not worth it.
It may be a "dumb experiment" if you already understand how the underlying systems (both the file system, and video encoding) work - but the author did not, and now having done the experiment, they understand more about them.
The reason I assumed deduplication wouldn't be effective on TV shows is because the intro isn't always at the same point as most shows have a prelude these days. Since the preludes are of different lengths, it means that certain things like key frames (if used on that particular codec) would appear on different frames of the intro sequence. Which makes me wonder how effective deduping would be on older TV shows where the intro is at the start of the programme.
As for compression being largely ineffective, the shows are already compressed anyway. In fact file system compression seems to be less relevant these days as most modern file formats have compression built in (even Office documents are just ZIP files). But as the author said, the overhead for compression isn't damaging for the performance* of ZFS - unlike with deduplication
* ZFS compression is particularly performant† when using the newer lz4 algorithm available in OpenZFS - which I'd highly recommend people using if they're not already.
† I know "performant" isn't technically a word. But it should be.
In the evolutionary sense languages, I agree. But there's also a lot of people who strictly disagree with slang, "verbing" and other colloquialisms - instead believing language should be applied strictly (people otherwise known as "language Nazis").
My "performant" footnote was just an attempt to pre-empt such attacks; though ironically including the footnote has now sparked the tangent itself.
It's a fun thing to point out to those people how many of the words they're using to lambast you have actually changed meaning or are new words. They get -very- flustered and sulky.
The semantic problem I see with "performant", which is sharpened by the word transformation, is that there almost always multiple dimensions of performance.
For instance, "performance" in terms of time is relevant to all IT systems, but there is usually some other dimension, for instance, compression ratio or accuracy, that matters too.
Thus, "fast" is a better adjective when it applies because it is more clear.
As for accuracy, well you'd expect any and all deflation compression to be 100% accurate anyway. Anything less than 100% would corrupt your data (unlike with audio / video "lossy" compression where you can remove / group non-perceivable data)
Since videos from the iTunes Store are encrypted, it seems like the individual files should look like random noise to the filesystem anyway. Unless there is something I don't understand happening.
There's basically no way to make this work because, as has been mentioned, in compressed formats there's every chance that title sequences will occur at different times in the show, so will occur at different points in the group-of-pictures sequences in compressed media.
And especially for rips from an already-compressed signal source (like TV), the odds are well against a sequence, although identical visually, encoding into exactly the same sequence of bits digitally, as there are so many factors that can vary the datastream you receive, even before it's ripped and re-encoded.
To stand any chance of this working, you'd really need to be storing and comparing uncompressed frames direct from the master, before ANY kind of variable encoding or compression. But if you have enough storage to be working with files on that scale, deduplication is probably not your biggest concern. :)
A codec that could search for reference frames from large collections of media might be pretty interesting for some applications, especially ones with large storage demands and repetitive material. The indexing system would be an interesting problem since it would have to be very low overhead. Wonder if any systems like this exist?
A few bits of change in uncompressed video can change the following stream in the compressed video completely. Deduplication isn't intelligent enough to understand "video" and compress for that -- that's what video codecs are for. So it's not surprising that deduplication would not prove useful for compressed video. You could see the advantage in uncompressed streams.
I am curious and did not read the docs: Does ZFS deduplication work on parts of files? I would have expected that filesystem deduplication just deduplicate complete files (for conveniance). Deduplication of file parts could of course be good for some types of files, with a fixed structure and with no "noise" -- but with videos, there is no chance I guess without manually linking parts.
Yes, ZFS deduplication is not very useful when trying to deduplicate encrypted and compressed video files. Encryption is supposed to make the contents of the file look random. Compression is itself a form of in-file deduplication. Where ZFS deduplication will be extremely useful: on a SAN in a video editing studio where originals are kept in raw format. Even then, file-level and block-level dedups will probably be the least effective means of deduplication. What we need is byte-level deduplication which computes anchor points from which data ought to be deduplicated. Even then, it sounds like the dedup feature would be scripted so that it be a part of the tooling, and would be updated with the team's projects. I would go as far as to say the author chose a deliberately eye-catching headline and coupled up a few Linux commands with some "hilarious" GIFs to address the wannabe geeks out there while displaying a staggering pile of shamelessness to even the remotely competent of his blog's visitors.
The files are not encrypted. I mention the aspect of doing it with files that are already compressed and that I never expected it to work for that very reason. I just wanted to to for once actually do it and I don't see why you need to get so mad about that.
Unless you stripped the DRM from the files, yes, they are encrypted. At least according to the Wikipedia page on FairPlay [http://en.wikipedia.org/wiki/FairPlay], all the iTunes files are encrypted.
Also, wouldn't proving it doesn't work by setting up a deliberately flawed scenario amount to a bit of blogsturbation? Should I write an article about "finding out whether my computer will turn on even if it's unplugged"? Answer: no. But read my article about me TRYING it just to be sure.
I think you should write a follow-up article where you splice together raw video files and include a similar segment in all the different files. Then, put that through the dedup test. I would actually be interested to see the ability of ZFS implementations to find worthwhile anchor points within files and do smart byte-level deduplication.
I can see where he's coming from. Any differences in the data caused by encryption or compression of the final video files don't even come into play here unless the raw video data they are produced from is bit-for-bit identical from one episode to the next, say in a precise lossless format like CCIR 601. Maybe this is true at the production company but I wouldn't be surprised if it isn't. It would actually be somewhat interesting to know what the typical workflow at the production company is for gluing the title sequence together with the rest of the show. Your blog post doesn't do enough to clarify these issues.
If you were going to do this for science, a good start would be to analyze the intro of shows in question, using rudimentary tools such as md5sum on a per-block basis.
You would then quickly discover that the chances of huge gains would be small, without ever even looking into the ZFS aspects.
ZFS supports byte-level deduplication where it can compute anchor points so that it works on unaligned data. However, quite a few a few assumptions about the entropy of the data set could have been inferred from the "iTunes TV Show" file choice, which clearly means the files would be compressed and encrypted.
This is incorrect. Byte-level deduplication does work with anchor points, but it is extremely expensive. Jeff decided to let the application do byte-level deduplication and implement block-level deduplication instead. This gave us something that was reasonable to implement at the filesystem level.
Reminds me of the DVD sets of Sailor Moon I made years ago where I spliced the same video sequences in for the intro and outro. I never got so far as deduplicating the henshi sequences because there often are different voice-overs during them.
Perhaps I'm just in a bad mood today, but really, why must technical articles be festooned with meme GIFs featuring abusive language and children injuring themselves? What does it add? If your content is so boring to you, or to your prospective audience, that you think it needs livening up with that sort of garbage, maybe just don't write it in the first place.
I tend to agree with you, but the article wasn't supposed to be very technical in the first place, and: both those GIFs are from Dexter, the show of which I used a season in my tests. It's not completely unrelated.
I didn't realise they were from Dexter, though I stand by what I said. Clearly the masses on HN agree with your choices, though, since I'm getting down-voted!
I'm writing some technical documentation for a project right now - perhaps I'll put in a few tasteful images from Baywatch to make sure the project managers who have to read it will make it to the end.
Tastes vary, and what one finds offensive another might find hilarious. As long as we all get along in public, it shouldn't matter too much what makes us laugh when among friends.
I challenge you to record 30 seconds of a fixed test pattern from TV on two separate occasions and have it encode to identical bits. I don't think it can be done.
Many media formats (including all digital TV formats) encode a rolling hardware timestamp of 33 bits (or more) which won't be the same for two separate segments by random chance. Audio, video, subtitles and other metadata will be ordered differently in the stream because they all comes from sources that have their own separate clocks. Synchronization between clocks at different stages in the media pipeline will cause different frames in the sequence to be dropped, padded, made into keyframes, etc, which then affects every bit in subsequent frames. TV stations use time-based watermarks. TV stations use digital compositing software that may retain bits from previous frames on an ongoing basis. Many studio media pipelines still involve analog steps.
The list of complications goes on.
Unless your pipeline is lossless, uses only digital sources and uses totally synchronized clocks for all stages, you're about as likely to get an MD5 hash collision by accident as you are to get any two non-trivial sequences of compressed video to be bit-identical.
Could make for an interesting codec idea though! A codebook based compressor that works on corpora of audio/video seems doable. More interesting would be a codec that aims not only to exploit models of human perception to throw away bits, but also exploits those models to produce "canonical" compressed forms that would be suitable for byte/block level deduplication. Of course, interesting things are hard, and errors would be hilarious. (See: Xerox and their copiers that change letters and numbers in documents...)
Also, I don't know if this is already implemented in some codec, but it would be interesting to re-use key frames from previous shots. For instance when you have a dialog and the camera switches from one person to another, a conventional codec probably creates a new i-frame for each shot whereas an intelligent codec would recognize that this is the continuation of the previous shot.
I agree for TV, but what about the case of the author, where he gets the episode from iTunes? Blu-Ray and iTunes episodes have a chance to have the intro timed perfectly the same as they are probably exported from a kind-of video editing workflow where they copy-paste the intro. At least, that's what I imagine. So the intro in their workflow is probably the same (I mean, the same video file), but somehow the export to h264 makes it different for each episode. Maybe the psy optimizations use dithering or another trick involving randomization.
I've always dreamed that subbers would just provide one version of the credits in their season packs and provide a bash/mencoder/whatever script to glue the bits together after download. We're wasting so much bandwidth on opening and closing credits that people end up skipping anyway, it's crazy...
54 comments
[ 2.8 ms ] story [ 92.7 ms ] threadAlternately, you could decompress the intro frames - that might get ZFS dedup working, if the length of the file header is the same with all episodes.
Those groups often seem strangely bleeding-edge. They always adopt new features years before western groups will... I wonder why.
As a result they're always on the hunt for options which could help them do their stuff and:
* they'll find out about other possibly interesting features at the same time as they're poring over release notes
* because they might have to re-export their rips multiple times during QC, they can toy around extensively with features & settings (especially on early exports when they'll most likely have typos & al and will have to redo the export anyway)
tl;dr: they have plenty of opportunities to find and try out bleeding-edge features.
I think the long answer has something to do with what they do benefitting from it more than other things, and the short answer is "because they're geeks"; so they perhaps care less about slow hardware adoption of things like 10-bit than other encoders.
Changes to those standards can cause significant upheaval, as happened when the standard def TV groups multilaterally agreed to switch from XviD in AVI to h.264 in MP4 container.
AIUI, the tertiary anime groups, who take h.264/MKV/ASS releases and re-encode as MP4 with burned in subtitles, cater to those using lower spec hardware but who still want high def video.
Hence any compression applied will always produce different results between episodes of a show. This would make any de-duplication extremely difficult.
There will always be some sort of noise, pixels aligned differently etc., in a production like a tv series, and expecting the encoded output to be identical/matching on a blocklevel is pretty naive to say the least.
I did some experimenting using ZFS deduplication on MPEG2 files, where I encoded hundreds of MPEG2 dvd-sized videos, where 90% of the material was identical (the last 10% was affected by applying different watermarking techniques to the footage), and got some decent deduplication ratio (x1.2:1 or so).. But ZFS deduplication is expensive in memory/SSD, and it was definitely not worth it.
As for compression being largely ineffective, the shows are already compressed anyway. In fact file system compression seems to be less relevant these days as most modern file formats have compression built in (even Office documents are just ZIP files). But as the author said, the overhead for compression isn't damaging for the performance* of ZFS - unlike with deduplication
* ZFS compression is particularly performant† when using the newer lz4 algorithm available in OpenZFS - which I'd highly recommend people using if they're not already.
† I know "performant" isn't technically a word. But it should be.
My "performant" footnote was just an attempt to pre-empt such attacks; though ironically including the footnote has now sparked the tangent itself.
For instance, "performance" in terms of time is relevant to all IT systems, but there is usually some other dimension, for instance, compression ratio or accuracy, that matters too.
Thus, "fast" is a better adjective when it applies because it is more clear.
As for accuracy, well you'd expect any and all deflation compression to be 100% accurate anyway. Anything less than 100% would corrupt your data (unlike with audio / video "lossy" compression where you can remove / group non-perceivable data)
And especially for rips from an already-compressed signal source (like TV), the odds are well against a sequence, although identical visually, encoding into exactly the same sequence of bits digitally, as there are so many factors that can vary the datastream you receive, even before it's ripped and re-encoded.
To stand any chance of this working, you'd really need to be storing and comparing uncompressed frames direct from the master, before ANY kind of variable encoding or compression. But if you have enough storage to be working with files on that scale, deduplication is probably not your biggest concern. :)
The seeking might make it impractical.
Also, wouldn't proving it doesn't work by setting up a deliberately flawed scenario amount to a bit of blogsturbation? Should I write an article about "finding out whether my computer will turn on even if it's unplugged"? Answer: no. But read my article about me TRYING it just to be sure.
I think you should write a follow-up article where you splice together raw video files and include a similar segment in all the different files. Then, put that through the dedup test. I would actually be interested to see the ability of ZFS implementations to find worthwhile anchor points within files and do smart byte-level deduplication.
You would then quickly discover that the chances of huge gains would be small, without ever even looking into the ZFS aspects.
Oracle seems to be doing alot of the lately where they are posting really old collaterial
I'm writing some technical documentation for a project right now - perhaps I'll put in a few tasteful images from Baywatch to make sure the project managers who have to read it will make it to the end.
Many media formats (including all digital TV formats) encode a rolling hardware timestamp of 33 bits (or more) which won't be the same for two separate segments by random chance. Audio, video, subtitles and other metadata will be ordered differently in the stream because they all comes from sources that have their own separate clocks. Synchronization between clocks at different stages in the media pipeline will cause different frames in the sequence to be dropped, padded, made into keyframes, etc, which then affects every bit in subsequent frames. TV stations use time-based watermarks. TV stations use digital compositing software that may retain bits from previous frames on an ongoing basis. Many studio media pipelines still involve analog steps.
The list of complications goes on.
Unless your pipeline is lossless, uses only digital sources and uses totally synchronized clocks for all stages, you're about as likely to get an MD5 hash collision by accident as you are to get any two non-trivial sequences of compressed video to be bit-identical.