123 comments

[ 2.5 ms ] story [ 212 ms ] thread
I understand AV1 has attempted to avoid existing patents. Is there a consensus that they succeeded? Or might the HEVC patent pool come after them still with lawsuits?
Patent law is complicated and noone can give a final guarantee that AV1 will be fine.

However given AV1 has some large companies like Google, Facebook and Microsoft behind it and they probably have an interest in making this a success they probably asked for their legal teams opinion. I'd say chances are high that it's gonna be fine.

Those three would not be enough for the kind of push we see here, but a lengthy list of major players in tech makes a difference: Amazon, AMD, Apple, ARM, Broadcom, Cisco, Facebook, Google, Hulu, IBM, Intel, Microsoft, Mozilla, Netflix, Nvidia [1]

[1]: https://en.wikipedia.org/wiki/Alliance_for_Open_Media

The cluster of non-AOMedia cos. I'm most curious about is mobile/SoC folks: Samsung, Qualcomm, Huawei, MediaTek.

Not sure what it signals--the optimistic version is they just want to keep their options open at this stage, and figure they can integrate ARM's or Google's design when the time comes. Pessimistic version is that at least one of them would like to impede adoption by using patents + not supporting AV1 on their chips or devices, maybe in hopes of squeezing money out of the rest of the world.

If Apple (a member, as you noted) ships hardware AV1 support and Google and content providers keep backing it, seems like that would put pressure on the holdouts: your flagships may have to go without a feature a competitor has until you get on board.

No idea--I don't know much about these cos' strategy/politics at all. Just speculating like most others here.

Simple AV1 accelerators are small enough that they might put them in the silicon of shipping devices to hedge their bets, giving themselves the ability to enable AV1 or HEVC perhaps as little as a few weeks before a device is released.

There are also substantial overlaps in hardware video decode accelerators, and they could well both share silicon.

The largest owners of video coding IP have agreed publicly not to litigate on the enormous body of existing patents that apply to AV1.
It's pretty hard to definitively succeed. The threat is not just the HEVC companies that aren't in AOMedia; unknown unknowns are a factor too.

Like submarine patents: patent applicants can intentionally delay the issuance/publication (keep amending their application for example), and because the initial filing date is all that matters, they still get to use that patent to go after others.

And the whole situation is complicated enough that I wouldn't be shocked if there are published patents where legal review decided it wasn't a threat (that it doesn't apply to AV1) but someone sues anyway, or review simply didn't consider the key patent/claim at all.

Some things AV1 has going for it:

- Some techniques are already in use out there (like in VP9), are patented by AOMedia members, or are essentially just bumping some numbers to take advantage of stronger hardware (prediction directions, reference frames you can use). So those parts seem less likely to see patent claims, either because they may not be patentable or because we'd've seen it in someone suing over VP9 or such.

- There is a legal defense fund. And, more informally, there are large companies to which it's worth a lot to keep the codec as unencumbered as possible.

- AOMedia members are allowed to use their patents against entities filing patent suits, which might discourage cos. that actually have products (as opposed to pure patent holding companies) from suing, or strengthen the AOMedia member's position in negotiating for a settlement etc. (I wonder whether this part is structured to also allow retaliation when a company e.g. sells their patents to hostile pools or holding companies.)

I'm sure there will be threats and suits (the system just seems to make it inevitable someone will try to profit off this) and it might be someone in one of the HEVC pools doing it in hopes they can help HEVC win. But seems possible AOMedia could win or settle any essential ones (things core to the codec rather than just specific implementation techniques).

If the whole thing failed catastrophically--a lawsuit covering all approaches to implementing some essential codec feature succeeded--AV1 would still likely end up much less expensive and encumbered than HEVC, which of course didn't even try to avoid patents. I also wonder if AOMedia might get to work on a workaround (like, define a codec or profile that's more or less AV1 with the specific feature from the lawsuit disabled, removed or replaced). That could obviously be hugely expensive depending on things like compatibility with existing HW/SW, but could still work out cheaper than royalties or whatever.

"Under patent rules adopted from the World Wide Web Consortium (W3C), technology contributors license their AV1-connected patents to anyone, anywhere, anytime based on reciprocity, i.e. as long as the user does not engage in patent litigation."

https://en.wikipedia.org/wiki/AV1#cite_note-patent_license-2...

We can't know for certain, but with the number of big players joining the alliance, I find it unlikely that a company would want to give up their rights to use AV1.

Not to mention the PR nightmare, boycotting, and developer exodus that would ensue for that company.

Wait, is that "as long as the user does not engage in patent litigation" against that consortium member? Or at all? I can't imagine a big corp. forfeiting the right to all patent actions.
> PR nightmare, boycotting, and developer exodus that would ensue

Perhaps, but I have less and less faith in these 'corrective forces'. Tech companies often seem to discount such consequences, and they get away with it just fine.

More optimistically though, I can't recall any instances of a major tech company reneging on a promise like this. The 'Microsoft Community Promise' has been honoured, for instance. (I accept it's quite possible I'm just being forgetful, mind.)

Is there an advantage to writing ASM without intrinsics?
Yes. For intrinsics, the compiler chooses whether and when to spill variables to the stack. For hand-written assembly, you are explicitly burdened with this yourself, usually causing you to optimize the flow of the function so it minimizes stack spilling. In addition, your function will perform the same regardless of the compiler.

In FFmpeg, we have found in the past that functions written in pure assembly are typically a few % faste than those written in intrinsics. Depending on your use case, that may make it worth it.

The obvious disadvantage is that pure assembly is harder, so writing functions takes a bit longer.

> the compiler chooses whether and when to spill variables to the stack.

In my experience the behavior is stable i.e. doesn't often change between rebuilds (at least in VC++). I can manage this by looking at compiler's output: when I find there something I don't want, be it stack spilling or other reasons for me not liking the assembly, I go fix my C++ code.

> so writing functions takes a bit longer

Also reading is hard, therefore refactoring/improvements is hard, too. I have an impression the assembly is write only language.

> In my experience the behavior is stable i.e. doesn't often change between rebuilds (at least in VC++).

This is less true when you try and support many versions of many compilers, however, in my experience.

Yeah, that makes sense. I usually deliver results of my work in binaries so I can choose the compiler, and inspect compiler output of performance critical parts. Authors of that decoder didn’t have that luxury.
Assembly is very readable with enough practice.

Source: I've ported multiple assembly language games from one assembly language to another.

I think I have enough practice. I find assembly very readable on small scale. But when I need to write larger pieces of vectorized code, higher-level C++ constructs help a lot with readability, following reasons.

Local variables and function arguments are named, as opposed to xmm/ymm register numbers.

I can use (forcefully inlined) C++ functions to compose code from smaller pieces. In some cases, even OOP.

Even in vectorized code there’re pieces running on scalar blocks of cores: loops, addresses calculations, conditions. With assembly you have to use it for the whole things.

Doesn't asm have the disadvantage that you end up spilling if you exceed 8 xmm registers even when running on x86_64 with 16 xmm registers available?

Or do you actually write separate asm for x86 and x86_64?

Many. For me the biggest one is code readability. I am very, very happy with their choice of NASM syntax.

The idea behind intrinsics is that you use one, or several assembly instructions in your code base, letting the compiler manage registers for you. But (especially these days) there is rarely a single instruction that makes your code go faster: you have to write entire large blocks of assembly. That is much better done using assembly than intrinsics.

Not if you use a backend like the one in ICC.
I was trying to remember if autovectorization worked better with intrinsics, but I don't really know that much about it.
Yes, everything works better with intrinsics:

- codegen because the compiler spills better than you, and knows what register to save. It knows about register renaming

- inlining works better

- loop unrolling works better

- any kind of hoisting

And it's also 3x faster to write, and much faster to refactor back to C++. Provided you have the right backend, it will lead to less maintenance issue with about 5% better performance.

>BSD is not copyleft, why?

Really? People are going to complain because a licence is too free? ...

Comments like this will only lead to arguing/flaming/etc.
Since Vorbis, we've all worked to make the codec code as restriction free as possible because we need wide deployment for success. It's more important for a royalty-free codec to win than for these codebases to be copyleft. Once AV1 is the default solution, we can start worrying about software licenses :)
The complaint usually comes in due to an interpretation of "free."

The author includes RMS' opinion, which is true for this particular piece of software, though in many cases it's a warranted complaint. It gets into a political argument usually, but it boils down to this argument: "If I don't require copyleft, then I can't always guarantee that users of my code will get all the freedoms I provided to them."

Here in detail: Why Open Source misses the point of Free Software - https://www.gnu.org/philosophy/open-source-misses-the-point....
Maybe I missed something, but this essay seems to deal with the relationship of Open Source v Free Software, not Free Software v Copyleft Software.
You don't need an explicit essay on that comparison. Morever it's not vs Copyleft, but why copyleft is generally material to the purpose of Free Software.

In the linked entry Stallman primarily explains what is understood as Free Software. The comparison with Open Source merely serves as a device to illustrate the nuances in a more relatable way.

>Another misunderstanding of “open source” is the idea that it means “not using the GNU GPL.” This tends to accompany another misunderstanding that “free software” means “GPL-covered software.” These are both mistaken, since the GNU GPL qualifies as an open source license and most of the open source licenses qualify as free software licenses. There are many free software licenses aside from the GNU GPL.

I find that distinction very much of interest, and sure enough, RMS also has an explicit text available which deals with that matter more directly [0].

While free software protects the freedoms of the user only, copyleft software protects the freedoms of the developer as well. Both types of software are ethical (as mentioned in [1]), so it comes down to personal choice of the original author, who can use it as a tool to ensure their goals.

[0] https://www.gnu.org/philosophy/pragmatic.html

[1] https://www.gnu.org/philosophy/free-sw.en.html

Yes, we have had some remarks.

VideoLAN usually does copyleft (LGPL) projects, so it needed clarifications for our community.

I don't understand how can someone argue "your licence is too free, you should make it more restrictive so fewer people can use your software", but what do I know
The concern is typically from developers (the copyright owners), not end users.
If your definition of a good license is "the terms are as unrestrictive as possible" why not release all software into the public domain?

There's legitimate arguments to be made for more restrictive licenses, even if I don't agree with them all the time.

IIRC, "public domain" means different things in different countries, and can make it hard to enforce as a software license. See https://opensource.org/faq#public-domain

A BSD-style license is probably the closest to public domain while still being easy to read/understand.

The choice makes no difference to who can use this version of this software, but copyleft protects users from future derivatives getting locked away in exchange for forbidding proprietary derivatives altogether.

In the case of popularising a video codec the rise of some proprietary derivative seems an acceptable risk but for some random program what's the upside?

The upside is us getting the iPhone or the PS4, for example.
I take issue with BSD/MIT because of situations like the PS4. Sony took FreeBSD, heavily modified and improved it to make it into the OS for their new console, and upstreamed (afaik) absolutely nothing, nor did they donate a cent. They were perfectly within their rights to do this due to the license, but holy shit it was a dickish move.

Linux would not be in such a strong position, IMO, if they were not GPL. So many companies have been forced to reluctantly release things like drivers for their hardware because they wanted to leverage the power of linux in their products. BSDs will never get contributions like that.

For video decoders though, contributions are less important and so I think BSD is a good idea. We desperately need AV1 to succeed, and that means everyone and their dog using it.

Why is that "dickish"? Was the FreeBSD team upset?
I have no idea. I'm talking about my personal judgement on the subject, not theirs.
Since they wrote software under a permissive license, they're explicitly giving anyone permission to do that. I would therefore assume that they're okay with it.

I'm not okay with it, so if I write something I default to licensing it under the GPL (and I recommend doing the same), but my opinions aren't universal.

When Apache board members were busy defending the fiasco that is OpenOffice a couple of years back one of the give away claims they made was that their big contributors abandoned the project because LibreOffice used code from it under the terms of the license and didn't give back.

Not being required to give back being the entire point of Apache's licensing versus LibreOffice.

They (board members) basically refused to make any connection between this obvious hypocrisy and the failure of their project.

But they followed the license. That's kind of the point, isn't it? That this license doesn't force you to push your stuff back out, like the GPL does.
Well, yes. I did say they followed the license. I think it's a bad license for an OS because it doesn't encourage/require bad community members to open their source. Companies like Sony have no excuse for not upstreaming their improvements. They didn't because they weren't required to.
I think it's a good licence for an OS or for any software because it gets more products pushed out there which all in all gives society more stuff to play with, which is kind of the point of writing free software, at least for me.
That's what the freebsd licence encourages...?

https://en.m.wikipedia.org/wiki/List_of_products_based_on_Fr...

There's plenty of network vendors in the non free list. Are they also dicks for doing the same?

How about Apple using FreeBSD code?

Yes, it does, but I think that's bad. Any large company which takes freely available code and builds a commercial product with it without either upstreaming improvements or making significant donations is bad. Seriously, contribute or get out and write your own damn code. In my opinion.

Has everyone forgotten Heartbleed? Y'all are defending behaviour which got us all in that mess.

Copyleft does not restrict "use," only distribution. In the case of VLC, the only people who would be restricted are people who want to make proprietary versions of it, and I can see why VideoLAN aren't too concerned about these "users."
I look at it as a trade-off between short-term freedom and long-term freedom.

BSD gives more short-term freedom. GPL trades some short-term freedom to attempt to better preserve long-term freedom.

To me that trade is a reasonable strategy, but I also think BSD has a hidden(?) benefit on a different front in the battle for openness. If you have a libre implementation of something but nobody uses it, it's not much use. If the availability of BSD-licensed code induces a corporation to use that instead of developing their own proprietary technology, it might be a net win, especially if that big corporation has a lot of influence over users.

What we want here are more users of the AV1 codec. As such, an implementation that doesn't suck that anyone can use and MIGHT want to contribute fixes/optimizations upstream so they don't have to carry them on their own is a more of a net win than trying to use a specific implementation of that decoder library as a club to open up other software that happens to use it.
> Really? People are going to complain because a licence is too free? ...

Software licenses exist to balance the compromise between "user freedom" and "vendor freedom". In this context, speaking about "freedom" in general ('too free') doesn't make too much sense.

A definition of "free" only meaning "no contractual restrictions" is an overly simplified one, which doesn't recognize the existence of the compromise.

BTW, this distinction is by no means specific to software - think "employer's freedom" vs "employee's freedom".

I'd be interested in comparisons to https://github.com/xiph/rav1e.
Rav1e is an encoder. dav1d is a decoder. (Hence the last letter).
Am I right to conclude that av1 takes like 100x+ the duration of the video to encode? Like it's extremely impractical at the moment?
Not impractical, just resource intensive. :) IIRC extreme optimizations in favor of the decoder even at the expense of encoder burden was an explicit design goal of the codec, to meet the needs of Netflix-esque 'encode once, decode thousands of times' scenarios.
Improvements for performance as well as hardware specific versions of the algorithm or parts of generally come a few years after a specific encoding has a first working release. h.265 for example still has room to grow in hardware accelerated encoding, it's very fast. And x265 being software based is much better, but slower.

It takes time.

It depends on what your usecase is. If you are Netflix and you have a show that gets watched 5 million times then having 100x on encoding is no big deal.

However if you want to do live-streaming to a group of friends, its very impractical.

Its only really worth doing a lot of work the encoder once you are sure its not gone change anymore, so work on that is surely ongoing.

What you are referring to is the encoding time using the libaom reference encoder, which was developed as a research code base to test bitstream features. AV1 (the format) was designed to be used for a variety of use cases including real-time and interactive streaming and it is possible to create a non-libaom based encoder that does so (see the https://github.com/xiph/rav1e project).

And still, for some companies the bandwidth savings of AV1 are worth deploying today even with the longer encode times of libaom. For example, you can try AV1 streams on YouTube right now:

https://www.youtube.com/testtube

So here is a possibly naive question.

From my viewing of Youtube, Google transcodes uploaded videos (typically H.264 stuff) to VP9 only when a certain view threshold is reached, which makes sense from their perspective.

However, I have also noticed that due to the chain of encodes source -> H.264 -> VP9 (latter two available to Google), the VP9 stream is often of noticeably lower quality. Thus, whenever I can, I use an H.264 stream.

This problem will not go away with AV1. In fact, from an archival/local usage standpoint, as others have noted here, AV1 is pretty much impractical due to heavy encoding time increases that will unlikely go away with SIMD as compared to x265 or x264.

As such, from an end user experience point of view, what does AV1 offer that H.265/H.264 do not already?

As others have said, it is for encode once, decode many scenarios. Same quality, less bandwidth and cpu time spent decoding.
Youtube will transcode your H.264 anyway, AFAIK there's no way to prevent that. Hopefully it also saves original file, so VP9 won't be of worse quality.
YouTube encodes are always done from the source. When new codecs like AV1 are introduced they reencode from the originally uploaded source not from other YouTube encodes.

The immediate benefit of AV1 will be felt by people with decent machines but terrible bandwidth as the higher compression will give them higher quality. There's talks given by YouTube employees about this process when VP9 initially rolled out and how different parts of the globe benefitted depending on their tech and infrastructure levels.

TIL what is a recursive acronym.
Have you heard of GNU?
(comment deleted)
I don't know what GNU is, but I know what it's not.
Could someone confirm if dav1d uses hardware acceleration to decode?
If you're referring to SIMD, then the answer is not yet, very soon though.
What does it mean that the ask is written in NASM/GAS syntax? Those are two wildly different asm syntaxes with different macro languages etc.

Is some asm written in one variant and some in the other, or is it all written both ways? Maybe for some targets GAS is used and others NASM (<-- it's this one as answered below)?

x86/x64 is Nasm.

ARM is gas.

> Therefore, the VideoLAN, VLC and FFmpeg communities have started to work on a new decoder

Isn't VideoLAN the same as VLC?

maybe VLC is just one project worked on by VideoLan people ?
Seems so, but they are listed as if they are different "communities". I wonder if it's just bad wording.
If there are people in the VideoLAN community who don't work on VLC, then they are different communities. One might be a superset of the other, but they aren't equivalent.

To me, it's not bad wording to mention communities that have a lot of overlap. There's no rule that says you should only list things that are disjoint.

VideoLAN has several projects, including VLC. But many people confuses both, so they are both listed.
> written in C99 (without VLAs)

Can someone fill me in on why VLAs are a non-feature?

Because MSVC does not support those, and VLA became optional in C11.

That's why we decided to not use them in dav1d.

The most obvious reason is portability. Even though it's been nearly twenty years since the C99 standard, some compilers still haven't implemented VLAs. To make things worse, the next revision of the C standard (C11) made VLAs optional, so these compilers now have an excuse to never implement it.

But even discounting that, there are still some issues with VLAs. For instance, the Linux kernel decided to remove all uses of VLAs they had, because they generate worse code than a partially-used fixed-size buffer, and also make it easier to overflow the kernel stack. VLAs, like their ancestor alloca(), seem convenient but often are more trouble than they're worth.

Very easy to blow up your stack with unsanitized input
Though that is true of a lot of elements of C.
They're universally regarded as a dangerous, difficult to use feature (yes, even by C programmers) and are no longer required in C11. VLAs can have nasty side effects like stack corruption and are a leaky abstraction at best. You can't expose a VLA's size through the ABI, for instance.
> You can't expose a VLA's size through the ABI, for instance.

Can you clarify what you mean by this? It's not clear to me what it would even mean to expose a VLA in the ABI: they are function scoped and as a result it doesn't seem like they could escape to the ABI.

GCC added an extension years back that allows VLAs as struct members and function arguments. Clang had the good sense not to, thankfully.
One of the main issue is no error checking on this kind of "allocation" (it's the same as alloca()). This issue also somehow exists with a fixed size buffer, but at least you get a much more predictable outcome (you have full control on the stack usage, unless you have a user controlled recursion).
You can check for errors with alloca. It returns NULL on failure. You have the same problem as with error checking malloc. On platforms with overcommit (any fork-based UNIX, except maybe Solaris) it doesn't fail at allocation time, it fails when you try to access the memory and fail to map a page. But at least on 32-bit systems it can detect when you exhaust your address space (an actual problem for things like web browsers that still commonly ship 32-bit binaries and use lots and lots of threads).

But yeah, the real reason is that MSVC will never implement VLAs.

I worry you are mixing up alloca and malloc. Alloca allocates off the stack. All modern platforms have irritatingly small stack sizes, and if you overflow them your code just defaults with an annoyingly imprecise error. There is no overcommitting with stacks in practice.
It returns NULL to indicate errors? Not any platform I've used lately.

The Linux man page is in fact explicit that it never returns NULL:

  RETURN VALUE
       The alloca() function returns a pointer to the
       beginning of the allocated space.  If the
       allocation causes stack overflow, program behavior
       is undefined.

  NOTES
       ...
       The inlined code often consists of a single          
       instruction adjusting the stack pointer, and
       does not check for stack overflow.  Thus, there
       is no NULL error return.
As a practical matter it is difficult for an implementation to have alloca return an error value even if they wanted to: it would be both two slow and in some cases "impossible" to know ahead of time in the same way that malloc is unlikely to return null on some systems.
How could alloca return NULL? You would have to catch bumping into a guard page would you not?
_theoretically_ a C implementation could track the maximum stack size it has allocated, detect would be overflows, and report them.

guard pages are one mechanism for doing that.

> you have full control on the stack usage, unless you have a user controlled recursion

IMO, the reality with C is that one needs to be aware of input, whether using VLAs, recursion, or any number of other mechanisms (overflowing calculated values passed to malloc, etc).

It isn't clear that VLAs are a significant departure from what C already provides in the way of unsafety.

Because VLA are considered a design mistake, which made C even more insecure, and as such have been made optional in C11.

Additionally gcc and clang are probably the only C compilers that ever bothered to support it.

pcc, tcc, and icc all appear to support variable length arrays.

It may be more accurate to state that "msvc is probably the only C compiler that didn't bother to support it"?

(Though it would be interesting to perform a more complete survey of C compilers to determine how widely supported VLAs are)

(For the uninformed like me, VLA stands for Variabe-Length Array)
Great, so awesome to see the AV1 ecosystem grow like that. This will make the end of the old patent mafia.

One thing I would like to see is that this runs on RISC-V. Both are very important new open technologies and it would be great to see it run on that officially.

(comment deleted)
"AV1 has the potential to be up to 20% better than the HEVC codec"

Can you clarify what is meant by better here? More compressed?

Bitrate reduction for the same quality. So a file of identical quality - in some metric - will be 20% smaller (e.g. 8 MB instead of 10 MB for a video).
How quality is measured? I never saw any quality estimations in software, e.g. when I'm exporting JPEG image, I'd like to have some kind of result quality estimation, it might be hard to judge it yourself.
Some function of the difference in pixels between the original and decoded compressed version. This is a complicated subject, because a good error function should account for human perception and weight unimportant differences less.
Can anyone test this on a Raspberry Pi 3?
I think if they want wider adoption of the codec, they have to port this to GPU. They are much faster in term of throughput. They are more power efficient per FLOP. Also for players, the result needs to be in VRAM anyway.

Unfortunately, it’s very hard to do in cross-platform way, for Windows you have to use D3D, for iOS Metal, etc… AFAIK, cross-platform GPGPU doesn’t work yet, at least when you want to target wide audience i.e. both Intel/Nvidia/AMD GPUs including older models, with default drivers.

It's always possible to use GL compute or Vulkan compute with MoltenVK/gfx on MacOS/iOS.
Well, some people are working on this: https://github.com/gfx-rs/gfx/blob/master/README.md

I think at this point it's clear that we'll NEVER have a cross-platform graphics API. The incentives for platform holders speak against it, and design by committee means the attempts are so inferior that devs would rather avoid them.

We have to make wrappers. Ideally, they should translate at compile time.

> it's clear that we'll NEVER have a cross-platform graphics API.

I think we will, in a few years. I never used VK in production but I’ve read about it a lot, and I like the technology.

Only I don’t think it happened just yet. On PCs Intel is problematic, VK requires Skylake GPUs (2015), i.e. most PCs older than 2-3 years don’t support it.

Yeah, I bet GPU-aided decoding will get there, but a cross-platform CPU decoder seems like a reasonable place for work now--provide a baseline that can work a lot of places.
Why is this written in C? I would have thought Rust would be the perfect fit for this.
There are some: https://github.com/xiph/rav1e

The projects behind this project are all in C, so I imagine that’s why. They probably don’t want to commit to Rust on the basis of this codec alone.

afaik safe rust would not perform that well compared to unsafe. The main safe part could be the command line or API but using Rust brings you into the Rust ecosystem and it may just not be worth it to do if most of their code is going to be unsafe and purposefully use performance hacks and SIMD.
Safe Rust can outperform unsafe Rust. It really just depends.
I don't see why most code would have to be unsafe. Presumably some of it would, but that unsafe code can generally be wrapped up in a safe interface (this is practically the entire purpose of libstd).
Why is this a stand-alone library and not a part of libavcodec? To avoid LGPL?