Impressive results - on CD-quality audio too, not just speech as with many recent codecs. One thing I was not able to find, either in the blog post or the paper (https://arxiv.org/pdf/2210.13438.pdf), was the size of the model.
EDIT: I downloaded the model from the code (https://github.com/facebookresearch/encodec/blob/3837f0db2a3...). It looks to have 15M parameters in total (this is a CPU-only model, by the way). For inference, I assume we can get away with 16-bit floats, so this clocks in at about 30 MiB. Not particularly favorable compared with LAME on my system (164 KiB), but not at all unreasonable to use - these days apps take 30 MiB dependencies all the time.
The phrasing they use is a bit confusing. "Speech" is not a clear quality description, so it's odd to compare speech to a CD quality recording (presumably 44.1 kHz). Speech can also be sampled at CD quality. If they meant to emphasize the difference in frequency range or dynamic range - there were, perhaps, better ways to do that (e.g. "telephone conversation" vs "symphonic orchestra recording").
My best guess is that all they meant by "CD quality" was that it was targeting full band stereo replication, rather than narrow band as is typically used for speech. Not that it could achieve transparent compression at any particular bitrate.
The tiny music clip in the sample, encoded at 6 kbps, is obviously not any kind of evidence for "CD quality" one way or the other. (The clip itself, if you download it from the page, is re-encoded with 64 kbps AAC.) No way to know how it would stack up against 96 kbps Opus on a stack of CDs with blind testing, I don't think.
It is impressive, but it's also not something I would want to use for music listening. I guess the "10x" goal makes for an impressive chart, but I'd like to know what are the more realistic goals and gains.
The interesting part to me was that they used specifically a 64kbps example, not a higher bitrate that would be more appropriate for music listening. Just speculating, but if they managed to get 10x higher compression rate compared to 64kbps MP3, could they achieve an even higher compression rate when compared to 320kbps MP3? If the algorithm is so good that it can compress audio down to 6kbps with just a few artifacts, would it sound almost good at 12kbps? 24kbps?
Between this effort and the recently announced Google-led multi-channel/immersive audio codec initiative, I am pretty excited about the future of audio streaming and distribution.
> While such techniques have been explored before for speech, we are the first to make it work for 48 kHz sampled stereo audio (i.e., CD quality), which is the standard for music distribution.
Reading this extremely charitably, I think the contrast is supposed to be to speech which is normally uses much smaller sample rates. E.g. AMR-WB samples at 16 kHz. Speex, back in the day, supported up to 32 kHz.
Full quality digitally distributed audio is often sampled at 48 kHz (e.g. Opus - the default audio codec on YouTube and many other sources), so I think "CD quality" is just supposed to emphasize that it's full-band rather than wide band or narrow band.
Since there is no perceptual difference to humans between 44.1 and 48 kHz, the statement about CD quality is perfectly fine - even more so as they operate on the higher sample rate.
Most research in that field operates at 16kHz. Enough for most speech applications, not enough for CD quality. In general, CD quality in this field means a high enough sample rate that you can play all audible frequencies.
I understand all of this reads like pointless pedantry, but a) it feels good indulging in it b) certain word combinations (such as "CD quality") have meanings backed by broad consensus, please use them responsibly so readers do not have to overexert their loose interpretation muscle.
If they mean wide band, say wide band, noone will bat an eye.
Will the next step be piracy detection? If two audio files compress into the same output, can they be considered the same? Similar to soundex or double metaphone?
Shazam produces an embarrassing number of false matches, which I assume are hash collisions or something similar in the world of their fingerprinting system?
Shazams correct hit rate has certainly lowered in the many years I've been using it. Presumably because the pool of music it now is being asked to decipher is becoming unwieldy.
I am permanently pleased to learn one size does not fit all even in gargantuan saas companies.
I doubt it will be that simple. Audio fingerprinting a la Shazam works by identifying key elements in some transformation of the sound, and setting some threshold for the number of matches. I suppose the output of this compression method offers a good way for identifying a song by a bit of sound, but the identification algorithm will be much more complex.
I'd like to see what happens when you modify the stream. If the representation is really that compact, making small changes should change the output considerably. Treat it like a synthesizer, really. That should tell us something about its usefulness in identifying audio.
Our world is near-limitless in bandwidth, but highly restrained in cache size and latency. 10x compression means you can keep 10x more stuff in cache.
And it doesn't matter what level you operate on, cache and latency is always relevant. Whether it's registers, L1, L2, L3, same-core NUMA RAM, cross-core, SSD, disk controller cache, disk, same-location distribution server, cross-location distribution server, tape archive backup, etc, going up a level of cache is always a lot slower regardless of bandwidth if you're doing a small read.
> None of which really seems to make any difference in a world of near-limitless bandwith.
Does to me - I have more than a terabyte of field-recorded FLACs that I'd like to put up on the net for people to listen to or download.
Compressing them to q1 OGG (and I'm not sure that's good enough quality for some of them) only gets that down to about 20-25% of the size (eg 200408_0403.flac at 540M goes down to 128M) - even if I host them on S3 or B2, it's still going to be costing me a not inconsiderable sum if people actively listen or download.
If this can get them down to 5-10% with usable quality, that makes life a lot easier (but obviously would depend on browser support, etc.)
Doesn't really let people play or download them singly - what I'm after is basically a self-hosted Soundcloud where I can add commentary around each file (location, content warnings[1], etc.) and people can listen on the page.
Currently contemplating doing this using Hugo since there's nothing really workable out there.
[1] I've got a lot of recordings of the local square and there's frequently screaming children, screaming alcoholics, the occasional person having a mental crisis, dogs barking violently, etc.
> None of which really seems to make any difference in a world of near-limitless bandwith. It just isn't the constraint it once was.
Tell that to Youtube, Netflix and Facebook - These companies don't have near-limitless bandwith by default, they have to spend many millions to get their networks serving customers.
Or people like me, who can quite easily max out my mobile data-plan with Youtube and high-quality audio.
Near-limitless bandwidth just means that the world generates more ephemeral trash, like another Youtube video or TikTok bandwagoning on the topic of the day. These will be viewed for maybe a few weeks, and then they'll never come up on anyone's feed again, they'll just lie dormant on a server somewhere.
Typically, applications where bandwidth is sufficiently limited to warrant AI heavy lossy compression are also application with tight latency and computational constraints. Not to disparage these results, but the area of very low bitrate codecs is more or less over, voice and audio streaming is a solved problem. In terms of relative bitrate you can do much better with AI, but it will have diminishing returns in the real world.
On the other hand, I think AI-informed video compression will be absolutely transformational; it could enable practical transmission and reproduction of high definition holography (reproduction of an entire wavefront, allowing real 3D scenes viewable from multiple angles) that currently requires a small data-center to process.
I'm excited to see this get implementations for mobile platforms. a 30MiB model sounds like a good tradeoff to stream audio over low bandwidth, or even less reliable mobile connections.
O nice, I wonder how 3 kbps speech compares to Google's Lyra.
https://github.com/google/lyra/releases
1.5 kbps speech is impressive, I wonder how it compares with Codec2 https://github.com/drowe67/codec2
or how all three compare to each other, useful for speech recordings, the only problem is that there is no widespread support yet.
I'm curious what the 'artefacts' are going to sound like with schemes like this.
Using a generative-adversarial network as a compressor (with a 'perceptual' discriminator, how ever that works) probably means that all introduced artefacts will sound entirely plausible. I.e., it's possible that you didn't mishear the word, but that it was mis-compressed to a perceptually very similar word.
Using compression like this for storage is kind of going back to ancient times when songs and stories were passed down through people listening, learning and re-telling them. Each reproduction was a bit different because the person telling it sometimes couldn't remember the details and filled in the blanks with plausible replacements they came up with on the spot.
My only experience with Facebook AI has been "meh". I'm using FB research transformers and it subtly changes what is supposed to be deterministic output.
I have more fun trying to break stable diffusion than getting interesting pictures anyhow.
My favorite song for testing these audio compression algorithms is Adele’s hello because any changes to her voice instantly pop out.
It did a great job of reducing FLAC size from 100mb to less than 1mb using stereo 24kbps preset but audio quality suffers a lot in some places. Maybe training it with 48kbps or 64kbps would make it a feasible alternative for storing music without much quality loss.
In comparison, lame insane preset (320kbps) produces 12mb mp3 with almost indistinguishable quality from flac.
I wonder how well this would work for the basis of a FLAC-like lossless encoder. FLAC works by approximating the audio stream with a lossy linear predictive code, and then storing the LPC encoding and its residuals (i.e. the delta between the original signal and its lossy approximation). It turns out that LPC+residuals are a lot more amenable to lossless compression (via Huffman coding) than the raw audio signal itself.
If the LPC were replaced with this neural network based encoding, would the resulting encoding+residuals also be amenable to lossless compression?
I think the main difficulty is that a neural decoder is allowed to make up lots of plausible phase information, which likely leads to pretty large L2 errors while retaining perceptual quality. So then you'll end up with large residuals even though you might only barely discern the difference perceptually.
The sound of the AI encoder is distinct and probably not suited for music at that bitrate, but would probably serve fine for Facebook videos and podcasts. I'd be really interested in seeing a model optimized to compress human speech...
Reading the paper, it's a reimplementation of Soundstream with a tweaked loss and layer norm added, and adds a 2-layer lstm to the encoder. These changes give a real quality boost over the quality of lyra v2 (which is also Soundstream, but lighter weight). Note that encodec is also using much more compute compared to Lyra v2. Additionally, LayerNorm is adding a lot of latency, which is why they advertise it as file compression instead of as a streaming codec.
The entropy coding seems like a lightweight version of AudioLM.
Yes, I believe it had been done with Google’s Wavenet for example where normally it’s trained to generates speech conditioned on textual input. For fun they trained it on piano music and gave it no conditioning and it improvised piano pieces.
So Jukebox can actually be broken down into 2 components / 3 steps:
A) compression (take 44khz audio and compress into tokens using 3 VQVAE layers)
B) generation (take the compressed tokens and use that on a transformer to generate sequences of those tokens)
C) de-compresss (reverse the VQVAE) -> this is BY FAR the longest/most computationally expensive step
Diffusion currently wont really help with (B) or even (A)... but there is a lot of new experiments going on with (3)... where the decompression using stable-diffusion is providing very interesting results using far less compute... check out JuicyJukebox notebooks (link to come... cant access github colab from my work computer).
I'll have to go and get the source and compare it to AAC. I'm really curious why that was left out of the comparison since it's the modern equivalent of MP3 (aka MP4 audio). It feels disingenuous.
It's a shame that most authors are French and didn't mention even a single time the work previously done by Fabrice Bellard. Plus, no lossless implementation? Maybe it's a good follow up.
71 comments
[ 4.1 ms ] story [ 153 ms ] threadEDIT: I downloaded the model from the code (https://github.com/facebookresearch/encodec/blob/3837f0db2a3...). It looks to have 15M parameters in total (this is a CPU-only model, by the way). For inference, I assume we can get away with 16-bit floats, so this clocks in at about 30 MiB. Not particularly favorable compared with LAME on my system (164 KiB), but not at all unreasonable to use - these days apps take 30 MiB dependencies all the time.
The tiny music clip in the sample, encoded at 6 kbps, is obviously not any kind of evidence for "CD quality" one way or the other. (The clip itself, if you download it from the page, is re-encoded with 64 kbps AAC.) No way to know how it would stack up against 96 kbps Opus on a stack of CDs with blind testing, I don't think.
"CD quality" is 44.1khz over 16bit.
> We achieve an approximate 10x compression rate compared with MP3 at 64 kbps, without a loss of quality.
but then later:
> The key to lossy compression is to identify changes that will not be perceivable by humans
On your second point: they probably meant that there is no loss in perceived quality, of course it's a lossy algorithm.
6kbps sound demo is really impressive! Initially I was turned off by aliasing artifacts (hiss) but to be fair 6kbps is a really really low bitrate.
Between this effort and the recently announced Google-led multi-channel/immersive audio codec initiative, I am pretty excited about the future of audio streaming and distribution.
... but only if you are a bat.
Link?
The flute in particular suffers quite a lot and the triangle disappears completely!
Reading this extremely charitably, I think the contrast is supposed to be to speech which is normally uses much smaller sample rates. E.g. AMR-WB samples at 16 kHz. Speex, back in the day, supported up to 32 kHz.
Full quality digitally distributed audio is often sampled at 48 kHz (e.g. Opus - the default audio codec on YouTube and many other sources), so I think "CD quality" is just supposed to emphasize that it's full-band rather than wide band or narrow band.
Most research in that field operates at 16kHz. Enough for most speech applications, not enough for CD quality. In general, CD quality in this field means a high enough sample rate that you can play all audible frequencies.
I understand all of this reads like pointless pedantry, but a) it feels good indulging in it b) certain word combinations (such as "CD quality") have meanings backed by broad consensus, please use them responsibly so readers do not have to overexert their loose interpretation muscle.
If they mean wide band, say wide band, noone will bat an eye.
> Nb: complaints require an account in good standing at as this is your third strike your account is now suspended.
Near enough is good enough for most people paying-for/selling the tech.
I am permanently pleased to learn one size does not fit all even in gargantuan saas companies.
I'd like to see what happens when you modify the stream. If the representation is really that compact, making small changes should change the output considerably. Treat it like a synthesizer, really. That should tell us something about its usefulness in identifying audio.
None of which really seems to make any difference in a world of near-limitless bandwith. It just isn't the constraint it once was.
And it doesn't matter what level you operate on, cache and latency is always relevant. Whether it's registers, L1, L2, L3, same-core NUMA RAM, cross-core, SSD, disk controller cache, disk, same-location distribution server, cross-location distribution server, tape archive backup, etc, going up a level of cache is always a lot slower regardless of bandwidth if you're doing a small read.
Does to me - I have more than a terabyte of field-recorded FLACs that I'd like to put up on the net for people to listen to or download.
Compressing them to q1 OGG (and I'm not sure that's good enough quality for some of them) only gets that down to about 20-25% of the size (eg 200408_0403.flac at 540M goes down to 128M) - even if I host them on S3 or B2, it's still going to be costing me a not inconsiderable sum if people actively listen or download.
If this can get them down to 5-10% with usable quality, that makes life a lot easier (but obviously would depend on browser support, etc.)
Currently contemplating doing this using Hugo since there's nothing really workable out there.
[1] I've got a lot of recordings of the local square and there's frequently screaming children, screaming alcoholics, the occasional person having a mental crisis, dogs barking violently, etc.
Or suppose your are trying to stream a realistic 6dof VR view of a motorcycle ride through a city.
Or just want to quickly load a realistic scene as a user goes through a portal, without waiting 20 seconds for GBs of texture and geometry to load.
Or you just want a Mozilla Hubs scene to load quickly.
Tell that to Youtube, Netflix and Facebook - These companies don't have near-limitless bandwith by default, they have to spend many millions to get their networks serving customers.
Or people like me, who can quite easily max out my mobile data-plan with Youtube and high-quality audio.
Until truly unlimited mobile plans without throttling are available everywhere for cheap, that simply isn't true.
Near-limitless bandwidth just means that the world generates more ephemeral trash, like another Youtube video or TikTok bandwagoning on the topic of the day. These will be viewed for maybe a few weeks, and then they'll never come up on anyone's feed again, they'll just lie dormant on a server somewhere.
On the other hand, I think AI-informed video compression will be absolutely transformational; it could enable practical transmission and reproduction of high definition holography (reproduction of an entire wavefront, allowing real 3D scenes viewable from multiple angles) that currently requires a small data-center to process.
A laptop can easily compress h. 265 or avi format without requiring intensive hard wire like AI.
Using a generative-adversarial network as a compressor (with a 'perceptual' discriminator, how ever that works) probably means that all introduced artefacts will sound entirely plausible. I.e., it's possible that you didn't mishear the word, but that it was mis-compressed to a perceptually very similar word.
Xerox copier flaw changes numbers in scanned docs [2013]
I have more fun trying to break stable diffusion than getting interesting pictures anyhow.
It did a great job of reducing FLAC size from 100mb to less than 1mb using stereo 24kbps preset but audio quality suffers a lot in some places. Maybe training it with 48kbps or 64kbps would make it a feasible alternative for storing music without much quality loss.
In comparison, lame insane preset (320kbps) produces 12mb mp3 with almost indistinguishable quality from flac.
For those who want to listen to sample: https://a.pomf.cat/pcjynr.wav (first flac then encodec)
The sound of the AI encoder is distinct and probably not suited for music at that bitrate, but would probably serve fine for Facebook videos and podcasts. I'd be really interested in seeing a model optimized to compress human speech...
https://www.mentalfloss.com/article/19727/how-toms-diner-tun...
The entropy coding seems like a lightweight version of AudioLM.
1.6kbit/sec for speech.
Also see OpenAI’s jukebox.
Diffusion currently wont really help with (B) or even (A)... but there is a lot of new experiments going on with (3)... where the decompression using stable-diffusion is providing very interesting results using far less compute... check out JuicyJukebox notebooks (link to come... cant access github colab from my work computer).
https://bellard.org/nncp/
Hell, even the slightly outdated combo of LAME for personal audio library and AAC for multi-channel audio in movies is good enough.