71 comments

[ 3.4 ms ] story [ 134 ms ] thread
Very interesting! I was relieved that I managed to pick out the uncompressed WAV in most of them even on MacBook speakers - I failed with Jay Z and Coldplay as the telltale snare and cymbal artefacts are hidden under layers of compression already (and the "snare" on Jay Z's song is a sample anyway). The real tell-tale song was Suzanne Vega's a cappella song, where the MP3 versions sound like she's got a terrible snotty cold.

Modern songs have much compression and hard limiting which makes everything distorted (you can hear distortion on the Coldplay sample I think towards the beginning, unless I am mistaken?). Other offenders for brick wall limiting are most Chili Peppers albums, Paul McCartney's Memory Almost Full, and California Breed's (sadly) one and only album. Jason Bonham's snare gets lost for 75% of the album.

Interesting that you were able to distinguish the 320 MP3s from the uncompressed files. Couldn't make out any differences at all on decent headphones. Even the 128s were pretty good esp. for the more modern mainstream music where the quality is questionable from the beginning :)
I personally think the samples are not ill-suited for the purpose of distinction between the various bitrates.
The 128kbps MP3s are easy to distinguish - they sound like audio in a washing machine, going round and round. Listen to cymbals that instead of going CRASH go CBBRASPHHHHHH with the ending being the giveaway. Snares are also affected as they won't have such high frequency peaks so lose their SNAP sound.

Bass drums are also easy to pick out - if there isn't much bottom end then you know it has been compressed.

The a cappella version sounds like she's got a throaty cold on the MP3s. The SSSSSes sound different on MP3s too - it ssssoundsss like they've got a lissssppppp with MP3s.

You may find it easier to listen quietly because then your ears only pick up the highlights instead of getting fatigued with too much volume (where they will shut down after a while).

Haha, yeah. Multi-band compressed till the waveform is a bar-code.
Why was I downvoted? It's very much true.
It is indeed very true, with compressed samples put through more compression and then limited. Look at the waveforms of some tracks and they'll be like square waves.

Rush Vapor Trails is the worst offender, perhaps?

Same here. Missed the Coldplay one because I couldn't find artifacts in that "solid wall of sound" style production. The rest were pretty obvious. This with middle-of-the-line Sennheisers, a cheap headphone amp, and whatever sound chip is in my laptop.
Not a HP laptop with Beats is it? :-)

You'd hear nothing but BASS

Want to quickly win? Have a slow connection and pick the one that loads the slowest.
I knew my Third World citizenship would be of some use one day! haha
Exactly what happened to me. Use a crappy 1Mbit WISP connection and raw audio takes ages to load
Apparently I prefer 128. My definition of better was crispness. I found the uncompressed to be muddied, at least coming through a Bose Color Soundlink. 128 was bright and clear.
I remember reading somewhere that people who grew up with 128kbps audio (Generation Napster) prefer the sound of it.
I'm not one of those people who automatically trashes Bose equipment, but Bose does a lot of processing/EQ/"coloration" to music to make it sound more pleasing.

There is absolutely nothing wrong with preferring that! Nearly all consumer-oriented speakers/headphones do this, because there are other models specifically intended for monitoring use. However, it does make Bose equipment poorly suited to engineering-type applications such as recording work, comparing mp3 compression algorithms, etc.

Good point. I do wonder if Bose washes out the middle.
I feel sorry for people who can tell the difference between high bitrate mp3 and wav because they then have to spend extra on headphones and pono players and amps and whatnot just to elimate the perceived difference that the rest of us are happily oblivious of.
While I can tell the difference (6/6 in this test), in practice, it doesn't actually matter to me personally. I'm usually listening to music while there are a cacophony of other noises going on (talking/cooking/driving), so the slight difference in the way cymbals are rendered or the loss of nuance in a large bass hit are far from the primary degradations to my listening experience...
It is not really about people IMHO but about their headphone and amplifier. Better equipment require more expensive up-keeping, that is usual.
Exactly. Most people could hear it with pro equipment. It's a dangerous cycle. Buy better gear, hear more, repeat.
It's all about ear training! I couldn't tune a guitar by ear at one point but now know how out of tune a guitar is by ear, and know if their string is sharp or flat (it helps to remember a song in that key and replay that note in your head compared to what you're hearing).

The same is true of audio quality. I thought 128kbps MP3s were fine for years (I even had some MP2s somewhere) but now they sound like sludge.

yes, but this is like saying you'd prefer poorer eyesight just so you can wear glasses as a fashion statement. I have the same problem with beer - people seem to love it, but all I ever taste is sour and bitter. Sure I save money on beer, but it's an experience I miss out on.
The people I'm jealous of are the people who can listen to 128kbps and below - at that point I can't enjoy the music because I fixate on all the compression artifacts.

I prefer to keep FLACs (lossless compression, so just like a WAV except smaller) myself, just so I can re-encode to my lossy format/bitrate of choice without incurring generational loss. 320kbps is probably clean enough but who cares, disk space is cheap.

But for listening, 320kbps MP3 is A-OK, I don't know whether or not I could ABX it but I don't hear enough artifacts to trigger me. For more situations where size matters, I use V0 or V1 (depending on what I'm playing it on and the type of music). At V1 I start hearing a few artifacts in tough passages, but it takes an enormous chunk out of the filesize relative to FLAC.

I used to have an iRiver H320 - with replacement firmware (RockBox) I could even play FLAC on it. I eventually replaced the 20GB HDD with a 16GB CompactFlash as a ghetto SSD, used to get like 20 hours of battery life on it. It could also do optical out, recorded in MP3 or WAV, and was super easy to change parts on. Great player for a hacker. I used it daily and hard for probably 7 years before the screws would no longer grab, I need to get it back out and see if I can get it going again.

This was surprising. With all that poking fun of audiophiles, I expected there would not be much of a difference.

I got 5 out of 6 correct, and the one I missed was pretty near miss (I picked at random between 320 kbps and uncompressed sample). And these were quite clear choices, many times I just needed few seconds: 128 kbps sounded worse every single time, 320 kbps vs uncompressed was a bit harder, but still pretty noticeable if I paid attention.

It wouldn't probably make a big practical difference for a typical "background noise" listening, but it may have impact if you just want to sit back, relax and focus on music (lower bitrates for me sounded "muddled", losing details in high frequencies).

BTW I'm no audiophile, no special audio gear, just cheap (but decent) 9 EUR in-ear headphones plugged into a notebook.

My results were similar. I got 4 out of 6 and the 2 I missed I picked the 320 kbps. It was more difficult to tell the difference between uncompressed and 320 kbps when the composition was busier, as with the Coldplay track. The piano track at the end was clear as day, the uncompressed version was warmer, fuller, had clearer reverb. Same with the Vega track.
> With all that poking fun of audiophiles, I expected

> there would not be much of a difference.

The fun poking is mostly about things like 192kHz sample rates and thousand dollar power cables. It's uncontroversial that people can hear mp3 artifacts.

192kHz sample rate are unnecessary. In truth, 96kHz is common for live too (as it is half the latency of a 48kHz system).

48kHz and 24bits is what most audio is recorded at nowadays, then downsampled with aliasing for CD quality audio.

> 96kHz is common for live too (as it is half the latency of a 48kHz system).

I don't really understand this, but I guess what it means is that the first sample of a digital signal gets from some place to some other place a ninety-six thousandth of a second more quickly. I'm not sure why that would matter. It's the time it takes sound to travel 4mm, which seems inconsequential, and the overall shape will be the same phase.

In many cases (like, if there's a computer involved anywhere), a high sample rate means you need higher latency to avoid the risk of underruns.

If you've got to send it from stage to the mixing console at the back and then back to the stage or wherever the amps powering the line array is, then latency becomes more important? What if the console is half a km away?

Also, if the sound processors are performing calculations in the idle time between samples (like calculating FIR filters or something like that) or in the idle processor time remaining after processing the sample, a higher sample rate will mean the calculation gets done faster (and is therefore audible faster). Else you'd change a setting and wait to hear it (and it would be noticeable perhaps), I guess?

> What if the console is half a km away?

Then a ninety-six thousandth of a second's worth of latency seems pretty irrelevant? Even if you were decreasing the latency by using a higher sample rate, which you aren't.

The signal coming out of a FIR filter will come out at the same time whatever the sampling rate. I guess it's conceivable, if you have no buffering whatsoever, that the very first sample will come out slightly quicker, but that is honestly irrelevant. The overall signal will have the same timing at either sample rate. Unless you've had to introduce more latency to cope with the demands of the higher sample rate.

I meant that the FIR might be calculated between samples, so the higher the clock the faster the calculation.
Yeah, enthusiasts were A/Bing MP3s a decade ago. There are new frontiers now.
One person, 5 out of 6, could also be attributed to chance.

The random average for a large number of people and 2 options would be getting 3 out of 6 (like a coin toss). But in the samples would be several 4, 5 and even 6 out of 6 too. In this case it's like 15/18, while is very good but still possible.

This was surprising. With all that poking fun of audiophiles, I expected there would not be much of a difference.

What I see more and more online, is that people are falling into a pattern of poking fun of groups without really understanding the scientific, factual, or ideological basis for doing so. Furthermore, most people fall into the pattern after a semantics-free pattern match, quickly making a decision without substance. (One might suppose that the true priority in these situations is the opportunity to have fun at someone's expense, not the ideological or scientific issue at hand.)

When I was college aged, we called such jumping to conclusions "prejudice." One is coming prematurely to a conclusion, possibly contrary to a properly informed decision. Even in such a vaunted forum as HN, I see people proudly announcing how they have jumped to a conclusion based on signalling. How is this any different from a Mad Men character deciding another's credibility based on their alma mater and the cut of their jacket?

When it comes down to it, the "audiophile" set has myths and disinformation floating around within it mixed in with actual science. Note that this is true for any set of people derived from a shallow labeling, like "programmer."

I am guilty of poking fun at audiophiles. I got 5/6 correct; the error was 320kbps. 128kbps was clearly inferior every time. And I was listening on my Mac's speaker with volume at 60%. Now I'm going to have to re-rip my CDs into FLAC. I'll admit I was wrong... a little.
I poke fun at the audiophiles all the time, and I got 1 out of 4 (I didn't try them all). So we're even?
I did this yesterday; I picked 320 / WAV / WAV / 320 / 320 / 128. I cannot distinguish 320 from WAV, so I was mostly just trying to spot the 128 and pick one of the other two. I found it a bit fatiguing to listen to so much new music.

Also, it's hard to know if I'm listening to a WAV of some 808 drum loop or an actual recording of a snare/cymbal encoded. The classical piano seemed easiest to spot the 128.

I listened through IEMs plugged into a Macbook Air in a quiet room and I wouldn't say it was easy.

Similarly, I got 3 wavs and 3 320s. I could distinguish the 128s but not between the other two. Please note that I did this wearing studio headphones (shure srh840).
Same here 3 of 6. I always spotted the 128, but the others were a coin toss. This was on an iPad (chrome browser) with Bose over-ear phones.
I'm old enough to be losing auditory acuity, and I really despise golden-ears equipment reviews, especially of speaker cable, but I find the qualitative tests for compression technologies unsatisfying.

Taking a gang of schlubs out of a shopping mall and playing audio, video, or still images to them and asking for an opinion could be OK in some contexts, like "Can you tolerate us putting this low-bit-rate codec in your mobile phone?" If the answer is "Huh, can't tell" then go ahead.

But is a JPG of an Ansel Adams print still a work of art if 90% of those same schulbs can't tell the difference?

For most people it's like:

In self-assured comments in forums? Very well.

In actual, properly conducted, A/B tests? Not so much.

The digital audio system here in the UK is terrible. They divide up the available bandwidth so much to cram in more channels the stations sound worse than FM (and they use MPEG2 audio too, which makes it worse). Last week, they crammed in a special Eurovision channel and reduced several other channels from 128kbps to 112kbps to cope. 1xtra (the station I listen to) suddenly sounded so "dull" and slushy.. even just with a 16kbps drop! I had no idea what was going on so asked on DigitalSpy and learnt the above.

Long story short, my takeaway is that perhaps it's not just about absolute levels of quality, but also what effect minor effects in bitrate can have on the underlying codec.

Is this with a satellite feed or Freeview? I have Sky (basically Freesat because I won't want to pay Sky any money thanks) and the quality is abysmal, even for video. The audio is compressed massively and LOUD all the time, and the video is blocky as can be, with snowy or rainy scenes being unintelligible. You can notice on some adverts that the small text at the bottom is completely unreadable.
No, DAB (for people overseas, DAB is a terrestrially transmitted digital audio service - imagine FM radio but digital). If I'm not in the car, I'll listen online instead (notably, BBC stations sound better streamed over 3G but this has big UX implications when driving).

The quality on Sky used to be substantially better than DAB when I used it 10 years ago, but I don't know about it now. That said, I have Sky TV and the picture quality is variable. Most of the channels I watch are fine and all HD channels are very clear.

Informative, thank you. I am glad to know that Sky is decent quality on the HD channels. Strangely they believe that you spend your entire life watching TV and can't comprehend you doing anything else (like typing comments on HN at 9 in the evening.....)

A chap at work listens to DAB over some big Questeds and the glitching and artefacting from Radio 2 (ugh I hate hearing that all day) is irritating (down to poor reception). Those Questeds sound great if proper audio is put through them though! It seems a waste to shove DAB Radio 2 and Jeremy Vine and his argument "show" through them.

Sadly the only way you'll get a better experience from your coworker's DAB radio is to convince them to switch to Radio 3. It's the only DAB station in the UK with a 192kbps bitrate (although only 256kbps is considered to be better than FM under perfect conditions).

(Well, technically, DAB+ would make things a ton better as it uses AAC instead, but progress is moving forward very slowly in rolling it out..)

My ears suck :-( and I tell myself I'm an amateur musician too.
With a dac and decent, but not pro, headphones I could pick out the WAV, but I was always unsure.
I can hear the difference between mp3(320) and flac/alac easily on my stereo both in the car and in my room. The funny part is that you can easily recognize mp3 compression based on high it trims the high frequencies and the high hats are affected by it quite a bit. You can try different lossy formats but i guess it is just damage control. The only reason I use alac and not flac is because iPods only work with alac (and it is limited to 16bit, 44100Hz) and my car only supports iPods :(
Or you think you hear the difference because you actually know which files you play. Do a double blind test and you will fail. Professional musicians fail at 320 vs original tests.
Nope. I can pass the double blind test easily.
Not to mention that I am more of a sound engineer who has a lot to do with sound quality while musicians care less about that, I know several musicians who could not setup even basic sound system. It is also pretty common that DJs (and other types of musicians) require the roads to help them to achieve the best sound of their music. The musician is in charge of the music, while the sound engineers are in charge of the sound. Quiet different subjects.
This test reinforced what I've found in the past:

1. I can reliably tell the difference between 128kbps mp3s and higher-quality files

2. I can not tell the difference between 320kbps CBR mp3s and uncompressed originals. (Or between ~256kbps VBR mp3s, 256kbps iTunes Plus mp4s, and uncompressed originals... though those weren't a part of this test)

About the same as what I've AXB'd in the past as well. Most tracks at ~256kbps AAC are the same to me as the uncompressed version (some doesn't compress that well so still arifact, but are rare). Around 128kbps is fine, but I can pick out which is which. Lower than ~100kbps is awful - the arfiacts are instantly obnoxious and I hate listening to it.

It doesn't really apply to streaming, but I keep my music as lossless files despite that. It removes any questions of "Could this sound better?" and I can transcode to a device-appropiate format without compounding the quality loss (desktop has lossless files, laptop has a 256kbps copy, mobile devices have a 128kbps copy)

This requires a proper, double-blind "A X B" test. Sample X is guaranteed to be the same article as either A or B, which are different. You, the subject are asked whether sample X is identical to A, or B.

Double-blind means that the person administering the test doesn't know whether A is the compressed file and B the uncompressed or vice versa, and whether X is a copy of A or B.

Without a properly conducted test, users who want there to exist a difference between 320 kbps that they are able to hear will convince themselves that they can hear it.

But likewise, those who believe they shouldn't be able to hear any difference will convince themselves that they don't.

The A X B test eliminates this, because though you know that X is exactly the same article as A or B, you don't know which, and you don't know which of A and B is the higher fidelity one. If you believe that there is no audible difference, you can at best randomly guess at the identity of X. If a large number of subjects are tested and they all believe that A and B sound the same, the distribution of their identifications of X will be consistent with a random binary choice. If there are some subjects who can in fact tell the difference, that will show up in the data as bias toward the correct identification of X.

Those who think they have "golden ears" and convince themselves they can hear something they in fact cannot are called out by this test procedure, by their failure to actually identify X better than a random guess.

NPR botched the protocol in other ways. The samples have to be sync'd, and the tester to be able to switch between them at will -- with a small crossfade to prevent the glitch from throwing you off.

An old buddy of mine, Dan Dugan, hung two pink sheets at an Audio Engineering Society conference on opposite walls to illustrate this point, asking people, "Are they the same color?" It would be trivial to distinguish them side by side, but accuracy is greatly reduced when they are distant in time or space. Human perception is all about edges, not about absolute measurements.

I don't agree. What you propose would be the test "are you able to tell the difference between the compressed and the uncompressed sound when you compare them directly"? While this test is "are you able to appreciate the difference between a compressed and an uncompressed piece of music"? If the two seconds it takes you to switch from one version to the other are enough to prevent you from spotting the difference, I'd say that whatever difference there is is likely to be irrelevant to the listening experience.
I used a similar test when I was picking an MP3 encoder to use when ripping my CDs for use on a portable device in the early '00s, back when size mattered. Mine wasn't quite "A X B", though. I wonder if that makes a difference?

Mine was "A X not-X B". I had a command line program that I could run and give two files, the reference file (typically a straight uncompressed rip from the CD) and a test file, typically an MP3 made from the reference file.

My program would then conduct a run of N trials. For each trial, it would assign the reference file as file A, the test file as file B, and it would randomly assign either the reference or test as file X. There were buttons I could press to listen to file A, file B, file X, or the file that was not file X. When I thought I had figured out whether X was A or B, I could signal my choice, and the next run would start. At the end, it would tell me how many I had gotten right.

To make sure I wasn't picking up clues from any slight delay from increased overhead in starting playing an MP3 file compared to starting a wave file, the program tossed in a random delay before executing a play command.

If I were doing this again, I think I would also toss in some minor randomization of the volume level on playback.

Anyway, the results were about what I expected. I do not claim to be particular astute at subtle audio listening, and found that the LAME encoder at 128 kbps was fine.

I was able to easily hear problems with BladeEnc at that bitrate. For instance, the opening piano chords of Cat Steven's "Morning Has Broken" just completely disintegrated with BladeEnc. I could also hear a problem with the trumpet on Dire Straits "Your Latest Trick". It's hard to describe, but if I were to compare it to being hit in the face with a baseball bat, the original and the LAME MP3 were like getting hit with a solid bat, and the BladeEnc MP3 was like getting hit with a hollowed out bat...the shell was there, but it couldn't do much damage.

When I say I could hear these problems, I mean that I could pick out the MP3 with BladeEnc every time in repeated runs of 20 trials when I was testing with those two tracks.

This test is flawed as kazinator alluded to. It is just try to find the lowest quality one and pick one of the remaining two as choice. So even with that, chance plays too much of a factor. A true test needs to be double-blind and I would wager that very very few could reliably tell a difference between wav and modern lossy codecs at 192kbit/s or higher. And I know MANY can't even tell anymore, in proper double-blind testing, at 128 or even 96kbit/s using todays better codecs of aac or ogg (mp3 improvement has stalled and currently aac as well as ogg or opus are much better).

In fact, I was JUST reading about this sort of stuff last night when I decided I didn't think I have a need to keep my archived lossless flac almbums anymore. I carefully ripped mine and many borrowed CD's to flac using EAC like 6 years ago with intention that I could get rid of the cd's themselves to Goodwill. I then just converted them to 128kbit/s mp3 for portable/phone listening. Knowing at the time 128 mp3 had its limitations I would just re-convert in the future as the writing was on the wall we were getting close to near perfect compression in regards to the limits of human hearing.

I re-converted several of my favorite albums to Nero aac a couple years ago as I could not distinguish at the time with enough certainty Nero aac (@ 192kbit/s) vs flac using good headphones and my computer. Nero has also stopped development around that time but aac has continued to advance through Apple and Fraunhofer to where we are today.

So last night I ripped all my archived albums to 160kbs iTunes aac and just deleted the archives themselves from my hard drive saving huge chunk of GB's. One, today's codecs are basically transparent at that level for 99.9% of us and two, if I ever do want a full lossless copy of an album in the future for any reason, my high speed internet connection can get them in about 1-2 minutes each (since I purchased them once already I legally owned a copy? Technically though I don't have the hard version anymore).

Then went about enjoying some music for an hour that I had forgotten about.

Some links. More on aac: http://en.wikipedia.org/wiki/Advanced_Audio_Coding

more on Opus: http://en.wikipedia.org/wiki/Opus_%28audio_format%29

Here is more info on ABx testing and codecs as well as some results of scientific listening tests done by the audiophille community.

http://en.wikipedia.org/wiki/Codec_listening_test

Particularly the most recent test in 2014 where Opus is basically transparent to very discerning eats at a surprisingly low 96kbs bitrate. Apple aac also at 96kbs as well as Ogg and latest LAME mp3(needed higher around 136kbs) are not far behind. We are just about at end-game for lossy formats.

http://listening-test.coresv.net/results.htm

4/6. I could definitely tell which was the 128/320/WAV for the 4 I got right, but on the Katy Perry and Neil Young, all 3 were indistinguishable to me.

But still, this is when comparing small sections of songs directly to one another, and being explicitly told there are quality differences. The 128 didn't sound bad, there's just some difference in transients and high frequency content (the hi-hats on the Jay Z track were an immediate giveaway for me).

Maybe a better test would be only having one audio track per song (instead of 3) and having to choose if it is 128/320/WAV. I wonder if anyone could distinguish the difference there reliably.

I've gotten a new phone 3 times in the past year (I'm clumsy...), and each time, I go a month or two with Spotify set to the default quality instead of "Extreme Quality", and I never notice until I listen to an album I know extremely well. WAV is great, but mp3 is still pretty good, even at 128kbps.

I consistently picked 320 over WAV, fwiw. I could definitely hear the difference but always preferred the 320.
Monty from Xiph/Mozilla wrote a great article about why 24-bit/192kHz format (which is what Neil Young's Pono delivers) makes no sense:

http://xiph.org/~xiphmont/demo/neil-young.html

Good article, but irrelevant. This is about lossless vs 320kbps mp3, not 24/192k vs 16/44.1k
Well, it's the next frontier in digital audio for those who claim to have golden ears.
Tidal has a similar test[1] (though only between 320 and lossless - no 128 track). Now, it's trying to sell you their high-quality streaming option, so I would take the results with a bit of a grain of salt as there are a number of ways their bias could have affected the results (for example, they could have cherry-picked tracks that didn't encode to mp3 as nicely). However, the format is much better than this one: the two play in sync, and you can seamlessly switch between A&B as it plays. I found it much easier to find subtle differences between the tracks that way over having to restart the track each time.

But more importantly, that test format preloads the tracks. In this one, some of the lossless tracks took several seconds to load for me, which completely killed its double-blindness.

[1] http://test.tidalhifi.com/

I think this experiment is slightly flawed, because what compression will do depends on the original recording. For me, the difference was most apparent in Susanne Vega and Katy Perry. Mozart and Neil Young were a complete shot in the dark, I could hear a slight difference with Coldplay.

I suspect this very much depends on how the analog recording was digitzed in the first place, if there was an analog recording to begin with. A sample from a CD is not the same as one made from a vinyl or a master tape.

Bottom line - there definitely is a difference, but in some cases it's hard to tell.

First time around only got 1 out of 6 but all the others I chose were the 320 wav using cheap over the ear headphones straight out of my desktop. The second time I got 3 out of 6 correct using the same headphones but using a cheap DAC and chose the 320 wav files for the other 3. I would love to do the test with my higher quality DAC and my higher quality over the ear headphones but I do not have access to them at this time. I rip all my CD's to FLAC at home and then convert to 192kH MP3 to load on to my phone. The only time I listen to the FLAC is when I use my higher quality DAC and higher quality headphones. I have bought 3 albums from high definition websites and on only one (Van Morrison Astral Weeks) can I tell the difference between the high definition version and the CD version in blind testing. Given this I have not purchased more high definition CD's.