Mega Lossy v Lossless Test

[TonyL]

...and I look forward to Tony's "How To Spot Lossy Files 101"

I'll do my best...

My suspicion is that some folk go into these tests a little confused by the terminology and may assume 'compression' means 'dynamic compression', i.e. that the word carries the same meaning as it does in a recording studio or when discussing the compression used on radio or TV (e.g. the BBC proms where a solo flute is nearly as loud as an orchestra at full tilt). This is not the case at all, and looking for the wrong thing may in this case help the right thing escape detection!

With MP3s, AACs etc it is the file size that is being compressed, not the dynamic range. The loud bits are just as loud, the quiet bits just as quiet, the thing that is missing is certain detail that has been thrown away in order to make a smaller file. It works just like JPEG compression does with pictures, and there are certainly parallels in what is lost. Lossy compression loses some low-level detail, and if you are aware of what kind of low-level detail it loses you can actively look for it. To my ears it mainly screws up reverb / depth / space and becomes confused when the going gets tough, i.e. it becomes harder to identify instruments or recording acoustic in busy or noisy passages.

In this test I used the short versions of three tracks (I did not bother to download the rest):

The Jimmy Smith. This was the first one I went for as I wanted to hear it (I like Jimmy Smith and was lucky enough to see him live), and it proved the hardest. I really struggled with this one to be honest. I felt there was a little more acoustic space around the applause, the cymbals just a little better defined, a little more sense of things occurring in a plausible acoustic, but it was the best preserved of the tracks I listened to by far, and based upon this track alone I'm not sure I'd have committed and may well have hit the 'don't know' option.

The Jane Monheit. The vocal reverb gave a coherent sense of three dimensional space on the wav and sounded flatter, crunched, and more like a tacky FX unit on the AAC. The piano also sounded more real / better recorded / and in a more believable acoustic space on the wav. I was certain of my choice by now and I'd have happily committed to option 2 without listening to anything else.

RATM. Lossy compression is always pretty crap with heavily distorted guitars, it just loses that 'thing' that makes a well mic'd overdriven Marshall stack sound so much better than a 50 quid distortion pedal. It's also a lot easier to hear the vocal quality and kit metalwork. This was the easiest IMO, the wav just sounds a lot better, cleaner, better separated, more real, the AAC kind of blurs it altogether into one homogenised sound.

That's it! No magic, no golden ears! Just zone in on what the technology buggers up and you'll spot it assuming the replay equipment is up to it. I used a pair of Sennheiser HD-600 headphones plugged into my MacBook and played the files in Audacity. I just A B'd between the files a couple of times concentrating on the specific aspects detailed above.

Tony.

 

[jcbrum]

My point to JC is that the more complicated the test, the more downloading and listener segmentation that is required, the fewer take part.

Based on the statistics you have published, only three people 'got it right' .....

.... each of those three people had a 50/50 chance of selecting the correct answer.

Also, based on your earlier remarks, 'a handful' of people claim to be able to get it right with a high degree of certainty ...

Let us assume there are six people who claim this definite skill. Each of those has a 50/50 chance of getting it right by pure chance .......

.......... and that is exactly the result you got ! ... three of them succeeded

That means those six people who claim special skills are no better than a larger group, who have now been conclusively shown that in reality .. they just can't tell the difference

Maybe you know who the three people with 'correct' answers are ....

... If so, with their permission why not say who they are, and invite just those three to each listen to a list of twenty tracks and then pick out the one and only AAC amongst the others which are wavs.

Statistically they would only have a 5% chance of selecting it by luck, rather than a 50/50 chance, which has been the result so far.

Imo, the existing tests, and the results, show that no-one can tell the difference, ...

.... not even any of those who claim to, and none of those who you say can tell.

The results, so far, do prove that a large group can't tell the difference ... and also that a group of six who claim special skills can't either.

This is not intended as criticsm in the derogatory sense, but simply that on a mathematical analysis the tests haven't advanced the knowledge at all

JC.

 

[TonyL]

JC, I know you have an agenda to prove, but I have never got this kind of poll wrong. Ever. On very rare occasions I can't spot the compression (i.e. would go for a 'don't know' option), but I have never in my life picked the wrong file. This is not a chance thing, I'll happily do it 100 times if you pay for my time.

Tony.

 

[Labarum]

Tony, what do you listen on, and what do you listen for?

 

[TonyL]

Tony, what do you listen on, and what do you listen for?

I don't quite follow. I've highlighted the kit used (MacBook, HD-600s), and in this case I was listening for lossy compression, as that was the poll question! My usual kit is a Garrard, tubes and big Tannoys. I use that to listen for teh musics!

Tony.

PS I used to run a small recoding studio so I am very used to listening for faults, in fact it's a skill I've tried very hard to unlearn over the past 20 years or so as it can seriously get in the way of enjoyment. My aim here is not to criticise the lossy files or those who use them (which includes myself as I have an iPod), but the poll question was can I identify them, and yes, I can.

 

[Labarum]

I don't quite follow. I've highlighted the kit used (MacBook, HD-600s), and in this case I was listening for lossy compression . . .

OK. Sorry. I had missed that you had already declared the kit used.

Yes, you were listening for lossy compression, but what are the give-aways, what are the clues you listen for?

 

[jcbrum]

JC, I know you have an agenda to prove, but I have never got this wrong. Ever. On very rare occasions I can't spot the compression (i.e. would go for a 'don't know' option), but I have never in my life picked the wrong file. This is not a chance thing, I'll happily do it 100 times if you pay for my time.

Tony.

I don't have any agenda at all Tony, ... my remarks are merely an analysis of the results.

I don't think anyone needs you to do it 100 times and it's certainly not something I would spend any money on.

You don't seem to have any difficulty in predicting that you can do it with close to 100% accuracy, and so far fwiw, I believe you, but no-one has seen it actually done, because all the tests show that when statistically examined, no-one can tell

It's as simple as that really, and is not intended to be impugning your claimed skills.



JC

 

[Labarum]

Quote of the millennium:

I don't have any agenda at all

 

[TonyL]

Yes, you were listening for lossy compression, but what are the give-aways, what are the clues you listen for?

Reverb, acoustic space, the 'air' around instruments, all the ultra-low level 'round-earth' stuff as it is this which tends to be mangled or missing in action once the file has been compressed. Listen to the second two tracks I mention above. Concentrate on the vocal reverb on the Jane Monheit track, all the info needed to make a choice is in that acoustic space, then listen to the guitars, vocal and drum kit metalwork on the RATM track, listen to how separated the instruments are, try and figure out how many guitar tracks there are, listen to how the voice fairs when the guitars are really loud, what happens to the cymbals at this point etc etc. Assuming you have a decent pair of headphones or whatever it should be pretty obvious which file has been tampered with IMO.

Tony.

 

[jcbrum]

.. it should be pretty obvious which file has been tampered with ..
Tony.

But it isn't ...... is it ?

The results say that the overwhelming majority say it isn't ..... 84% in fact .....

It's not an agenda ............ it's just arithmetic

JC

 

[Labarum]

Reverb, acoustic space, the 'air' around instruments

When I think I can tell the difference, it is the area of ambience. I can't hear it on my headphones (but I'm not a great headphone listener), but I think I can on my big speakers - but they are in Nicosia and I am not.

I want to listen to Rob's files again when I get back, but now I know which is which.

When D-Dur was transmitting in FLAC I could flick between the FLAC stream and the OGG 256 stream. I thought the FLAC caught the acoustics of the recording space where the OGG did not - but they were so close I could not reliably tell them apart.

 

[RobHolt]

This is not intended as criticsm in the derogatory sense, but simply that on a mathematical analysis the tests haven't advanced the knowledge at all

JC.

What knowledge?

Pretty much the default view on forums is that people can easily detect lossy files at this bitrate. In fact hi-fi buffs will claim the ability to detect all sorts, including far smaller differences than seen here, and things like and hi-res from standard red book.

This has to be taken one step at a time. All a test such as this will show is most people in the above category cannot reliably detect files at this level of compression, when they believe they can.

A few can do it and with 100% reliability. Now I've witnessed a couple of them do this on multiple tests and not just this one.
So, step two - advancing the knowledge - is to determine what criteria they are using and discuss it. We will then post suitable example files and then ask people to listen armed with the additional knowledge. if the vote numbers move quite obviously in the other direction we are onto something.

 

[TonyL]

But it isn't ...... is it ?

The results say that the overwhelming majority say it isn't ..... 84% in fact .....

It's not an agenda ............ it's just arithmetic

The test itself is non-scientific. For a start the monitoring kit is a complete unknown. IIRC you (or Ash, I can't remember) have stated that even 128kbs files can't be detected using the replay kit you favour, people using computer speakers or cheap earbuds will be even more disadvantaged. I also suspect some folk didn't know what to listen for and were desperately trying to detect dynamic compression, which is perfectly understandable given that the terminology used is so ambiguous. This is not a meaningful assessment of anything really, certainly not something one could apply arithmetic to.

Tony.

 

[Paul Ranson]

The files sound different, which sounds better in this type of audition is more arbitrary. Sometimes simpler appeals...

I had one of Rob's samples on CD so I've had a listen to the difference, what the AAC left out. It's quite interesting.

Paul

 

[jcbrum]

What knowledge?

The knowledge that people generally can't tell the difference.

Pretty much the default view on forums is that people can easily detect lossy files at this bitrate.

I know they think that, ..... but when you put them to the test - they can't. Your series of tests so far have shown that.

In fact hi-fi buffs will claim the ability to detect all sorts, including far smaller differences than seen here, and things like and hi-res from standard red book.

I am aware that they make those claims, but generally they are merely assertions, and when scientifically tested, - they fail.

This has to be taken one step at a time. All a test such as this will show is most people in the above category cannot reliably detect files at this level of compression, when they believe they can.

I agree, .... I'm simply saying we already knew that, despite assertions to the contrary, so therefore whilst it is useful verification, it doesn't yet provide any advance of that knowledge.

A few can do it and with 100% reliability. Now I've witnessed a couple of them do this on multiple tests and not just this one.

I haven't seen any individuals results published, and anyway it's only two people seemingly, who make a choice based on a 50/50 option. Mathematically that's little more than the same results based on random luck.

So, step two - advancing the knowledge - is to determine what criteria they are using and discuss it. We will then post suitable example files and then ask people to listen armed with the additional knowledge. if the vote numbers move quite obviously in the other direction we are onto something.

Perhaps, ........ but first, in order to test whether their advice was likely to be useful, I was considering testing them, with a statistically more meaningful test, to see whether their advice was of any real value.

If it turned out not to be, it would save everyone a lot of time and trouble in following up what might be a wild goose chase.

I was only trying to suggest that careful analysis of the results is best, so as to avoid coming to false or unreliable conclusions.

JC

 

[jcbrum]

This is not a meaningful assessment of anything really, certainly not something one could apply arithmetic to.

Tony.

So you're saying it's not a meaningful assessment of your ability to tell the difference, or not, then ? .......

and you're saying there's no point in counting the votes and using that to determine whether anyone can tell the difference then, are you ?

Just exactly what then, is the point of this test ?


JC.

 

[RobHolt]

Perhaps, ........ but first, in order to test whether their advice was likely to be useful, I was considering testing them, with a statistically more meaningful test, to see whether their advice was of any real value.

If it turned out not to be, it would save everyone a lot of time and trouble in following up what might be a wild goose chase.

I was only trying to suggest that careful analysis of the results is best, so as to avoid coming to false or unreliable conclusions.

JC

Then conduct your test by putting up the required files and asking those certain of identifying the differences to participate in something more elaborate.

I've already seen sufficient evidence (by observing this and other tests) that they can tell the difference reliably. I'm not the one that needs convincing though.

 

[TonyL]

So you're saying it's not a meaningful assessment of your ability to tell the difference, or not, then ? .......

No, it's fine for that. I have a good idea what I'm listening for, I'm using a good pair of £300 headphones that are more than capable of highlighting the differences, and as such I get it right every time. This aspect of the poll is irrelevant to me anyway, I know I can do it using this level of kit and I have no desire or need to prove anything to anyone. I am far more interested in the other results and what other people are looking for, hearing and preferring.

and you're saying there's no point in counting the votes and using that to determine whether anyone can tell the difference then, are you ?

I'm just saying that the test conditions are not sufficiently rigorous to produce a meaningful result, not least of which due to the no doubt huge variability in monitoring equipment. If people are trying to do this through computer speakers, cheap earbuds etc it is not a level playing field and the poll becomes as much an assessment of their replay kit as anything else.

Paul also makes a very good point indeed - if we try viewing the poll results as other than 'right', 'wrong' and 'dunno', and we take the first two as a firm preference / rejection of a particular file-set / sound we have a whole new, and probably more interesting thing to think about, i.e. do some people actively prefer digital compression? I'd have liked the test to have had a 'sureness' parameter, i.e. how confident the listener was in their choice, thus indicating a preference.

Tony.

 

[jcbrum]

Then conduct your test by putting up the required files and asking those certain of identifying the differences to participate in something more elaborate.

......... they can tell the difference reliably .........

Hmmmm, ....... whom should I ask ? ....... other than Tony ? .........


Is any one else prepared to reply ..... and say that they can assure us that they can tell the difference every time ............... ??

 

[jcbrum]

....... I'm using a good pair of £300 headphones that are more than capable of highlighting the differences, and as such I get it right every time.

I know I can do it using this level of kit ........

... I've highlighted the kit used (MacBook, HD-600s) .....

So you are saying that the built in DAC in a Macbook is easily good enough, to be able to tell the difference every time.

I agree with you, ... but I think any good DAC will do the job as well.

..... I'm just saying that the test conditions are not sufficiently rigorous to produce a meaningful result ...

Yet you, and Rob, say the result is meaningful ....... and shows that you and two others can get it right every time ..... ???

I think the results are plain, ... and available, ...... and mathematically correct, .....

........ it's how to interpret them appropriately that is important.

.... Paul also makes a very good point indeed - if we try viewing the poll results as other than 'right', 'wrong' and 'dunno', and we take the first two as a firm preference / rejection of a particular file-set / sound we have a whole new, and probably more interesting thing to think about ......

It was actually me that made that point, ..... in post #66, when replying to jonesi

.. as I get it wrong every time ...

It isn't really 'getting it wrong' ..... you are presumably, merely expressing a preference.

Paul's comment came later, but he is in agreement.

However the question under discussion is not 'which do you prefer', but is actually 'can you tell the difference and correctly identify them' ...

That is what I am examining ........ 'preference' is an entirely different question.



So, .. I agree, .... we do need a new test .....

Here is the challenge ...........



Who else then, .... is certain that they can get it right every time .... ??

Any takers ?? ............

JC