Mega Lossy v Lossless Test

Which menu contains AAC lossy encoded music?

  • Menu 1 contains AAC music

    Votes: 8 47.1%

  • Menu 2 contains AAC music

    Votes: 3 17.6%

  • Too close to call

    Votes: 6 35.3%
  • Total voters
    17
Cav said:
Well if it is valid, then most people (by a big margin) cannot tell lossless from kosher. Personally, I have no problem with that.
Click to expand...

That is probably the case, as other tests support the results found here.

We can all find plenty of reasons to explain away the findings of blinds tests such as this and all of the reasons have some validity. The big question, as I keep on saying, is have you developed a test that is better than the default alternative. The default alternative is a sighted demo. I'd argue the method used here is better by a good margin, though not perfect.
 
RobHolt said:
The big question, as I keep on saying, is have you developed a test that is better than the default alternative. The default alternative is a sighted demo. I'd argue the method used here is better by a good margin, though not perfect.
Click to expand...

This comparison is very good IMO, but it needs to be viewed in the context that it is assessing a combination of things: the listener, their monitoring equipment, and whatever method they wish to go about the test (fast A B switches, listening to whole tracks, how many tracks used, how many repeat listens etc). As such it has next to no scientific value IMO as it's simply not a level playing field, but it has a lot of use in that the listener gets to assess the compression technology without any expectation bias in their own chosen way and to draw whatever conclusions they wish from the experience, i.e. the audition process itself is of far more worth / value than the poll results it generates. It was a good thing.

Tony.
 
Agreed, it removes some of the usual barriers and that is realistically all that we can hope for.

On the the equipment issue, I think it can be safely discounted as a factor.
Each listener, had they been doing a standard sighted test at home, will have used the equipment to hand. They've done the same here. In fact you can argue that with each listener using familiar equipment they have fewer variables than if trying to asses through unfamiliar systems. I'd argue the same for the needledrops and cable tests, etc.

I certainly wouldn't call it a scientific test, but then none are. However many tests are very unreliable.

If the software allowed it would have been useful to hide the rolling poll result as that for me was the biggest negative, but still a lot better than openly saying 'here is a 256k compressed and here is a lossless one - which sounds best?'
That would produce only one very strong response.
 
RobHolt said:
On the the equipment issue, I think it can be safely discounted as a factor.
Click to expand...

I don't agree. As an example I listened to these files using a pair of Sennheiser HD-600 headphones, a true monitor grade pair of cans that are not a million miles away from the current state of the art. If I only had something like a pair of white iPod earbuds, or worse, say typical computer speakers knocking about I'd not have stood the slightest chance on this test. To hear differences you clearly need to be using kit capable of resolving them!

Tony.
 
TonyL said:
I don't agree. As an example I listened to these files using a pair of Sennheiser HD-600 headphones, a true monitor grade pair of cans that are not a million miles away from the current state of the art. If I only had something like a pair of white iPod earbuds, or worse, say typical computer speakers knocking about I'd not have stood the slightest chance on this test. To hear differences you clearly need to be using kit capable of resolving them!

Tony.
Click to expand...

True but the test is on a hi-fi forum and reasonable replay equipment should be a given.
If someone uses a pair of plastic PC speakers they aren't taking it very seriously.

If this were the iPod Monthly forum I'd place more weight on the equipment argument.

Having said all of that, a few did just use modest headphones from a laptop and heard differences, so the question of how good does the kit need to be is far from simple IMO.

FWIW I think that some differences are more readily heard even on cheap headphones compared to even high end audio systems. The phones really focus the ears on the tiniest of details, stuff that gets lost in a listening room over speakers. Listen to something like a decaying piano note over speakers at a couple of metres distance and time the period until you no longer hear the note. Repeat on even modest headphones and you'll get a longer time because the phones allow you to hear down closer to the noise floor. In a test such as this where such low level detail really matters, you can argue that modest phones are certainly no barrier.
If we were measuring something else - scale, dynamics, bandwidth for example then we have a different argument entirely.
 
I think this test, and the results, do suggest one particular course of action for further investigation ....

If the results suggest that 83% of people hear one thing, ... and 17% of people hear something different ......

Then it is the 17% that requires further investigation, ..... not the 83%


:confused:


JC
 
Late to the party on this one. I'm fairly confident I would have been able to pick correctly as I have done so pretty consistently in the past (in fact I was able to correctly assign four different samples to their codec/bitrate blind, publicly).

That is not to say that the differences are substantial. What this test has suggested--all that is has suggested--is that any differences between AAC and 256 are subtle at best.

The fun part is that it shows that mp3-phobia is baseless.

But lets not go too far. From this thread it seems a number voted based on computer speakers or cheap headphones. Those votes are useless--they poison the pool. All the claims about psy models for data compression are based on high fidelity playback and good hearing. Without those, results are unreliable.

As far as what I listen for, I focus entirely on transients. Drums, cymbals, surface noise (that's how I picked out the codecs in the above test on a Massive Attack track). I don't know if that's just because I'm very sensitive to the sounds of transients in hi-fi reproduction in general, or because transients are the most complex waveforms, or because the design of the codecs did not place much emphasis on dynamics--most of the test tracks in the Fraunhofer development didn't even have drums.
 
Not sure what your point is: that the test was unreliable; that the results (those that you disagree with) were invalid; that "mp3-phobia" is not a problem; that lossy reporduction is a problem but only if you have golden ears and a sufficiently revealing system...?
 
This has been a really helpful way to conduct a double blind test: but with only 17 votes, we need to be very careful about drawing wider conclusions! A percentage always sounds impressive, but in such a tiny sample, it's a misleading statistic.

What we have auditioned, though - perhaps more than our ears - is our systems. When I setup my system for resolution rather than fun, there was a very, very marked difference between just 10 seconds of one track: no need to listen further.

But all the tell-tales of top-and-tail truncation, hash-and-splash and soundstage collapse that are a dead giveaway of compression could easily have been lost in a bad room, or via unresolving speakers.

Any basic pro audio setup makes mincemeat of such comparisons: the trouble is that much hi-fi gear is designed to flatter rather than reveal. That's a pretty way of saying: 'it lies'.

For heaven's sake let's not encourage record companies to continue mastering CDs as if no-one cared about the difference: many off-the-sheld RedBooks sound like MP3 as it is: we should be sending the message that we all care ENORMOUSLY about the acoustic and decay and fine-grained resolution of our favourite music!
 
Last edited by a moderator:
hubsand said:
But all the tell-tales of top-and-tail truncation, hash-and-splash and soundstage collapse that are a dead giveaway of compression could easily have been lost in a bad room, or via unresolving speakers.
Click to expand...

Most people, including, I suspect many on hi-fi forums, listen to music in real world environments, through real world systems. In that case lossy formats are not that apparent.
 
hubsand said:
This has been a really helpful way to conduct a double blind test: but with only 17 votes, we need to be very careful about drawing wider conclusions! A percentage always sounds impressive, but in such a tiny sample, it's a misleading statistic.
Click to expand...

46 votes - we had this running here and on PFM.
 
Guilty: I only picked up on it today. I'm not saying I have the most golden ears, or the most resolute system, but compression has a distinct sound that's not hard to identify.

By way of making this thread educational, perhaps we should compile a 'listener's guide' to the tell-tales of digital compression, then repeat the test. I bet many of those who failed to identify the AAC would then be able to do so.

Although many 'real world systems' aren't capable of making the distinction, it's not a financial issue: a cheaply treated room, a £200 DAC and a £350 pair of monitors would be good enough. Or even some decent headphones.
 
hubsand said:
By way of making this thread educational, perhaps we should compile a 'listener's guide' to the tell-tales of digital compression, then repeat the test. I bet many of those who failed to identify the AAC would then be able to do so.
Click to expand...

I've been banned for a month from one forum for criticising just such an approach. What do you want to do - tell people how to spot the difference and then repeat the test as often as it takes to get the "correct" result?

What difference does it make if people can or cannot identify one from another?
 
Posts made after the result should make clear that we are doing exactly as suggested above.

Tony L has posted what he feels are the key indicators for a lossy file.

We certainly shouldn't continue repeating the test, but we will do a small follow-up test and see if the result differs markedly from this one.
 
Why?!

Cav said:
What difference does it make if people can or cannot identify one from another?
Click to expand...

Because it's not a subjective matter: one is better.
Because that's what learning is: training our faculties to discern differences.
Because learning is intrinsically a good thing.

Because many listeners don't use revealing systems, record companies have become negligent about the quality of our recordings, which are often blatantly and cynically degraded from the studio masters.

And because promoting the patent untruth that the difference between compressed and lossy recordings is negligible, or doesn't matter, literally damages music.

It's bad science.

It implies we shouldn't care about recording quality.

It fosters a climate of indifference that has delayed the release of 24-bit recordings for decades.

It makes people buy more cruddy MP3 players.

It sets people down a path of compromise: why not compress the files? Why not play them back on an AV amp, or a phone, or through a sock? They're all convenient, too . . . that way lies musically unsatisfying experiences.

It's a pretty dumb question, ultimately: storage is now SO cheap, why compress the files anyway? Does anyone here believe that compression - in theory, or in practice, will IMPROVE quality? It should sound worse; it does sound worse: the question is: do we care?
 
Cav said:
What do you want to do - tell people how to spot the difference and then repeat the test as often as it takes to get the "correct" result?
Click to expand...

Yes: sounds like a great idea. Anyone here opposed to learning?
 
RobHolt said:
Posts made after the result should make clear that we are doing exactly as suggested above.

Tony L has posted what he feels are the key indicators for a lossy file.

We certainly shouldn't continue repeating the test, but we will do a small follow-up test and see if the result differs markedly from this one.
Click to expand...

A follow-up would be interesting.

If participants were given listening guidelines (if they're unsure how to tell the difference), it wouldn't diminish the validity of blind testing: in fact, it would make the results more robust.

You wouldn't poll a team of accountants for fashion advice: the better informed the participants, the more meaningful the outcome.
 

Latest posts

Back
Top