Hi,
Rob, sometimes we have face the facts. In this case the facts are that what is practical is not appropriate.
Of course, that does not mean you cannot do your blind tests, however you cannot claim they are superior in any way to a sighted test, because, scientifically speaking, they are not.
Yes they are. If the only thing to have changed is that the identity of the products has been removed, you have a better test by default. You have removed a huge and very important element of potential bias.
From my experiences with blind testing you have just set out out a protocol and approach that will maximise expectation bias and test situation stress.
May I ask, has your blind test EVER returned a positive?
Yes it has, with cables in fact and DNM in particular.
I have also given you examples previously where positives have been obtained. The fact that you disagree with ideas of those conducting the tests is a different question.
The idea that blind tests never deliver a positive result is plain wrong. Strawman argument. You have to consider that what we regard and obvious, provable differences are rarely subjected to blind tests, because we see no need. We tend to use such tests for identifying things which are relatively subtle at best. It is therefore no surprise that we more nulls than positives. In many instances there is simply nothing of significance to discover.
Start with a fully blind test designed to minimise expectation bias. So, make sure participants do NOT know what you are going to test, so they do not entertain preconceived notions, but listen with open ears and minds. Secondly, arrange things relaxed, so stress is reduced, short test sequences, long breaks... Also, use a more informative system that "same/different", but base it on preference.
The nature of the test depends on what you are actually trying to test.
In the one described, we are seeking to determine the truth of both pre determined opinions and those formed via a sighted evaluation.
Following such a session with a repeat blind session and obtaining different responses, must, by its very nature, throw the original opinions into serious doubt.
Comparing the results between this blind & sighted and other blind options can be quite interesting. One thing you remove is the Now prove to me what you claim to hear" part, which invariably sends stress levels up and acuity down in most participants.
I would dispute that. It can equally be argued that focusing the mind actually
heightens acuity. The whole stress issue is overplayed and is in my opinion a feeble attempt to debunk the merits of blind testing. There is significant research into concentration and the periods to which this can be relied upon. Between 30 and 45 minutes depending on who's research you read. This can usefully be taken into account, however it remain for all types of listening test and is not unique to BT.
It doesn't put BT at any disadvantage.
My question would be if the difference "disappears" because it did not actually exist or because the test is insensitive?
Many of the things we test under blind conditions are what you might call 'accepted wisdom'. So you will often find that opinions expressed during a sighted analysis tally closely with wider opinion - reviews and other forms of 'expert' opinion.
In this situation, the differences identified are accepted as the prevailing view - so they exist even if the basis for their existance is decidedly shaky.
If these things vanish when you convert to blind conditions, you must as the very least seriously question the accepted wisdom.
Test sensitivity needs to reflect the purpose.
A loudspeaker listening test requires significantly less consideration of sensitivity than say, evaluation of coupling capacitors.
We need to use the brains we posses to adjust the complexity and sensitivity of the test (inc possibly the participants) to determine what is appropriate.
No, they are not necessarily useless, for example to a single individual. Neither small scale biased DB Tests useless. However neither kind of test has any statistical power, in other words any attempts to generalise from these individual tests (which is normally the object) must fail or be met with ridicule.
We seem to be agreeing.
What do you think I am proposing?
I'm not proposing that anything I can do will be good enough to satisfy the highest statistical standards, or that anything I might 'discover' is going to be industry changing. I don't have the time or resources (or inclination) to go there.
But I and others can help to focus the mind of the individual or small group that they should never rely wholly on sighted assessment, their own or that of others.
I will add one small rider to the above.
Simple and small group blind testing of the more silly and implausible audiophile beliefs that do the rounds can carry significant weight and you can extend the scope of the findings.
In a recent discussion I had in another place, a poster claimed quite categorically that placing 2p coins between the bottom of his Dac and the supporting spikes, made a
huge difference. In such a situation, the only way to debunk this (and there is nothing wrong with debunking when the situation demands....) is to ask the listener to identify this huge difference blind.
Now, you will say that I cannot possibly then extend this to a comment that such an action
never makes a difference, and you would be correct. But again we have to use our intelligence and say that armed with the blind test experience (and we can repeat using more listeners)
and the weight of scientific knowledge in the field, we can justifiably claim that our test shows the original reaction to be false.
If 'false' is too strong for you, substitute 'highly unlikely'.
