Like what you have done with the place...

Hi Thorsten, hope you and yours are fine.

On (what's become the) topic:

The other beef I have with DBT is that the treatment of test results is generally less than rigorous. For example, how do you define whether a difference is audible or not? In my book, if one single test participant picks out a difference with statisticical reliability, that difference is proven to be audible. Whether or not the other 9,999 participants are able to pick it out is irrelevant.

The tables are turned when the question asked is whether a difference is important to a general audience. In that case, I'd want the difference to be picked up by a significant percentage of participants.

But the question of DBT and audio has been argued ad nauseam before (one of the better instances is this one).

 

Hi Nando,

I think we have two different issues.

One is what anyone does, personally, in and for their own use.

In this case "I like B better than A and so I am buying B" is all that is needed.

However if we wish to have a debate in the scientific realm, or about principles in audio, many will not accept such a result to hold any relevance insofar as more global conclusions (such as "B is better than A") are concerned and will demand quantifiable metrics to show that B is better than A.

It is here where we run into trouble.

First, existing metrics for Audio show poor corellation with preference or what people consider "good sound".

Second, the "meter-readers" nevertheless insist that something that performs in these metrics is "bad", even if it is preferred by the buyer.

Third, some of this group will very vocally argue the "good sound" is actually because "they like distortion etc.", to explain why something that performs poorer in their metrics is taken to sound better (the total fallacy of this argument should so obvious, it needs not explaining).

Fourth, eventually someone will note that Customers are being sold equipment that performs poorly in standard tests, often at higher prices than equipment that performs better and often as "better sounding" and will conclude that fraud is being perpetrated and go into "consumer protection mode" and insist on scientific proof for such statements...

Fifth, the people in 4 will decide to make "scientific tests", ignore in the process that tests that are practical are sorely lacking in statistical power and then use the results of such fundamentally flawed tests to start a nice little witch-hunt.

Note, the tests in 5 where never intended to actually be genuinely scientific, or to provide real answers to real questions, but instead where always intended as tool of self professed debunkers and used to add a false veneer of science to what the core was plain and simple the same that they accused the ones they opposed of, namely plain and simple fraud.

Decades lathe the likes of me and Robholt are still debating the whole subject as there never was any comprehensive attempt to discredit these debunkers and their results are still making their rounds, nowadays distributed appropriatly on the "Web of a million lies".

Sadly some of quite sensible modern people seem to entertain the mistaken belief that because some test claims to use scientific methodology it actually does do so and that it was carried out with due attention to detail and due diligence and that care was taken to maximise the tests sensitivity, when in fact the opposite is true.

Furthermore some apparently believe that a blind tests that has no statistical power and whose results can only be considered meaningless is somehow better than a sighted that has no statistical power and whose results can only be considered meaningless (except possibly to participating individuals), something that from a scientific and statistical viewpoint is simply not the case.

Ciao T

 

Markus,

Doing fine. Now (almost) divorced and with another gal though, we have a 1 1/2 year old Baby Daughter.

On (what's become the) topic:

Yes, the opening post in the linked thread very well sumarises my views. Both on the medical and the audio side.

Ciao T

 

No problem with that.

But you still have to face the reality of what is practical and appropriate to the task.

It it exactly why I will always combine a sighted and unsighted test.
For me, I'm interested in eliminating the effects of sighted bias and accept that I can only improve testing by doing so, not remove all doubt.
It simply takes sighted testing, where a difference has been identified, one stage further by saying, 'now produce the same result unsighted'.

The problem with your argument is that many differences that fail to show on a blind test are not claimed to be subtle at all. You will routinely see some quite wild claims made for equipment where the science indicates this to be nonsense.
These effects invariably get identified on sighted testing, yet disappear on even the most basic unsighted test. You must ask why, and to dismiss the result simply because every conceivable variable (which also exist in the sighted test) wren't addressed is quite wrong IMO.

If we follow and pursue your and Thorsten's requirements to the logical conclusion, all sighted tests are utterly useless, since known, strong forms of bias and deliberately left in place.

If we are truly seeking to identify very subtle effects, then yes I would support the call for far more advanced and controlled blind testing, with larger samples.
But I'd also argue that if such refined testing is required in order to identify such differences, they probably don't matter!

Different argument entirely, though the two often get conflated.

I know very few 'meter readers' who claim that because something fails a test it will necessarily sound 'bad'.
Blind testing is all about identifying and confirming differences - start making a connection to listener preference and taste and you end up with yet more complication.

Listener assessment of good, bad or indifferent is entirely subjective, and it has nothing to do with developing useful blind testing.

Exactly the point I made earlier.

There are two discussions to be had:

1 - improving basic home listening tests to eliminate several strong known forms of bias, thus making them better though not perfect by any means. These tests are particularly useful for debunking some of the more extreme claims in audio. This does not require overly complex methods or large samples and the result isn't 'useless' - simply because the disparity between the sighted/unsighted results will cause the listener to question what he hears. This can only be a good thing.

2 - Extensive research in order to establish a result beyond any reasonable doubt, perhaps applicable as a general rule or finding across the industry.
Much time and effort need to be expended on such testing, and yes every conceivable variable needs to be identified.

Various levels exist between the two.

All are superior to any sighted test.

 

sorry i forgot, i used to be sound engineer for a very well known label, i know nothing

 

Hi,

Rob, sometimes we have face the facts. In this case the facts are that what is practical is not appropriate.

Of course, that does not mean you cannot do your blind tests, however you cannot claim they are superior in any way to a sighted test, because, scientifically speaking, they are not.

From my experiences with blind testing you have just set out out a protocol and approach that will maximise expectation bias and test situation stress.

May I ask, has your blind test EVER returned a positive?

I would suggest you try the reverse procedure.

Start with a fully blind test designed to minimise expectation bias. So, make sure participants do NOT know what you are going to test, so they do not entertain preconceived notions, but listen with open ears and minds. Secondly, arrange things relaxed, so stress is reduced, short test sequences, long breaks... Also, use a more informative system that "same/different", but base it on preference.

After completing the blind part, then move on to sighted "hands on" testing, preferably WITHOUT telling the participants their "score" from the blind test.

Comparing the results between this blind & sighted and other blind options can be quite interesting. One thing you remove is the Now prove to me what you claim to hear" part, which invariably sends stress levels up and acuity down in most participants.

You replace it "Lets listen, write down what we hear and then I'll tell you what you where listening to.".

I should add that I never do blinds tests in another way and have found over time that such truely blind tests even with small numbers of participants can be very revealing of small differences, more-so than the classic ABX variety tests, especially if the test items are known and prior sighted tests where performed, allowing people to develop opinions and attachments.

You are in debunker mode. You are already drawing conclusions prior to any experiment. You are already in effect starting your experiment, not to establish the truth or not of such claims, instead you do it to "debunk". Coming to an experiment with such preconceived notions and agenda is not a good idea. Everyone's subconscious is quite strongly influencing our actions. You may not want to admit it, but with such a mindset you invariably make choices in your test arrangements that you believe or even know will mitigate against audibility.

My question would be if the difference "disappears" because it did not actually exist or because the test is insensitive?

No, they are not necessarily useless, for example to a single individual. Neither small scale biased DB Tests useless. However neither kind of test has any statistical power, in other words any attempts to generalise from these individual tests (which is normally the object) must fail or be met with ridicule.

Ciao T

 

Yes they are. If the only thing to have changed is that the identity of the products has been removed, you have a better test by default. You have removed a huge and very important element of potential bias.

Yes it has, with cables in fact and DNM in particular.
I have also given you examples previously where positives have been obtained. The fact that you disagree with ideas of those conducting the tests is a different question.

The idea that blind tests never deliver a positive result is plain wrong. Strawman argument. You have to consider that what we regard and obvious, provable differences are rarely subjected to blind tests, because we see no need. We tend to use such tests for identifying things which are relatively subtle at best. It is therefore no surprise that we more nulls than positives. In many instances there is simply nothing of significance to discover.

The nature of the test depends on what you are actually trying to test.
In the one described, we are seeking to determine the truth of both pre determined opinions and those formed via a sighted evaluation.
Following such a session with a repeat blind session and obtaining different responses, must, by its very nature, throw the original opinions into serious doubt.

I would dispute that. It can equally be argued that focusing the mind actually heightens acuity. The whole stress issue is overplayed and is in my opinion a feeble attempt to debunk the merits of blind testing. There is significant research into concentration and the periods to which this can be relied upon. Between 30 and 45 minutes depending on who's research you read. This can usefully be taken into account, however it remain for all types of listening test and is not unique to BT.
It doesn't put BT at any disadvantage.

Many of the things we test under blind conditions are what you might call 'accepted wisdom'. So you will often find that opinions expressed during a sighted analysis tally closely with wider opinion - reviews and other forms of 'expert' opinion.
In this situation, the differences identified are accepted as the prevailing view - so they exist even if the basis for their existance is decidedly shaky.
If these things vanish when you convert to blind conditions, you must as the very least seriously question the accepted wisdom.

Test sensitivity needs to reflect the purpose.
A loudspeaker listening test requires significantly less consideration of sensitivity than say, evaluation of coupling capacitors.
We need to use the brains we posses to adjust the complexity and sensitivity of the test (inc possibly the participants) to determine what is appropriate.

We seem to be agreeing.
What do you think I am proposing?
I'm not proposing that anything I can do will be good enough to satisfy the highest statistical standards, or that anything I might 'discover' is going to be industry changing. I don't have the time or resources (or inclination) to go there.

But I and others can help to focus the mind of the individual or small group that they should never rely wholly on sighted assessment, their own or that of others.

I will add one small rider to the above.
Simple and small group blind testing of the more silly and implausible audiophile beliefs that do the rounds can carry significant weight and you can extend the scope of the findings.
In a recent discussion I had in another place, a poster claimed quite categorically that placing 2p coins between the bottom of his Dac and the supporting spikes, made a huge difference. In such a situation, the only way to debunk this (and there is nothing wrong with debunking when the situation demands....) is to ask the listener to identify this huge difference blind.
Now, you will say that I cannot possibly then extend this to a comment that such an action never makes a difference, and you would be correct. But again we have to use our intelligence and say that armed with the blind test experience (and we can repeat using more listeners) and the weight of scientific knowledge in the field, we can justifiably claim that our test shows the original reaction to be false.
If 'false' is too strong for you, substitute 'highly unlikely'.

 

I'm struggling with T's logic that says removing one known form of bias from a test does nothing to improve the test.

Sure a blind test can introduce stress, no doubt about it, but it also removes the single biggest source of false results- knowing what the DUT is by sight.

Blind testing only adds stress if you worry that you can't spot the 'differences' without 'knowing' what's what. Anyone sure of their ability shouldn't worry. Shills, idiots and liars of course should worry- they are about to be exposed.

I'll probably be heading off to see Steven Toy this week, to see if he can spot differences in his power chords or not, I might even listen to them myself...

 

Hi Thorsten,

Very nice to see you here again, even though 'here' is now called something else!

I gather you have a young child now, does she listen to music much? What does she like? I went to my brothers wedding this weekend and met my cousins 1 year old boy. He really liked dancing to Kaylee music and surprisingly learned how to move in rhythm from me in just about 30 seconds!

P.S. I'm engaged now to a Korean girl called Eunji.

 

Hi,

Again, this is not what I said, I said it does not improve statistical power, as there is non. Statistical power is a measure how much the test may be generalised, especially seeing that those who tout blind tests invariably use the results of their tests to claim that "A makes no difference" that implies relevance outside the specific test.

Plus, if you remove one set of bias it is of course good. If you then add several more back AND you do not account for them, that is worse than before.

However as this is clearly a religious issue, I am not trying to change any-ones religion.

Actually, he CAN hear the difference in a way that is sufficient to him, or he would be listening with what came in the box. However, it seems you need him to to give you proof.

I would suggest an around 90% risk of returning a null result in your test, if the kind of change that mains cables (can) make is at the average.

Ciao T

 

Hi Simon,

Sounds like our Girl. She likes anything with good beat. Not so fond of classical, but Jazz and Bossa Nova is okay.

Congrats, is she the one from the Photos on Facebook?

Ciao T

 

Yes indeed, but in the same way that I might hear voices telling me that Satan is living in my garden shed.

You point about generalising is well made, on that we share some common ground, but our mains cable exponent does precisely that - generalise wildly and he should be taken to task if he offers advice to others.
You won't have followed the particular discussion in that case, but the person in question was adamant that the degree of bass 'bloom' added or subtracted by his mains cables would be clearly audible above a lorry thundering past outside the house. Yes I know.....
I would suggest that a not very sensitive BT routine is required in this instance, given the claim

Good luck with that.
PM me the address. If we haven't heard from you after a few days I'll send in the search and rescue.

 

Anyway, enough of this circular banter.

Thorsten, what are you up to these days?
Are you still working with AMR or have you got other things on the go?

Still have my WAD units knocking around, and remember the day at Dev's where you sketched out the schematics, amended them for better performance and handed me the drawings saying, 'here you go , do this'

 

Does her dancing tell you if the hi-fi is sounding good or not? Apparently this little guy doesn't dance much to music at home but he likes it live! (They don't have a good hi-fi).

Yes I'm sure you'll have seen Eunji in Facebook. I'd like to see you and her getting along, it would be an interesting match of personalities.

As Rob asked, what are you doing work wise now?

 

Hi,

Yes still working AMR on design with AMR, doing some financial systems consultancy as well...

Yeah, I do things like that. You really should try it.

Ciao T

 

Simon,

Well, we do have a good HiFi and it even runs for TV and the cartoons she likes to watch. BTW, you can see what is in the system on my FB.

She does not really dance to anything outside, even if it has a good beat. Well, she grew up from before birth with "HiFi".

I normally get along with most people, I dislike however "blue meanies" (that is people who are constitutionally incapable of having fun and hence don't see why anyone else should have any)...

 

My son is 3 1/2 half now. Huge fun.

 

I am liking this!

 

I did! ...... and very good it was.

 

Ha ha!

These things just happen to you - beware. I went from 'no, heck no, no children evah!' to meeting a certain someone with a 2yr old in tow.. who promptly managed to seriously steal my heart (via orchestrated enemy action and disinformation, obviously...)

I'd even believe medicinal doses of the Spice Girls etc along the way was fun.

Anyway - Jess is off to Uni in a couple of months and I just don't know where the intervening years went.

Enjoy it all!

Best
Martin Clark