The Perils of Double Blind Testing

While in Japan I had an interesting discussion on whether or not one can hear the differences in cables. I am not in the ‘can hear a difference’ camp. To quote from one of my own articles:

The first and most obvious [explanation for claims that cables sound different] is that there are differences, and these listeners have the sufficiently acute hearing and refined listening experience to enable them to detect it. This has some unpleasant implications, though.To understand this, you need to accept that a cable cannot improve sound. It is a passive device. At its best, an interconnect cable will transport the sound without any degradation at all. But if it did alter the signal in some way, that alteration must be a degradation.

So, next time you are in a hifi shop that offers to sell you cable A for $100, or for even better results, cable B for $200, consider what they are saying. They are saying that they will charge you $100 for a cable which they know to actually damage the signal!

Anyway, I suggested that the believers could make a handy $US1 million by participating in the James Randi Educational Foundation‘s Million Dollar Challenge. The usual response was made: a respected reviewer of equipment had tried to undertake the challenge, but they were unable to agree on a suitable protocol. Having returned, I’ve been checking this out.

First, I am somewhat of an extreme sceptic (aka skeptic) myself about a wide range of matters including, obviously, many aspects of high fidelity. Second, Randi’s foundation really does offer a million bucks. It has the money in the bank, ready to go.

Third, I found a previously negotiated challenge on the Randi site which apparently fell through. The whole sorry saga is laid out under ‘MICHAEL ANDA, Audio Critic‘. Two things stood out for me about this: the Randi representative was, I think, needlessly aggressive about the matter, while the Anda chap appeared to have an unusually high level of self-regard. It never got to the point of any actual listening.

Fourth, I found an interesting blind test conducted outside the auspices of Randi’s Million Dollar Challenge. This is well summarised in the first post on this thread, and the person who actually undertook the blind listening tests, Mike Lavigne, adds more detail in this post. In brief, the test was abandoned after seven valid trials. The results of trying to pick the cables: right, wrong, wrong, wrong, right, right, wrong. Apparently a seriously high-end system was employed. Lavigne felt very confident, before he donned the blindfold, that he could tell two sets of cables apart. He felt confident during the test that he was getting it right. But he wasn’t. I have to admire this fellow’s courage and integrity.

Fifth, well known Stereophile reviewer Michael Fremer was on the verge of undertaking the Challenge, but the whole thing fell apart. In brief, the thing started with some snarky comments by Randi about Pear Cable’s Anjou range of speaker cables. These cost $US7,250 for a 12 foot (approx four metres) pair. Apparently Pear Cable were initially keen to provide a loaner for the test. Fremer and Randi were both keen. But Pear pulled out. At those prices, it wasn’t surprising that neither Fremer nor Randi were prepared to actually pay for them.

Gizmodo has a useful, and fairly neutral (a least, with regard to Randi and Fremer), account of the matter. It lays the blame squarely on Pear Cable.

According to Fremer, James Randi proclaimed on his website ‘that the “blowhard” Fremer would never take the challenge, that the matter was closed, and that it was time to move on to the next challenger.’ Fremer says he responded on Randi’s forums and Randi eventually, kind of, retreated.

At this point in the story, I was feeling pretty sorry for Fremer. I think he’s probably wrong, and is most likely imagining any differences he claims to hear in cables, but that doesn’t mean he’s insincere, or a blowhard. I was thinking that Randi, after fifty years of exposing squirming ‘psychics’ and other practitioners of the paranormal, was getting a bit short tempered.

But now I think there was a misunderstanding. For what was printed on Randi’s site (the link from Fremer’s column no longer works) under the heading ‘Blake Withdrawls from PEAR Cable Challenge‘ (sic) was an email from Adam Blake, the CEO of Pear Cables, which opened:

At the request of Michael Fremer, with whom we have been communicating regarding his challenging of your assertions regarding high-end audio cables, we would like to inform you directly of Pear Cable’s decision to not participate in your claimed challenge.

I don’t know about you, but to me that at first glance read as though Blake was claiming that Fremer had withdrawn. The whole sentence does not say that, but that was my first impression. I strongly suspect that Randi may have inferred that meaning, resulting in his remark later in the same post:

Third – and most interesting – this retreat by Adam Blake effectively closes the current challenge, much to the relief of both Fremer and Blake, of course. Actually, I must admit that this was a rather clever way of squirming out of the huge dilemma in which these two blowhards found themselves.

I think at the least both Fremer and Randi approached this in the beginning with good will. Fremer strongly believed that he would win the Challenge. Randi strongly believed he would not. Randi is highly suspicious of those who makes what he believes to be outlandish claims for the reasons outlined above, and over-reacted to a misinterpretation of the Blake letter.

As to whether cables make a difference in sound, all of this goes to prove … well, precisely nothing.

Even the failure to notice a difference mentioned under ‘Fourth’ above doesn’t prove anything. As correctly noted at certain places on that thread, the test failed to disclose any noticable differences by those participants alone in those conditions alone.

I have in my possession copies of articles showing unblind, followed by blind, tests of sound. The unblind tests had marked differences noted. The blind ones had no differences noted. Same people, same equipment.

There’s a pattern here. People are convinced that there are differences. But every time they are put to a reasonably well conducted test, the differences disappear. This, in itself, proves nothing for the reasons already mentioned.

But it is suggestive.

Even more suggestive would be even one well-conducted double blind test of a controversial claim that went the other way; that showed that a difference could be discerned. If anyone knows of such a test, please let me know. Because until such a test becomes known, it would seem prudent to remain sceptical.

This entry was posted in Audio, Mysticism, Rant, Testing. Bookmark the permalink.

2 Responses to The Perils of Double Blind Testing

  1. E Aldrich says:

    Having a lay person’s interest in both skeptical inquiry and audio equipment, I find it valuable to read anothers’ opinions on these unfortunately overlapping realms.
    Several years back I had reason to purchase a cable to kluge a computer soundcard to an amplifier’s headphone output for basic LP-to-digital recording trials. Imagine my surprise to find the best choice was a straightforward Radio Shack connector, with maybe gold-plated connectors as the only feature of note. The “rather monstrous” cables which cost about 5x more and did admitedly look swankier, actually introduced the worst dramatic hum, as did a few brands of cable options. Price was randomly corelated to apparent quality, with the pricier actually far worse. The only conclusion I found was that the entrenched Brand du Jure must have decided to actually make one or two adequate models, to sell for top dollar, then actually create sonically inferior products to fill the lower lines, so that people actually would be hearing a difference!
    As my old ears cannot detect anything above about 15000 hz, I have accepted the fact that aliens exist, and are distinguishable from mortals only by their insanely sensitive ears, capable of 40,000 Hz and the mystical talent of detecting chemical surface treatments to totally digital storage media.

  2. Pingback: rca feilkonstruert? - Side 9

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>