The Perils of Double Blind Testing

While in Japan I had an interesting discussion on whether or not one can hear the differences in cables. I am not in the ‘can hear a difference’ camp. To quote from one of my own articles:

The first and most obvious [explanation for claims that cables sound different] is that there are differences, and these listeners have the sufficiently acute hearing and refined listening experience to enable them to detect it. This has some unpleasant implications, though.To understand this, you need to accept that a cable cannot improve sound. It is a passive device. At its best, an interconnect cable will transport the sound without any degradation at all. But if it did alter the signal in some way, that alteration must be a degradation.

So, next time you are in a hifi shop that offers to sell you cable A for $100, or for even better results, cable B for $200, consider what they are saying. They are saying that they will charge you $100 for a cable which they know to actually damage the signal!

Anyway, I suggested that the believers could make a handy $US1 million by participating in the James Randi Educational Foundation‘s Million Dollar Challenge. The usual response was made: a respected reviewer of equipment had tried to undertake the challenge, but they were unable to agree on a suitable protocol. Having returned, I’ve been checking this out.

First, I am somewhat of an extreme sceptic (aka skeptic) myself about a wide range of matters including, obviously, many aspects of high fidelity. Second, Randi’s foundation really does offer a million bucks. It has the money in the bank, ready to go.

Third, I found a previously negotiated challenge on the Randi site which apparently fell through. The whole sorry saga is laid out under ‘MICHAEL ANDA, Audio Critic‘. Two things stood out for me about this: the Randi representative was, I think, needlessly aggressive about the matter, while the Anda chap appeared to have an unusually high level of self-regard. It never got to the point of any actual listening.

Fourth, I found an interesting blind test conducted outside the auspices of Randi’s Million Dollar Challenge. This is well summarised in the first post on this thread, and the person who actually undertook the blind listening tests, Mike Lavigne, adds more detail in this post. In brief, the test was abandoned after seven valid trials. The results of trying to pick the cables: right, wrong, wrong, wrong, right, right, wrong. Apparently a seriously high-end system was employed. Lavigne felt very confident, before he donned the blindfold, that he could tell two sets of cables apart. He felt confident during the test that he was getting it right. But he wasn’t. I have to admire this fellow’s courage and integrity.

Fifth, well known Stereophile reviewer Michael Fremer was on the verge of undertaking the Challenge, but the whole thing fell apart. In brief, the thing started with some snarky comments by Randi about Pear Cable’s Anjou range of speaker cables. These cost $US7,250 for a 12 foot (approx four metres) pair. Apparently Pear Cable were initially keen to provide a loaner for the test. Fremer and Randi were both keen. But Pear pulled out. At those prices, it wasn’t surprising that neither Fremer nor Randi were prepared to actually pay for them.

Gizmodo has a useful, and fairly neutral (a least, with regard to Randi and Fremer), account of the matter. It lays the blame squarely on Pear Cable.

According to Fremer, James Randi proclaimed on his website ‘that the “blowhard” Fremer would never take the challenge, that the matter was closed, and that it was time to move on to the next challenger.’ Fremer says he responded on Randi’s forums and Randi eventually, kind of, retreated.

At this point in the story, I was feeling pretty sorry for Fremer. I think he’s probably wrong, and is most likely imagining any differences he claims to hear in cables, but that doesn’t mean he’s insincere, or a blowhard. I was thinking that Randi, after fifty years of exposing squirming ‘psychics’ and other practitioners of the paranormal, was getting a bit short tempered.

But now I think there was a misunderstanding. For what was printed on Randi’s site (the link from Fremer’s column no longer works) under the heading ‘Blake Withdrawls from PEAR Cable Challenge‘ (sic) was an email from Adam Blake, the CEO of Pear Cables, which opened:

At the request of Michael Fremer, with whom we have been communicating regarding his challenging of your assertions regarding high-end audio cables, we would like to inform you directly of Pear Cable’s decision to not participate in your claimed challenge.

I don’t know about you, but to me that at first glance read as though Blake was claiming that Fremer had withdrawn. The whole sentence does not say that, but that was my first impression. I strongly suspect that Randi may have inferred that meaning, resulting in his remark later in the same post:

Third – and most interesting – this retreat by Adam Blake effectively closes the current challenge, much to the relief of both Fremer and Blake, of course. Actually, I must admit that this was a rather clever way of squirming out of the huge dilemma in which these two blowhards found themselves.

I think at the least both Fremer and Randi approached this in the beginning with good will. Fremer strongly believed that he would win the Challenge. Randi strongly believed he would not. Randi is highly suspicious of those who makes what he believes to be outlandish claims for the reasons outlined above, and over-reacted to a misinterpretation of the Blake letter.

As to whether cables make a difference in sound, all of this goes to prove … well, precisely nothing.

Even the failure to notice a difference mentioned under ‘Fourth’ above doesn’t prove anything. As correctly noted at certain places on that thread, the test failed to disclose any noticable differences by those participants alone in those conditions alone.

I have in my possession copies of articles showing unblind, followed by blind, tests of sound. The unblind tests had marked differences noted. The blind ones had no differences noted. Same people, same equipment.

There’s a pattern here. People are convinced that there are differences. But every time they are put to a reasonably well conducted test, the differences disappear. This, in itself, proves nothing for the reasons already mentioned.

But it is suggestive.

Even more suggestive would be even one well-conducted double blind test of a controversial claim that went the other way; that showed that a difference could be discerned. If anyone knows of such a test, please let me know. Because until such a test becomes known, it would seem prudent to remain sceptical.

Posted in Audio, Mysticism, Rant, Testing | 2 Comments

The 6th Day – a comparison

Go here to see my Blu-ray vs DVD comparison.

Posted in Blu-ray, Disc details, DVD | Leave a comment

Back from Hamamatsu

I was in Japan last week as a guest of Yamaha Music. Had a great time. Ate dinner at the restaurant which inspired Tarantino for the restaurant fight in Kill Bill Part 1. Saw some interesting products.

Yamaha has a truly superb home theatre room at its Hamamatsu headquarters, where it showed off its new RX-V2650 RX-V2065 home theatre receiver and one of its new Blu-ray players (the latter seems to be based on Sharp hardware).

Also had a tour of the whirlwind woodwind factory tour, and a somewhat slower one of the upright piano factory. More details in due course.

Posted in Equipment, Travel | Leave a comment

Yes, it is louder

Earlier in the week I remarked on the the live Pink Floyd release, ‘Pulse’, on DVD. You may recall that this has the surround soundtrack in two formats: 448kbps (the maximum specified for DVD) and 640kbps (the maximum specified for Dolby Digital in any format). I noted that:

my impression was that the 640kbps version was a decibel or two louder than the 448kbps version, so that would immediately invalidate any A-B comparison without a great deal of setting up

I’ve finally finished playing with receivers of limited capability, so this morning I installed a rather nice Denon AVR-4310. This has rather fuller signal information screens than the low cost receivers, so I played one of these discs again.

Yes, the 640kbps version has a Dolby Digital Dialogue Normalization (‘dialnorm‘) value of -4dB, which is a very common value for this. But the 448kbps version has a dialnorm of -5dB. Assuming that the same source mix was used for both encodes, the 448kbps will come out one decibel quieter than the 640kbps. If you see any comments around that 640kbps sounds markedly better than 448kbps based on listening tests on this disc, bear this in mind. A 1dB boost just about always makes things sound ‘better’.

Posted in Disc details, DVD, Testing | 1 Comment

Paramount’s BD-Live commitment

A while back I did a ‘Ten Best’ piece on Blu-rays in Sound and Image. Best picture quality, best sound and so on. For Best BD-Live I put down Transformers on the basis of its downloadable ‘Intelligence Mode’. As it happens, I go into this disc’s BD-Live section a lot, primarily to check that BD-Live is working okay in whichever Blu-ray player I happen to be reviewing.

The other day I noticed that there seemed to be more content in BD-Live for this disc than previously, and today I’ve had a proper look. Here’s what’s available now (obviously, quite a bit prompted by the sequel). I’ve copied the descriptions from the download area.

  • Intelligence mode: ‘An enhanced playback mode for the movie featuring on-screen widgets to track the action of the film. Widgets include: GPS Tracker, Robot Status Alerts and Data Panel’
  • Exclusive Sneak Peek: ‘Transformers: Revenge of the Fallen … We’re Gonna Die’
  • Exclusive Sneak Peek: ‘Transformers: Revenge of the Fallen … Shanghai’
  • Exclusive Sneak Peek: ‘Transformers: Revenge of the Fallen … Adobe Hut’
  • From the Set: ‘Day 1: An exclusive look at the first day of shooting Transformers: Revenge of the Fallen’
  • From the Set: ‘A Day in the Shade: A comical look at the differences between Megan and Shia’s on-set treatment while making Transformers: Revenge of the Fallen’
  • Transformers Deleted Scene: ‘Maggie and Glen Introduction’
  • Transformers Deleted Scene: ‘Fig Death’
  • Fly on the Set: Pentagon: ‘On the set filming of the Pentagon Military Command Center’
  • Music and Mayhem: ‘Renowned special effects house ILM deconstructs the blockbuster action of Transformers’
  • Metal in Motion: ‘Special effects tests and early robot animation’
  • Stunts 101: ‘An in depth look at the amazing stunts in Transformers’
  • Choose Your Weapon: ‘A fly on the wall glimpse into the selection of weapons and how they were used by the film’s soldiers’
  • The Man in the Ice: ‘From design to production, watch the Arctic come to life on various sets in Los Angeles, California’
  • Voices: ‘The voices behind the Autobots and Decepticons’
  • Audition Tapes: ‘Megan’s original Transformers audition’
  • International Trailer: ‘Trailer C English’
  • Robot Ninjas: ‘Using the stunt crew’s home videos, animatics and the final features, we break down one of the fight sequences from the film’
  • Menubots: ‘Transform your in-movie menubar to one featuring a Transformer. Select from multiple looks …’
  • Transformer Profiler: ‘… serves as an in-movie guide for learning more about the production including cast, crew, locations, etc. Once downloaded, access the Profiler by pressing the red button on the remote to turn on or off’

The addition of all this stuff does lend weight to the ‘Live’ part of BD-Live.

UPDATE (Thursday, 10 September 2009, Thursday, 10 September 2009): Unfortunately, the video (at least the one I checked), doesn’t download for retention on your persistent storage, but is streamed. That means that every time you watch it, it will cost you bandwidth. Pity. Things like the Intelligence mode and Menubots remain in your persistent storage so you can use them any time you like.

Posted in BD-Live, Blu-ray, Disc details | Leave a comment

HDMI Cable weirdness

What hair I have is so short at the moment that I am unable to get any purchase on it whatsoever. Which is just as well, otherwise after the last couple of hours I would have none left.

I start this period with a Pioneer Blu-ray player plugged into an Onkyo home theatre receiver, but I need to put a Sony Blu-ray player through its paces. So I pull out the Pioneer player and put the Sony in its place. All works fine.

I need to check out the audio decoding capabilities of the Sony, and the Onkyo doesn’t offer much information in this regard, so I plug in a Yamaha receiver. All works fine. I check out part of what I need to check on the Sony player. One of the weirdnesses of Sony Blu-ray players is that, Sony alleges, they will decode DTS-HD Master Audio, but I’ve never been able to get one to do so. Apparently they will only do this if set to ‘Direct’ output. With a receiver which does the decoding, they supply the bitstream instead. Consequently I’ve never been able to confirm that they really will decode DTS-HD Master Audio.

But I also have here an inexpensive Sony receiver which has HDMI input, but no decoders, and it provides signal information. So I pull out the Yamaha, and replace it with the Sony. The HDMI cable from the Sony Blu-ray goes into the ‘BD’ input on the Sony. I switch it on and it shows ‘HDMI’ on its front panel display, but no picture is coming through. I switch off the Blu-ray player and restart it. Likewise for the receiver, for the TV for everything. I change settings. I change inputs.

I walk away for five minutes and engage in a bit of cursing.

I go through the whole process again. Nothing.

Remember, everything was the same except that now I had a Sony receiver instead of a Yamaha one in place.

As a real longshot, I switched to a different Blu-ray player — the Oppo BDP-83 — with its own HDMI cable into a different input on the Sony receiver. Instantly I have a picture up on the display (a glorious 65 inch Panasonic plasma at the moment).

Initial diagnosis: Sony Blu-ray player won’t work with Sony receiver! But that seemed somewhat unlikely. Perhaps I’d wrecked the cable I had been using between the two Sony units. It’s an excellent cable from Kordz, but goodness knows it gets a fair old workout and has for some years. So I pulled the Oppo’s HDMI cable off and used it with the Sony player. It worked!

So I tried a different cable: another Kordz one identical to the first one, but five metres instead of two. It worked!

So I tried a third cable: my very first HDMI cable which is thin and nasty and cost $50 back when they were very hard to obtain. It worked!

So I put the original cable back into play. It worked!

So I have no idea what went on there. But, in the end, I can confirm that the Blu-ray player really does decode DTS-HD MA!

UPDATE (Thursday, 10 September 2009, 9:38 am): I think I’ve worked it out. Sony players seem to be a touch sensitive as to the precise angle at which some HDMI cables are inserted, presumably well-used ones such as my Kordz. I seem to recall a slight touchiness from Panasonic players too.

Posted in Cables | Leave a comment

The World’s Worst Cable

Tangled Digital Interconnect What happens when you search your cable cupboard, find a collection of random audio cables (at least one a quarter of a century old), join them all together into one 44 metre long hodge-podge, and use it for digital audio?

You might be surprised.

I did it, and wrote about it in the just-out September/October 2009 issue of Geare.

Posted in Cables | Leave a comment

640kbps Dolby Digital on DVD

One of my daughters gave me the Pink Floyd ‘Pulse’ DVD set for Fathers’ Day yesterday. Absolutely brilliant concert from 1994, with superb sound. I was startled to find under audio options that you can select Dolby Digital 5.1 sound at either 448kbps or 640kbps. The latter is common on Blu-ray, but is not part of the DVD specification, and a warning in the booklet that comes with the discs suggests that it may not work with some systems.

Still, it worked with mine and it sounded excellent. Better than the 448kbps version? Who knows. I switched quickly between the two, and my impression was that the 640kbps version was a decibel or two louder than the 448kbps version, so that would immediately invalidate any A-B comparison without a great deal of setting up.

Posted in Audio, DVD | Leave a comment

Elementary Arithmetic

You have 285,200 births in Australia in 2007, according to the radio this morning, of which an estimated 2,000 are home births. That, says the presenter, is ‘point zero, zero seven per cent of births’.

A recent issue of a Skeptics Magazine initially claimed that 7.6% of the US population was behind bars.

This article — which I otherwise think is excellent — includes the following statement: ‘but it represents about one third of a per cent (0.0033 per cent) of the world dolphin population’.

Is doing percentages really that hard? To find a percentage you divide the count of the category of interest by the total count … and then you multiply by 100. Pretty easy. The figure should be expressed the same whether written in words are spelt out.

The three real figures are, respectively, 0.7%, 0.76% and 0.33%.

Posted in Rant | Leave a comment

BD-Live Travails

Here’s an email I sent last night to ‘BD-Live Help’. I mentioned here a while back how my BD-Live logon with The Da Vinci Code had failed. I whinged to Sony’s PR company at the product launch mentioned in the post, and they undertook to do something about it. As a result, about three days ago I received an email from BD-Live Help with a reset password. So here’s what happened (logon IDs etc X’d out for security):

Dear BD-Live HelpThank you for resetting my password and advising a new one, as per your email below. Unfortunately, this has failed to resolve my problem. Here’s the sequence of events leading to this pass.

  1. I was early onto BD-Live, back with the original release in Australia of Men in Black. At that time (most likely) I registered for BD-Live as I explored the system’s capabilities.
  2. Since then I forgot my password. However I remembered my username: ‘XXXXXXX’. My email address was XXXXXXX@XXXXXXXX.
  3. When I was reviewing the Australian release of The Da Vinci Code, I attempted to use the cinechat feature, but without a password could not get it to work. So I tried creating a new registration. This insisted that a username had to consist of at least one letter and one number, so I tried XXXXXXX1. This registration failed on the basis that my email address was already included with another registration.
  4. Following my raising the matter with Sony management here in Australia and their PR, I received your email of 1 September.
  5. I have just loaded The Da Vinci Code and I successfully ‘Registered My Disc’ in the normal BD-Live section of the disc using my user name ‘XXXXXXX’ and the new password you provided.
  6. I then exited that section of the disc and entered the ‘cinechat’ section. This required me to logon again. I entered ‘XXXXXXX’ and the advised password, and received the following message:’Login Failed – The BD-Live user name/password you entered is invalid. Please select ‘Ok’ and try again.’

    Needless to say, I’ve ‘tried again’ several times, with great care as to make sure I entered the correct password. I received the same message on each occasion.

  7. So, I’m still stuck. Apparently the cinechat system uses a different verification mechanism to the regular BD-Live system. To confirm, I exited and went back to the normal BD-Live section of the disc and again used the ‘Register Your Disc’ item to logon. When I did so, after a pause the system returned a message saying ‘DISC IS ALREADY REGISTERED’, suggesting that it successfully logged on and checked the disc status. (Incidentally, when I entered the logon screen, the fields had been automatically filled, with the information loaded from the player’s persistent storage. Why doesn’t the cinechat feature also have this pre-filling capability?)

Suggestions?

Posted in BD-Live, Blu-ray, Testing | Leave a comment