A Primer on Scientific Testing

In comments recently there has been a discussion about such things as double blind testing. I’ve recently been listening to back issues of the Skeptoid podcast, and as it happens, Episode 13 is entitled ‘A Primer on Scientific Testing’. This is a good, basic and quick overview. It’s only about seven minutes long and the 7.5MB MP3 is here.

In addition to covering single and double blinded tests, it discusses triple blinded (the people analysing the results don’t know which subjects received the ‘real’ or placebo experience or product being tested) and peer review.

If you’re into skeptical thinking, it’s worth listening to these Skeptoid podcasts, which are produced by Brian Dunning. I listen to a lot of skeptical stuff, and much of infuriates me from time to time, simply because the podcasters clearly have no understanding of economics. I’m not saying that you have to be a master of econometrics and be able to model international trade (such models are likely even less reliable than climate ones), but have a general sense of how people do actually respond to costs and benefits.

His podcast on the peak oil scare (issued in early 2008, when the price of oil was still ridiculously high, and well before the recent batch of news in which massive reserves of oil and gas are being found all over the world whenever someone pokes a stick in the ground) puts that understanding to good use. It’s here (7MB).

This entry was posted in Mysticism, Testing. Bookmark the permalink.

6 Responses to A Primer on Scientific Testing

  1. treblid says:

    On the subject of scientific testing: I do find an approach like this more acceptable: http://news.discovery.com/human/music-dopamine-happiness-brain-110110.html

    Blind testing for me is no more than a statistical analysis (not that there’s anything inherently wrong with it). Experiments like the above, to quote the article itself, is more definitive and heaps more interesting. (Yet to read the actual journal article, not sure if I can view it for free, but I’m sure I can not understand it :p).

    BTW, received the BD disc today. Thanks.

  2. Victor says:

    Treblid, I think you misunderstand what blind testing is. It is definitely not some form of statistical analysis, but rather a control mechanism to ensure no bias is introduced into results. What level of rigour you introduce into your controls would depend on what is at stake. Testing a new drug would have a great number of controls, whereas an investigative “study”, such as the one you linked, would be less formal.

    If one were to take the dopamine study and extend it to, say, try and show these people suffered a higher rate of heart disease, then rest assured that more rigour, such as blind testing, would be applied.

  3. treblid says:

    You could be right Vic, sooner or later I will understand how blind testing, double BT, or triple BT works.

    Was just looking at a blog discussing CES, the cost of cables – $600 USB cables, $7000 power cables, $3000 power strip. These examples are probably extremes, but still it’s rather over the top.

    Do hope people can work out a proper (if not better) way to experiment audio. If anything, it may hopefully weed out some of the dubious products (e.g. firepower).

  4. Stephen Dawson says:

    As Victor says, blind testing is a technique to try to remove some of the inevitable human bias when testing something. It is most commonly used when what is being tested is some affect that the thing has on humans. The need for it is because people have expectations. If you expect a particular result, you can sometimes experience it even if it is not really present. This can be feeling better after the administration of a fake cure, or hearing an improvement with the substitution of an impressive looking interconnect cable.

    By keeping the information from the subject, you can tell whether there really is an experience, or it was formed purely by the expectation.

    It’s likely that blinding could be useful in the broader experimental community. For example, it is clear that researcher biases can result in incorrectly measured results.

    Incidentally, the headline in the article you linked to is entirely incorrect. I’ll do a post about that later.

  5. treblid says:

    Appreciate if you can do a post about this, and find a link to the actual article (I’ve yet to do that). I subsequently wikipedia’ed Dopamine last night, it seems people are still arguing/discovering what Dopamine can actually do. :/ Turns out it’s not really the smoking gun I thought it was… 🙁

    As for keeping the information from the subject. I guess at the end of the day, I fail to understand how one can conclusively tell whether the subject is experiencing it, or imaging it (expectations or otherwise).

    Unfortunately I’m not sure how to explain the thoughts in my head eloquently. Think back to the schooling days, where as chemistry students were were given a tube of gas (or compound) and our assignment is to identify it. As blinding goes that’s as blind as we get. There are experiments we can perform to conclusively tell us what gas it is (e.g. if it burns with a blue flame, etc). The same repeatable behaviour appears to be missing here, something in me still says it is incomplete. One word continues to pop up in my head everytime I hear somebody talks about BT, or showed me links to it – randomness.

    Of course I could be very wrong here. :p As said, all I need is a practical to observe one in action, and all my doubts and questions could be cleared.

  6. treblid says:

    argh. Typo. imaging = imagining.

Leave a Reply

Your email address will not be published. Required fields are marked *