Later, experimenters at Caltech felt that Feynman’s very presence exerted a sort of moral pressure on their findings and methods. He was mercilessly skeptical. He loved to talk about the famous oil-drop experiment of Caltech’s first great physicist Robert Millikan, which revealed the indivisible unit charge of the electron by isolating it in tiny, floating oil drops. The experiment was right but some of the numbers were wrong–and the record of subsequent experimenters stood as a permanent embarrassment to physics. They did not cluster around the correct result; rather, they slowly closed in on it. Millikan’s error exerted a psychological pull, like a distant magnet forcing their observations off center. If a Caltech experimenter told Feynman about a result reached after a complex process of correcting data, Feynman was sure to ask how the experimenter had decided when to stop correcting, and whether the decision had been made before the experimenter could see what effect it would have on the outcome. It was all too easy to fall into the trap of correcting until the answer looked right. To avoid it required an intimate acquaintanceship with the rules of the scientist’s game. It also required not just honesty, but a sense that honesty required exertion.
James Gleick: Genius: The Life and Science of Richard Feynman
Even something as seemingly free of bias as measuring stuff in physics can be biased in reality by expectations. Who would have thought that the great Millikan may have actually made a mistake?
So it isn’t really a surprise when you read something like ‘The Dishonesty of Sighted Listening Tests‘, which relates how virtually indistinguishable loudspeakers (as far as blind tests went) became very different when they could be seen, and how the ranking orders of them changed.