Tuesday, May 21, 2013

"Studies have shown..."

We've all been told that "studies have shown" something at one time or another.  Sometimes, our interlocutor is kind enough to give us a citation (and sometimes they aren't).  Well, let's do a thought experiment (if you're already familiar with significance testing, feel free to skim the next paragraph).

Suppose you give 20 labs a drug and a placebo, and tell them to test one against the other in clinical trials.  But instead of actually giving them a drug and a placebo, you give them two identical placebos (originally, I was going to use a homeopathic remedy vs. a placebo, but I didn't want to get sidetracked).  Assume the labs all use large sample sizes, statistical normalization, double blinding, and various other best practices.  None of them make any mistakes (or outright fraud, for that matter) and they all conduct proper, well-designed experiments.  Even under these ideal conditions, one of those labs (on average, and for pedants, we're assuming they all use α=5%) will tell you there's a statistically significant difference between the placebo and itself.