Publication bias

It is an open secret that research publication is biased in favor of favorable results. Suppose you spend a year and a hundred grand testing whether lidocaine helps relieve epilepsy. Journals aren't that interested in publishing "lidocaine didn't work" no matter how helpful it is to the field as a whole. Nor is the researcher that eager to trumpet the fact that he got it wrong.

This can be a serious problem with drug research. Suppose 5 researchers run 5 lidocaine tests and 1 finds that it makes epilepsy slightly worse, 3 find that it doesn't do much of anything, and 1 finds that it makes it slightly better. Guess which paper is more likely to get published?

In a study of whether bilingualism makes you smarter (it doesn't), Angela de Bruin came up with a clever way to get past part of the bias: look at conference proceedings instead.

The rationale was straightforward: conferences are places where people present in-progress research. They report on studies that they are running, initial results, initial thoughts. If there were a systematic bias in the field against reporting negative results—that is, results that show no effects of bilingualism—then there should be many more findings of that sort presented at conferences than actually become published.

That’s precisely what de Bruin found. At conferences, about half the presented results provided either complete or partial support for the bilingual advantage on certain tasks, while half provided partial or complete refutation. When it came to the publications that appeared after the preliminary presentation, though, the split was decidedly different. Sixty-eight per cent of the studies that demonstrated a bilingual advantage found a home in a scientific journal, compared to just twenty-nine per cent of those that found either no difference or a monolingual edge.

Kudos to Angela!