Science stories

Bethany has begun her series on Internet Science with a post on "how they reel you in," about how "Look, shiny!" stories and images catch your attention and play on your biases. I look forward to the rest.

There are a few things that raise red flags when I see them in headlines. Some of them are easy to describe.

1. History: I'm old enough to have seen a dozen breathless headlines about how some new scrolls and "suppressed gospels" may overturn our understanding of Christian history. Same old, same old. Whether it's because reporters have no knowledge of history or because editors think we don't, the same old discoveries and breakthroughs appear year after year. Sometimes a quick googling shows that the same researcher issued an almost identical claim 5 years ago, and 4 years before that and... Unfortunately familiarity with the history isn't something you can pick up in five minutes
2. Paradigm-changing breakthroughs are very rare. Incremental progress is the rule. If it sounds too dramatic, somebody is probably pulling your leg or trying to make a quick buck. This is especially true in attention-grabbing fields: anything to do with weight loss or aging or sex or politics.
   

   The IceCube experiment announced that they had probably discovered extra-galactic neutrinos, some of which were puzzlingly higher energy than expected. Did you notice the little word "expected" in there? To have spotted them at all is a tour de force, but pretty much everyone in the field expected there to be neutrinos, some from other galaxies. The general public didn't know anything about them, and the news (I hope) unveiled a new part of the world for them. But these are a tool for further research, not a paradigm-shifting breakthrough by themselves. A milestone.

3. Herding cats has nothing on coordinating scientists. "Every French soldier carries in his cartridge-pouch the baton of a marshal of France." Every scientist seems to have a place on the shelf for a Nobel Prize. If you're head of the team doing the cutting edge research, you are helping shape the culture of the field, but everybody else harbors a "but what if it were like this" question that they'd like to put to the test.
   

   If the question is simple (special relativity) there's virtual unanimity, but even there you find scientists saying "Have we proved it for every case yet?" If the question is complicated and simulations are hard to do, you should find plenty of discord in the wings. If unanimity is claimed for a difficult problem, be skeptical. Anthropogenic climate change is one good example: the problem is very hard, proxy measurements are not easy to define, and the climate models are hard to test. (They're no good for year-to-year variation, and as for predicting the far future... the Sun monkeys with medium term results, and I'll most likely not be alive in a century.)
   

   If at all possible, compare the story with what the received explanation is in the field. This is no guarantee (see Lysenko, AGW, etc), but it is generally a good rule that Newton stood on the shoulders of giants and so ought the scientists in the report.

4. Does the story have some political implications? If it purports to show that Your Team ™ is smarter than the rest, or that some resource is effectively infinite or on the contrary almost exhausted--qui bono?
5. Is it sociology? Or drug related? Wait for verification. Then wait some more. Sociologists tend to study WEIRD people and have sample sizes that are way too small. Drug test results are very heavily biased in favor of positive results--negative ones typically don't get published. When you only look at the right side of the distribution of results from small effects, things look much better than they really are. Plus, it turns out that lots of promising drugs to treat X wind up causing Y, and are quietly dropped from view.
   

   Sociological studies that show that people are happier if they are virtuous just aren't sexy; discovering that "Good things come to those who wait" is a sound rule doesn't get publicity or grant money.

6. Will the world be destroyed? Check to see if the scientist has just published a book--and needs publicity
7. Take a quick gander at the story itself, which often has a family resemblance to the headline but doesn't support it very well.
8. If it talks about other universes, read the funnies instead.

Read How to Lie With Statistics.