Roblog

The news is full of stories lifted straight from scientific papers, lots of which are of dubious quality. David Epstein highlights one recent example – a story with the headline “surgeons who listen to AC/DC are faster and more accurate” – which demonstrates a common abuse of statistics:

“I recommend that when you see this sort of ultra-nuanced effect in a news article, it may be a sign that the researchers inappropriately (but often not maliciously) sliced and diced their data in order to create some tantalizing positive finding, which – given enough data and enough slicing and dicing – they will inevitably find among the many possible false positives.

“Let’s say a study starts out asking whether music makes surgeons perform better, but the results show nothing. Dead boring. So then the researchers separate the data into surgeons who heard soft rock versus hard rock. Ok, now the hard rock shows an effect. But, hmmmm, still no effect for soft rock. But if you look only at the data when the music is low-volume, there’s an effect. Interesting! Headlines!

“But the reality is often that the researchers just sifted the data so much that they were bound to find false positives.”