Significance isn't always significant

The BBC report the new climate change data with the headline Global warming since 1995 "now significant". While there is an attempt to explain what this means, I think the headline is deceptive and the explanation is unclear. Let me try to do better.

Suppose I have a coin which I suspect is biased in favour of heads. I toss the coin ten times and it comes down heads six times. What conclusions can I draw from this? Well, if the coin is not biased I would have expected it to come down heads five times out of ten. But, crucially, coin tosses are random: if you repeatedly toss a fair coin ten times it doesn't come down heads exactly five times out of ten every single time. Sometimes you get six heads; sometimes you get four. So the fact that I got heads six times is entirely consistent with the coin being a fair coin: for an unbiased coin to come down heads six times out of ten is not that unusual. This evidence is not, therefore, sufficient, to support my suspicion that the coin is biased.

On the other hand, if my coin had come down heads nine times out of ten, I may start to feel justified in my belief that it is biased. Fair coins do sometimes come down heads nine times out of ten, but only quite rarely. If my coin did so I may prefer to believe that it is biased, since coming down heads nine times out of ten is not that unusual for a biased coin.

An outcome is said to be statistically significant if the probability of it occurring by chance is very low. How low is low is entirely subjective. A benchmark that is often used is 5%, though the researchers at CERN who are looking for the Higgs boson use a far lower probability, as did the physicists who announced they may have found particles travelling at a speed faster than that of light.

But what statisticians mean by "significant" is simply that there is evidence to suggest, for example, that the coin is biased. The term says nothing about whether the bias is meaningful in practical terms: practical significance. A conclusion of statistical significance does not distinguish between a coin that is biased in such a way that it comes down heads 50.1% of the time and a coin that comes down heads 90% of the time.

The latest climate change data does offer evidence that the temperature has increased. But there are two reasons to be cautious. First, it remains possible (though unlikely) that the increase is simply a random variation, equivalent to a fair coin coming up heads nine times out of ten. Second, even if the increase is real, it may not be significant in practical terms.