The Economist recently published a story about a competition that tested Stanley Milgram's famous "six degrees of separation" claim.

DARPA, the research arm of the Department of Defense in the US, staged the Red Balloon Challenge in 2009. Competitors had to locate ten red weather balloons that had been tethered at random locations across the US.

The intention was not that one person drive around the country with a pair of binoculars. Rather, I might ask all my friends on Facebook to look out for a red balloon and tell me if they saw one. They might then ask all their friends, and so on.

The winning team from MIT found all of the balloons in just nine hours using this type of strategy. But to encourage participation they offered $2,000 to the first person to send them the co-ordinates of a balloon. On its own this may not have been very efficient. So, crucially, they also offered $1,000 to whomever recruited that person to the challenge, and $500 to whomever recruited that person to the challenge, and so on.

One interesting question mathematically is how much money did the MIT team stand to lose? The Red Balloon Challenge offered a prize of $40,000.

In principle, the sender of a winning set of co-ordinates might have been been at the top of a long line of recruiters. Doesn't this mean that the MIT team risked an enormous payout?

Well, no. Consider a recruitment chain of seven people. The total payout would be:

$2,000+$1,000+$500+$250+$125+$62.50+$31.25 = $3,968.75

The seventh payment of $31.25 is pretty small. If there were more people in the chain, their payments would be even smaller. Nonetheless, lots of small amounts can quickly add up to a large amount.

Suppose there were 17 people in the chain. The total payout would be $3,999.97 (to the nearest cent). The seventeenth person would have got 3¢.

Even so, there could have been many more people in the chain, and might not the total have slowly grown to an unaffordable amount?

Suppose the total amount payable is T and that there are infinitely many people in the chain. Then,

T = 2000 + 1000 + 500 + 250 + ...

Now multiply both sides of this by 2:

2T = 4000 + 2000 + 1000 + 500 + ...

Finally subtract the first equation from the second:

2T – T = 4000 + 2000 + 1000 + 500 + ... – (2000 + 1000 + 500 + 250 + ...)

Which gives:

T=4000

So even if there had been infinitely many people in the chain, the total payout would have been $4,000. Since there were ten balloons that gives a grand total of $40,000 which was the value of the prize. MIT were certain to make at least a little profit, provided they actually won the prize. Since the kudos of winning was worth more than the prize, their investment was well worth the risk.

Series such as this one are called geometric series. One of the earliest examples of them was Zeno’s dichotomy paradox. For me to walk 4000 metres, I first have to walk 2000m, i.e. half the total distance. I then have to walk 1000m, i.e. half the remaining distance. Then 500m. Then 250m. And so on. No matter how far I've travelled there's always half the remaining distance left to go. So I have infinitely many stages to complete and will therefore never get to the end of them.

The flaw in the argument is that last sentence. It assumes that the infinitely many stages of the journey will take infinitely long to get through. But they won't.

Suppose I walk at 1m/s. Then the first stage will take me 2000s. The second will take 1000s. The third 500s, and so on. So the total time taken will be T = 2000 + 1000 + 500 + ... We now know that this adds up to 4000. So I can finish the journey in a finite amount of time. (Indeed the 4000 seconds you would expect me to need to walk 4000m at 1m/s.)

The above letter was published in today's Times Educational Supplement. The full text of the letter was:

Conrad Wolfram is right to argue that we need to rethink our priorities for mathematics education ("Computers do it better", TES, 30 March 2012). It is bizarre, for example, that we still teach students to use tables of probabilities three decades after the calculator ousted the books of logarithms so hated by our parents. Worse, exam boards often ask questions that specifically test the ability to use the tables, rather than focussing on candidates' understanding of what the probabilities really mean.

While such approaches rob the subject of its interest and practicality, resulting in fewer students pursuing it at higher levels, there is even more at stake. Dr Keith Devlin of Stanford University -- in an article titled "All the math taught at university can be outsourced. What now?" -- argues that the West's competitive advantage must come from mathematical creativity; countries like India are already our technical equals and charge a lot less for their skills.

Indeed, Hal Varian, chief economist at Google, has said that "the sexy job in the next ten years" will be that of statistician. Andreas Weigend, formerly chief scientist at Amazon, agrees: "Data is the new oil". Company surveys suggest there is likely to be a serious shortage of suitably qualified data analysts in the years ahead, yet we still seem to expect our bright students to become engineers.

If schools are serious when they claim to be preparing young people for the challenges of the 21st Century, they need to understand that forcing another generation to spend hours solving trigonometric equations and reading books of tables will only result in even more people telling me cheerfully how much they hated maths when they were at school.

In Radio 4's More or Less, sports statistician Rob Mastrodomenico took a look at a BBC blog that discusses whether premier league defending is getting worse. He focusses on the fact that the number of goals per game has risen from 2.59 in 2010/11 to 2.97 in 2011/12.

After discussing the football and the pundits, he turns to the statistics. It is unarguable that 2.97 is bigger than 2.59, but he asks whether the difference is statistically significant. He concludes that it is not and calculates how great the difference would need to be for it to be statistically significant. (I addressed the meaning of statistical significance in my blog posting Significance isn't always significant.)

Now the question of statistical significance arises when we have the necessarily limited information that we gain from a sample and we want to know what that tells us about a population. Because our sample is an incomplete picture of the population, we are met with uncertainty –- and that's where probability comes in. But if we have all of the data, there is no uncertainty and no role for statistical inference. If there is a difference between two things then we must conclude that, well, there is a difference between the two things.

Mastrodomenico's analysis of the BBC's football data treats the games played during the first half of each season as a sample of all the games that could theoretically be played. He sees each season as having an unknown, underlying goals-per-game average and the figures of 2.97 and 2.59 as estimates of their respective averages. It is thus reasonable to carry out standard statistical inference procedures.

It's rather like tossing a coin. We can suppose that any given coin has a particular propensity to come down heads. If we were to toss it many, many times we might end up with a very good idea of what that propensity is. It might be 50:50 heads:tails or, in the case of a biased coin, it might be 60:40. In practice we wouldn't toss the coin millions of times, say, but a couple of hundred times isn't too arduous and the resulting proportion of heads would give us an insight into the coin's underlying propensity towards heads. If we obtained 110 heads out of 200, we might reasonably argue that this is not inconsistent with a long-term propensity of 50:50 heads:tails -- although we would expect 100 heads out of 200 tosses, the process is a random one and a small deviation is not unlikely.

The problem for me in this approach is that I don't believe football matches are like coin tosses. Every coin toss is essentially the same as every other coin toss: though the outcome varies, the circumstances surrounding it are the same. And you can, in principle, repeatedly toss the coin as often as you want. But football matches are subject to many different variables: no two matches are the same. And you cannot endlessly repeat matches in an attempt to discern an underlying goals-per-game average.

In some respects probability and statistics are not like other mathematical disciplines: there are differing schools of thought. There isn't universal agreement, for example, on what a probability actually is and how you can calculate it. Some people believe that a probability is inherent to a situation so that all observers would agree on what that probability is. Others argue that if you have different information to someone else you can reasonably come to a different assessment of the probability of a particular event happening.

Mastrodomenico believes that any given football season has an underlying goals-per-game average and that we can gain insight into what that average is by looking at the actual games played. I look at it differently. I believe we can only go by those actual games. I think football matches are fundamentally different to coin tosses and cannot be treated in the same way. He concludes that we have insufficient evidence of a difference between last season and this season; I say there is a difference. Of course, whether that difference is meaningful and what caused it are completely different questions . . .

During the World Cup last year, a psychic octopus rose to fame by correctly predicting the outcome of eight football matches. Despite the relatively low probability of this happening by chance, most people are content to believe that the octopus was just guessing.

Does Young Apprentice work in the same way? Harry M has now been in the boardroom on the losing team six times out of six. Is it fair to conclude that he's the weakest contestant?

Perhaps he is, though he's had some notable and impressive moments. Yet given that the teams are regularly mixed up, is it not inevitable – or, at least, not unlikely – that one person would find themselves on the losing team every week simply by chance?

And does the winning team always win because they're the best? What role does randomness play? There seems to be a lot of post hoc ergo propter hoc analysis in the boardroom. James made great play of the fact that his team found a pocket watch at a considerably cheaper price than the opposition. But might that not have been more luck than judgement? As Zara countered, the watch she bought was the cheapest of the ones she saw; indeed it was cheapest by a large margin.

If the World Cup's octopus was lucky rather than psychic, might not Harry M be unlucky rather than a weak contestant?

More formal analyses on similar lines to the ones I'm suggesting here have been done in the sphere, for example, of investment management. There is evidence to suggest that many apparently successful fund managers may merely be just consistently lucky.

The BBC report the new climate change data with the headline Global warming since 1995 "now significant". While there is an attempt to explain what this means, I think the headline is deceptive and the explanation is unclear. Let me try to do better.

Suppose I have a coin which I suspect is biased in favour of heads. I toss the coin ten times and it comes down heads six times. What conclusions can I draw from this? Well, if the coin is not biased I would have expected it to come down heads five times out of ten. But, crucially, coin tosses are random: if you repeatedly toss a fair coin ten times it doesn't come down heads exactly five times out of ten every single time. Sometimes you get six heads; sometimes you get four. So the fact that I got heads six times is entirely consistent with the coin being a fair coin: for an unbiased coin to come down heads six times out of ten is not that unusual. This evidence is not, therefore, sufficient, to support my suspicion that the coin is biased.

On the other hand, if my coin had come down heads nine times out of ten, I may start to feel justified in my belief that it is biased. Fair coins do sometimes come down heads nine times out of ten, but only quite rarely. If my coin did so I may prefer to believe that it is biased, since coming down heads nine times out of ten is not that unusual for a biased coin.

An outcome is said to be statistically significant if the probability of it occurring by chance is very low. How low is low is entirely subjective. A benchmark that is often used is 5%, though the researchers at CERN who are looking for the Higgs boson use a far lower probability, as did the physicists who announced they may have found particles travelling at a speed faster than that of light.

But what statisticians mean by "significant" is simply that there is evidence to suggest, for example, that the coin is biased. The term says nothing about whether the bias is meaningful in practical terms: practical significance. A conclusion of statistical significance does not distinguish between a coin that is biased in such a way that it comes down heads 50.1% of the time and a coin that comes down heads 90% of the time.

The latest climate change data does offer evidence that the temperature has increased. But there are two reasons to be cautious. First, it remains possible (though unlikely) that the increase is simply a random variation, equivalent to a fair coin coming up heads nine times out of ten. Second, even if the increase is real, it may not be significant in practical terms.

Today Google celebrates Pierre de Fermat’s 410th birthday. He is most famously known for his “last theorem”, the subject of Simon Singh’s excellent book. (If you know about Fermat’s last theorem, jump down to "Molina’s Urns" below.)

The theorem is a generalisation of Pythagoras’s theorem. Every schoolchild knows that, for a right-angled triangle,

where x, y and z are the lengths of the three sides of the triangle, with z the length of the longest side.

Some right-angled triangles have sides whose lengths are whole numbers. For example, a triangle whose sides have lengths 3, 4 and 5 is right-angled because

Fermat was considering the more general equation

where n is a positive whole number. He wondered whether positive whole numbers x, y and z could be found that satisfied this equation if n was larger than 2.

He claimed that no such values could be found and he noted the fact in the margin of an algebra textbook he was reading. He wrote, “I have discovered a truly marvelous demonstration of this proposition that this margin is too narrow to contain.”

Well, that was in 1637. It wasn't until 1995 that Andrew Wiles and Richard Taylor published a proof of the theorem, some 358 year later.

(Incidentally, Fermat was both a lawyer and a mathematician. As am I :P)

Molina’s Urns

Some time before Fermat’s last theorem was proved, E. C. Molina invented a problem whose solution relied upon upon it. The problem would only have a solution if Fermat’s last theorem was proved to be false.

There are two urns each containing the same number of balls. Each ball is either black or white.

From each urn the same number of balls is selected. Each time a ball is selected from an urn its colour is noted and it is then put back into the urn before the next ball is selected.

The problem is this. Can the distributions of black and white balls in each urn be chosen such that the following condition is satisfied: the probability that the balls drawn from the first urn are all white must equal the probability that the balls drawn from the second urn are either all white or all black.

Let’s analyse the problem.

We define the following variables:

x = the number of white balls in the second urn

y = the number of black balls in the second urn

z = the number of white balls in the first urn

n = the number of balls selected from each urn

Since the total number of balls in each urn is the same, there are x + y balls in each urn.

The probability that a white ball is selected from the first urn is

Therefore the probability that all of the balls drawn from the first urn are white is

Similarly, the probability that all of the balls drawn from the second urn are white is

and the probability that all of the balls drawn from the second urn are black is

We require the probability that all of the balls drawn from the first urn are white to be equal to the probability that the balls drawn from the second urn are either all white or all black. In other words

which reduces to 

So in order to satisfy the requirement of the problem we must be able to find whole numbers which satisfy this equation.

We can certainly do this if n = 2, since the equation becomes Pythagoras’s theorem. Thus we can have 5 white balls and 2 black balls in the first urn, and three white balls and four black balls in the second urn.

If we draw two balls from each urn, the probability that both balls drawn from the first urn are white is

and the probability that the two balls drawn from the second urn are either both white or both black  is

which satisfies the requirement of the problem.

If, however, we choose more than two balls from each urn, Fermat’s last theorem tells us that there are no mixtures of black and white balls in each urn that will satisfy the requirement of the problem.

(I found this interesting problem in Fifty Challenging Problems in Probability, by Frederick Mosteller.)

There are many reasons to object to league tables, and the table above illustrates one of them very well: a tiny change in what you measure results in a big change in league table position.

The three columns rank research institutes according to three different, but similar, metrics. Yet the resulting rankings give very different results. For example, the Max Planck Society tops the first two tables, yet falls to 20th place in the third.

That 20th place ranking illustrates another objection I have to tables such as these. The metric on which the institutes are judged in third column is measured to two decimal places. This is bizarrely specific for something that has a number of subjective features. The definition of a "highly cited paper" is arbitrary, and being cited is itself somewhat arbitrary (and, indeed, may not be evidence that the paper being cited is good: the citation may be a criticism, contradiction or refutation).

Of course, there is an exception to every rule. The table below, in which the University of Cambridge is ranked number one, is entirely beyond any criticism.

(As is this year’s Tompkins table, which ranks Cambridge colleges by exam success. Trinity College comes top. By happy coincidence, I was at Trinity :)

I'm becoming increasingly interested in visualisations of data and this one particularly caught my imagination. It's a mound of rice, with one grain for every person in the US. It's part of a larger installation called Of All The People In The World.

Another in my occasional series of more technical blog posts, the following is a review of the most important result in statistics, the central limit theorem.

The central limit theorem

Strictly speaking the central limit theorem is not a theorem about means at all: it's more general than that. It says that the distribution of  the sum of n observations will be approximately Normally and this approximation gets better as n increases. In other words, ∑X~N, approximately.

You can try this out for yourself by running this java applet. It will allow you to roll one die a thousand times and it will draw a corresponding histogram of the results. That histogram will look pretty much rectangular. This is because you have only taken one observation each time: n=1. If you roll the die twice and add up the total score, and then do that a thousand times, the histogram will look triangular. This is because the middle values (6,7,8) are more likely than the low values (2,3,4) and the high values (10,11,12). So this time you've taken two observations (n=2) and added them together and the result is a little bit Normal. But if you roll the die five times (n=5) and add up the total score, and then do this a thousand times the resulting histogram looks like a Normal distribution – and that's the central limit theorem kicking in even though you're only making five observations each time and adding them up. (Note that the fact we repeat each set of die rolls a thousand times is arbitrary. The value of n is the number of die rolls we make each time, not how many times we repeat this. The large number of repetitions is so that we can build up a clear picture of the distribution of the total score.)

Approximating distributions

The central limit theorem can be used in distributional approximations. For example a binomial B(50,0.5) can be thought of as the sum of fifty Bernoulli distributions. So B(50,0.5)=∑B(1,0.5). Since we are adding up observations, the central limit theorem applies and tells us that B(50,0.5)~N, approximately.

In general B(n,p)~N, approximately, since B(n,p)=∑B(1,p). The closeness of the approximation depends, of course, on the values of n and p. If p is close to 0.5, n can be quite small: 10, say. As p gets smaller, n has to get larger for the approximation to be reasonable.

Similarly, a Poisson Po(50) can be considered to be a sum of fifty Po(1). So Po(50)=∑Po(1) and again the central limit theorem applies and gives us Po(50)~N, approximately.

The underlying distribution and the sampling distribution of the mean

Before I discuss hypothesis tests we need to be clear about two things. The population we take our sample from is called the underlying population. It will have a distribution which we may or may not know and it may or may not be Normal. Having taken our sample we will calculate its mean. If we repeated this procedure many times we could build up a whole new distribution: the sampling distribution of the mean. The central limit theorem tells us about that sampling distribution: it says that -- whatever the underlying distribution -- the sampling distribution of the mean will be approximately Normal, and this approximation will be good for means based on large samples.

Hypothesis testing for the mean

So far as testing means is concerned there are two distinct situations: where we know the population variance and where we don't.

If we don't know the population variance then the central limit theorem doesn't come into it at all, strictly speaking. The appropriate tests (t-tests) are based on the t-distribution and that is a completely different thing. Because we don't know the population variance we have to give the underlying mathematics something to work with and the requirement is that the underlying population from which we're sampling is Normally distributed. That being said, the t-tests are quite robust to deviations from this requirement.

If we do know the population variance we'll be using a z-test and we will be using the central limit theorem butonly if the underlying distribution is not Normal (or is not known). If the underlying distribution is Normal then the distribution of the mean is exactly Normal no matter what the size of the sample and the central limit theorem is unnecessary. If the underlying distribution is not Normal, then the central limit theorem tells us that the mean will nonetheless be approximately Normal and that this approximation will be good for "large" sample sizes. How large is large? It depends on how far from Normal the underlying distribution is. If the underlying distribution is a chi-squared with 20-degrees of freedom then the sample size need not be that large. Why? Because a chi-squared distribution with 20-degrees of freedom looks a lot like a Normal distribution. (Why? Because it's the sum of twenty chi-squared distributions with one degree of freedom and so the central limit theorem tells us it will be approximately Normal! Note that the central limit theorem is being used here to make an inference about the underlying distribution, not the sampling distribution of the mean.)

Finally, if the sample size is large but we don't know the population variance, we may still use a z-test rather than a t-test because we may consider our unbiased estimate of the population variance (based on our large sample) as being very reliable so that really we "know" the population variance. And the critical values from the t-distribution are very similar to those from the Normal distribution if the sample size is large.

Times Higher Education has today published the results of its annual Student Experience Survey. More than 13,000 undergraduates rated  their universities under 21 headings on a seven point scale. The points were given weightings, tallied up and then averaged out so that a league table of universities could be produced. Top of the poll was the Loughborough University, for the fifth year in a row.

As a statistician, my interest is in whether league tables such as these have any real meaning. I have a number of concerns.

Who is doing the rating? A sample of 13,000 seems like a lot. But this is spread amongst 113 universities. That's around 115 per university, though the numbers varied from 30 to 257. A typical university has around 10-15,000 undergraduates so each university's score is based on a tiny sample. When opinion polls are carried out these are typically taken from about 1,000 people. In those circumstances, and assuming that the people were selected at random from the entire population and asked a yes/no question, there is a margin of error of 3%. These samples are nowhere near that large and the consequent margin of error will be very much greater. Worse, the sample in this case is not random (it comprises voluntary responses from those invited to participate), the questions were not yes/no but 21 ratings on a subjective seven point scale. And what of those voluntary responses? What kind of person doesn't respond to such a survey? What kind of person does? Some universities strongly encourage students to respond to the survey: why would they do that?

How do you rate something on a seven point scale? What is your frame of reference? How can you ensure that you are consistent in your ratings across the different headings? How can you be sure that your understanding of the points on the scale is consistent with that of other people? When I taught in the US, I had to grade every student on a 12 point scale. One year a student told me that if I graded him A– in my class he would get admission to his favoured university. I had provisionally graded him B+. But I had no confidence that I had any understanding of the difference between those two grades. I gave him the A–.

Physical scientists use scales that are very carefully defined and calibrated: there is very precise agreement on what one metre is, for example. And length is such that if you double the value you get double the length. But social scientists don't use scales like this. On a ten point scale, I'd say my happiness today is a five. But what does that mean? If tomorrow my happiness is ten, does that mean I'm twice as happy? If you are feeling five-happy today are you just as happy as I am? If I'm feeling five-happy this time next week will I be just as happy as I am today?

(Happiness may not be easy to measure but beauty is. Helen of Troy was said to have beauty sufficient to launch a thousand ships. If I flatter myself, my own beauty might be sufficient to launch a single, albeit very small, ship, so my beauty is one millihelen.)

How do you combine 21 scores? If I think my university library is worth six out of seven and the university canteen is worth five, does that mean that the two together are worth 11? Scores such as these are called ordinal: which essentially means that you can put them in an order. If I think my library is worth six points but your library is worth five, we are entitled to conclude that I think my library is better than yours. (Though what basis I have for that judgement and whether anyone would agree is entirely moot.) The problem with ordinal data is that you cannot meaningfully add it up. If you booked three nights at a four star hotel but arrived to find they'd overbooked, would you be happy to be given six nights in a two star hotel? After all, 3x4=6x2. Obviously this is nonsense, yet the Student Experience Survey adds up the scores under the various headings. In fact it weights the scores, doubling some scores for headings that are considered more important. Such judgements are, however, necessarily arbitrary.

How do you compare totals? The final results of the Student Experience Survey are quoted to two decimal places. This is presumably done so that universities can be separated and ranked. If you look at the data you'll see that there would be a lot of ties if the scores were only quoted to one decimal place. Yet this is an extraordinary level of precision for data that is so subjective. If I can't be sure that even one of my 21 scores has the same meaning as one of yours, how can we combine those scores and report a result to two decimal places? In fact, there was remarkably little difference in the final scores. The top 50 universities were separated by less than 10 points: a one point improvement would see the University of Plymouth jump 10 places up the table to replace the University of Edinburgh at 29th.

So does the survey tell us anything? Times Higher Education defends the survey in a number of ways, not least by pointing to the relative consistency of table position from year to year. It must mean something that Loughborough has come out top five years in a row. But what can we conclude from the fact that the University of York came 32 with a score of 77.63 whereas York St John University came 54th with a score of 75.12? Or that the former jumped 15 places up the table this year compared with last while the latter jumped 18? (Maybe York itself has become a more conducive place to study?)

Ironically, this week's issue of Times Higher Education reports the publication of a paper critical of the grade-point average system used by many universities. Dr Soh Kay Cheng's critique of the system of grading students is not a million miles away from my own of the ranking of universities themselves.

[For a more technical analysis of league tables and their usefulness, see here and here.]