Feeds:
Posts

## What does a 1-sigma, a 3-sigma or a 5-sigma detection mean?

A few days ago, I blogged about the controversy over the BICEP2 result, and the possibility that their measured signal may actually be dominated by contamination from foreground Galactic dust. As Peter Coles’ blog mentions, their paper has now been published in Physical Review Letters. In the abstract to their paper, the BICEP2 team say

Cross correlating BICEP2 against 100 GHz maps from the BICEP1 experiment, the excess signal is confirmed with $3 \sigma$ significance and its spectral index is found to be consistent with that of the CMB, disfavoring dust at $1.7 \sigma$.

What does a phrase like “with $3 \sigma$ significance” actually mean? It is the significance with which scientists believe a result to be real as opposed to a random fluctuation in the background signal (the noise). In order to fully understand why scientists quote results to a particular $\sigma$, and what it means in detail, the first step is to understand something called the normal distribution.

You can read more about the BICEP2 result, and how its conclusions were withdrawn, in my book “The Cosmic Microwave Background – How it Changed Our Understanding of the Universe”. Follow this link for more details.

## The Normal Distribution

If you have a large number of independent measurements, then their distribution will tend towards something called the normal distribution. This distribution looks like the following, where on the x-axis we have some variable (such as the the background noise in a signal), and the y-axis represents the frequency with which that variable occurs. Normal distributions are usually normalised so that the total probability (the area under the curve) is unity (1), as the sum of all probabilities is always equal to one. The curve is often referred to as a bell curve for obvious reasons.

The mathematical formula for the normal distribution is given by something called the Gaussian function (and so another name for a normal distribution is a “Gaussian distribution”) and has the form
$f(x,\mu,\sigma) = \frac{ 1 }{ \sigma \sqrt{ 2 \pi} } e^{ - \frac{ (x - \mu)^{2} }{ 2 \sigma^{2} } }$

where $x$ is the variable, $\mu$ is the mean of the distribution, and $\sigma$ is the standard deviation of the distribution. Usually in statistics we have a mean, a median and a mode, but for a normal distribution they are all equal. The standard deviation is related to the width of the curve. For example, in the figure below we show four normal distributions. The blue, red and orange curves all have the same mean (zero), but different standard deviations, which is related to the curve’s width (the diagram actually quotes the variance, which is just the square of the standard deviation). The green curve has a mean of -2 not 0, and it has a different standard deviation to the other three.

As can be seen from these diagrams, if the total probability under each curve is unity, then the probability of a value being measured depends on what the mean is and what the standard deviation is. The further a measurement is from the mean (i.e. towards either end of the bell curve), the less and less likely it is of being measured at random, or to put it another way the less and less likely the signal is of being due to a fluctuation in the background.

## So what does a 3-sigma result mean?

We can work out the probability of a particular measurement once we know the mean and the standard deviation of a normal distribution. There are tables to do this, they give the area under the normal distribution function (which remember is related to probability) in terms of a parameter usually written as $Z$. Here is an example of such a table.

How do we use this table? The first thing to notice is that the normal distribution is symmetrical about the mean, so the probability from $-\infty$ up to the value of the mean is 0.5.

Suppose we have a normal distribution with a mean of $\mu = 2$ and a standard deviation of $\sigma = 0.5$. How would we use this table to calculate the probability of a value greater or equal to e.g. $3$ being real? (that is, any value greater and including 3).

The definition of $Z$ is

$Z = \frac{ | x - \mu | }{ \sigma }$

where the modulus in the numerator is so that $Z$ is always positive. With our example, $Z = (3 - 2)/0.5 = 2.0$. So, finding $Z = 2.0$ in the table gives the cumulative probability $P(Z)$ of the value $x$ being between $-\infty$ and $2$ being $P(Z=2) = 0.9772$. So the probability of a value of $x$ from $-\infty \text{ to } 3 \text{ is } 0.9772 \text{ or } 97.72 \%$.

If we are trying to work out the probability of measuring a value of $x > 3$ then we need to remember that the total probability is 1, so the probability of the value of $x > 3 \text{ is } 1-0.9772 = 0.0228$ or $2.28 \%$. Obviously, with our chosen value of $\sigma = 0.5$, a value of $x=3$ is 2-sigma away from the mean ($Z=2$), so a result quoted as a $2 \sigma$ result (or confidence) means that it has a $2.28 \%$ of being false, and a $97.72 \%$ of being real.

What would we get if we had chosen a value of 1-sigma from the mean, or in other words a value of $x = 2.5$? In this case, $Z = (2.5 - 2)/0.5 = 1$, and so using our table we find $P(Z) = 0.8413$. So the probability of $x$ being equal to or greater than 2.5 is $1 - 0.8413 = 0.1587$ or $15.87\%$. As you can see, a $84.13\%$ chance of a result being real (or a $15.87 \%$ chance of a result being false) is not very good, which is why a $1 \sigma$ detection of a signal is not usually considered good enough to be believed.

What would we get if we had chosen a value of 3-sigma from the mean, or in other words a value of $x = 3.5$? In this case, $Z = (3.5 - 2)/0.5 = 3$, and so using our table we find $P(Z=3) = 0.9987$, so the probability of obtaining a value of equal to or greater than 3.5 is $1 - 0.9987 = 0.0013$ or $0.13\%$. So, when we say that a detection is made at the 3-sigma level, what we are saying is that it is $99.87\%$ certain, or that it has just a $0.13\%$ probability of being false.

Usually in science, a 3-sigma detection is taken as being the minimum to be believed, and quite often 5-sigma is chosen, which is essentially $0\%$ probability of the result being false.

## Summary

The figure below summarises this graphically.

To translate between this figure and what we have calculated above, just note that the percentages to the left of the mean all add up to $50\%$, so if we wanted to work out the chance of a result being greater than $1\sigma$ above the mean we would work out $100\% - (50\% + 34.1\%) = 15.9\%$, just as we had above. For $3 \sigma$ we have $100\% - (50\% + 34.1\% + 13.6\% + 2.1\%) = 0.2\%$ (we got $0.13\%$ before, the difference is due to rounding).

And, here is a table summarising the significances, to two decimal places.

The significance of various levels of $\sigma$
$\sigma$ Confidence that result is real
$1 \sigma$ 84.13%
$1.5 \sigma$ 93.32%
$2 \sigma$ 97.73%
$2.5 \sigma$ 99.38%
$3 \sigma$ 99.87%
$3.5 \sigma$ 99.98%
$> 4 \sigma$ 100%

So, going back to the BICEP2 result, they state in their paper that their signal is in excess of the background (noise) signal by $3 \sigma$, which would mean that their signal is real with a $99.87\%$ certainty. But, of course, although there seems to be little doubt that their signal is real, what is still undecided and hotly disputed is whether the signal is nearly entirely due to the CMB or could be mainly due to foreground Galactic dust. We shall have to wait to find out the answer to that question!

## ***UPDATE***

In February 2015 the BICEP2 team withdrew their claim for having discovered primordial B-mode polarisation, and accepted that their detection was of Galactic dust. You can read far more about this fascinating story in my book “The Cosmic Microwave Background – How it Changed Our Understanding of the Universe”.

My book “The Cosmic Microwave Background – how it changed our understanding of the Universe” is published by Springer. Follow this link for more details.