A few days ago, I blogged about the controversy over the BICEP2 result, and the possibility that their measured signal may actually be dominated by contamination from foreground Galactic dust. As Peter Coles’ blog mentions, their paper has now been published in *Physical Review Letters*. In the abstract to their paper, the BICEP2 team say

Cross correlating BICEP2 against 100 GHz maps from the BICEP1 experiment, the excess signal is confirmed with significance and its spectral index is found to be consistent with that of the CMB, disfavoring dust at .

What does a phrase like *“with significance”* actually mean? It is the significance with which scientists believe a result to be real as opposed to a random fluctuation in the background signal (the noise). In order to fully understand why scientists quote results to a particular , and what it means in detail, the first step is to understand something called the *normal distribution*.

You can read more about the BICEP2 result, and how its conclusions were withdrawn, in my book *“The Cosmic Microwave Background – How it Changed Our Understanding of the Universe”*. Follow this link for more details.

## The Normal Distribution

If you have a large number of independent measurements, then their distribution will tend towards something called the *normal distribution*. This distribution looks like the following, where on the x-axis we have some variable (such as the the background noise in a signal), and the y-axis represents the frequency with which that variable occurs. Normal distributions are usually normalised so that the total probability (the area under the curve) is unity (1), as the sum of all probabilities is always equal to one. The curve is often referred to as a *bell curve* for obvious reasons.

The mathematical formula for the normal distribution is given by something called the *Gaussian function* (and so another name for a normal distribution is a *“Gaussian distribution”*) and has the form

where is the variable, is the mean of the distribution, and is the standard deviation of the distribution. Usually in statistics we have a mean, a median and a mode, but for a normal distribution they are all equal. The standard deviation is related to the width of the curve. For example, in the figure below we show four normal distributions. The blue, red and orange curves all have the same mean (zero), but different standard deviations, which is related to the curve’s width (the diagram actually quotes the *variance*, which is just the square of the standard deviation). The green curve has a mean of -2 not 0, and it has a different standard deviation to the other three.

As can be seen from these diagrams, if the total probability under each curve is unity, then the probability of a value being measured depends on what the mean is and what the standard deviation is. The further a measurement is from the mean (i.e. towards either end of the bell curve), the less and less likely it is of being measured at random, or to put it another way the less and less likely the signal is of being due to a fluctuation in the background.

## So what does a 3-sigma result mean?

We can work out the probability of a particular measurement once we know the mean and the standard deviation of a normal distribution. There are tables to do this, they give the area under the normal distribution function (which remember is related to probability) in terms of a parameter usually written as . Here is an example of such a table.

How do we use this table? The first thing to notice is that the normal distribution is symmetrical about the mean, so the probability from up to the value of the mean is 0.5.

Suppose we have a normal distribution with a mean of and a standard deviation of . How would we use this table to calculate the probability of a value greater or equal to e.g. being real? (that is, any value greater and including 3).

The definition of is

where the modulus in the numerator is so that is always positive. With our example, . So, finding in the table gives the cumulative probability of the value being between and being . So the probability of a value of from .

If we are trying to work out the probability of measuring a value of then we need to remember that the *total* probability is 1, so the probability of the value of or . Obviously, with our chosen value of , a value of is 2-sigma away from the mean (), so a result quoted as a result (or confidence) means that it has a of being false, and a of being real.

What would we get if we had chosen a value of 1-sigma from the mean, or in other words a value of ? In this case, , and so using our table we find . So the probability of being *equal to or greater than* 2.5 is or . As you can see, a chance of a result being real (or a chance of a result being false) is not very good, which is why a detection of a signal is not usually considered good enough to be believed.

What would we get if we had chosen a value of 3-sigma from the mean, or in other words a value of ? In this case, , and so using our table we find , so the probability of obtaining a value of *equal to or greater than* 3.5 is or . So, when we say that a detection is made at the 3-sigma level, what we are saying is that it is certain, or that it has just a probability of being false.

Usually in science, a 3-sigma detection is taken as being the minimum to be believed, and quite often 5-sigma is chosen, which is essentially probability of the result being false.

## Summary

The figure below summarises this graphically.

To translate between this figure and what we have calculated above, just note that the percentages to the left of the mean all add up to , so if we wanted to work out the chance of a result being greater than above the mean we would work out , just as we had above. For we have (we got before, the difference is due to rounding).

And, here is a table summarising the significances, to two decimal places.

Confidence that result is real | |
---|---|

84.13% | |

93.32% | |

97.73% | |

99.38% | |

99.87% | |

99.98% | |

100% |

So, going back to the BICEP2 result, they state in their paper that their signal is in excess of the background (noise) signal by , which would mean that their signal is real with a certainty. But, of course, although there seems to be little doubt that their signal is real, what is still undecided and hotly disputed is whether the signal is nearly entirely due to the CMB or could be mainly due to foreground Galactic dust. We shall have to wait to find out the answer to that question!

## ***UPDATE***

In February 2015 the BICEP2 team withdrew their claim for having discovered primordial B-mode polarisation, and accepted that their detection was of Galactic dust. You can read far more about this fascinating story in my book *“The Cosmic Microwave Background – How it Changed Our Understanding of the Universe”*.

on 26/06/2014 at 10:02 |Phillip HelbigBut these are all probabilities of the data, given the model, right? What one is really interested in is the probability of the model, given the data.

The two are not necessarily the same:

Data: person is pregnant

Model: person is female

Without further information, the probability of the data, given the model, is about 3%. The probability of the model, given the data, is 100%.

on 26/06/2014 at 10:25 |RhEvansGood point!

on 26/06/2014 at 14:27 |ianlibWonderful description. I was generally aware of the importance of the various sigma probabilities when Higgs was labelled sigma 5. Thanks for making aware of the math behind the categories.

on 26/06/2014 at 15:29 |RhEvansYou’re welcome, I’m glad you liked it 🙂

on 27/06/2014 at 08:35 |Andrew BlainThat’s fine, but when the probability distribution develops tails that exceed those of a Gaussian, and we continue to base decisions on the standard error function, then we might well end up running screaming from Wall Street into self-catering potato fields.

on 27/06/2014 at 20:46 |RhEvansI’d love to know more about self-catering potato fields. The potato fields where I grew up in Pembrokeshire (one of the major sources of “new potatoes” in the DUK) were definitely not self catering!

on 27/06/2014 at 20:29 |Matthew DoreyA very good description of the topic. Thanks. The first commenter raises a very interesting point too!

on 27/06/2014 at 20:34 |RhEvansThanks! Phillip usually does raise interesting points 😛

on 10/07/2014 at 07:00 |BICEP2 and Planck to share data | thecuriousastronomer[…] cosmic microwave background. I have previously blogged about this story, for example here, here and here. But, just to quickly recap, in March the BICEP2 team announced that they had detected the B-mode […]

on 10/07/2015 at 05:34 |dota2hack.orgHeya I don’t know if it’s me or possibly your web blog but it’s launching slow to me, it took me sort of a minute to finally load up still , gmail operates perfectly to

me. However thanks for submitting lovely blog post.

I do believe it really has been incredibly helpful individual who visit here.

I should mention that you actually have done brilliant

job with this plus hope to discover further wonderful content through you.

To obtain more knowledge by posts which you post, I have saved this site.

on 10/07/2015 at 05:36 |RhEvansThank you for liking my site.

on 01/03/2016 at 03:30 |Looking for cosmic and gamma rays in Namibia | thecuriousastronomer[…] A very high-energy gamma-ray image of RCW 86, a supernova remnant. To understand what 3, 5 or 7 significance means, read my blogpost here. […]

on 05/07/2016 at 07:31 |Euler’s Number (the mathematical constant ‘e’) | thecuriousastronomer[…] the normal or gaussian distribution. I blogged about that distribution in this blogpost here “What does a 1-sigma, 3-sigma or 5-sigma detection mean?”. The function which describes the normal distribution has the […]

on 19/04/2017 at 22:45 |A New Discovery by the LHC Hints at Physics Beyond the Standard Model[…] to draw a firm conclusion,” according to a report by CERN. This means that there’s still a 1 in 100 probability that the findings don’t really point to a new physics and are simply the result of […]

on 19/03/2018 at 21:47 |royalringball\sigma Confidence that result is real

1 \sigma 84.13%

1.5 \sigma 93.32%

2 \sigma 97.73%

2.5 \sigma 99.38%

3 \sigma 99.87%

3.5 \sigma 99.98%

> 4 \sigma 100%

on 08/01/2018 at 21:32 |Meet the AAS Keynote Speakers: Adam Riess | astrobites[…] this constant have diverged towards slightly different values. Today this tension level sits at 3.4 sigma; a difference which is statistically significant and therefore cannot be ignored. According to […]

on 05/04/2018 at 19:34 |SihemMany thanks for this very clear explanation.