Feeds:
Posts

## What does a 1-sigma, a 3-sigma or a 5-sigma detection mean?

A few days ago, I blogged about the controversy over the BICEP2 result, and the possibility that their measured signal may actually be dominated by contamination from foreground Galactic dust. As Peter Coles’ blog mentions, their paper has now been published in Physical Review Letters. In the abstract to their paper, the BICEP2 team say

Cross correlating BICEP2 against 100 GHz maps from the BICEP1 experiment, the excess signal is confirmed with $3 \sigma$ significance and its spectral index is found to be consistent with that of the CMB, disfavoring dust at $1.7 \sigma$.

What does a phrase like “with $3 \sigma$ significance” actually mean? It is the significance with which scientists believe a result to be real as opposed to a random fluctuation in the background signal (the noise). In order to fully understand why scientists quote results to a particular $\sigma$, and what it means in detail, the first step is to understand something called the normal distribution.

You can read more about the BICEP2 result, and how its conclusions were withdrawn, in my book “The Cosmic Microwave Background – How it Changed Our Understanding of the Universe”. Follow this link for more details. ## The Normal Distribution

If you have a large number of independent measurements, then their distribution will tend towards something called the normal distribution. This distribution looks like the following, where on the x-axis we have some variable (such as the the background noise in a signal), and the y-axis represents the frequency with which that variable occurs. Normal distributions are usually normalised so that the total probability (the area under the curve) is unity (1), as the sum of all probabilities is always equal to one. The curve is often referred to as a bell curve for obvious reasons.

The mathematical formula for the normal distribution is given by something called the Gaussian function (and so another name for a normal distribution is a “Gaussian distribution”) and has the form $f(x,\mu,\sigma) = \frac{ 1 }{ \sigma \sqrt{ 2 \pi} } e^{ - \frac{ (x - \mu)^{2} }{ 2 \sigma^{2} } }$

where $x$ is the variable, $\mu$ is the mean of the distribution, and $\sigma$ is the standard deviation of the distribution. Usually in statistics we have a mean, a median and a mode, but for a normal distribution they are all equal. The standard deviation is related to the width of the curve. For example, in the figure below we show four normal distributions. The blue, red and orange curves all have the same mean (zero), but different standard deviations, which is related to the curve’s width (the diagram actually quotes the variance, which is just the square of the standard deviation). The green curve has a mean of -2 not 0, and it has a different standard deviation to the other three.

As can be seen from these diagrams, if the total probability under each curve is unity, then the probability of a value being measured depends on what the mean is and what the standard deviation is. The further a measurement is from the mean (i.e. towards either end of the bell curve), the less and less likely it is of being measured at random, or to put it another way the less and less likely the signal is of being due to a fluctuation in the background.

## So what does a 3-sigma result mean?

We can work out the probability of a particular measurement once we know the mean and the standard deviation of a normal distribution. There are tables to do this, they give the area under the normal distribution function (which remember is related to probability) in terms of a parameter usually written as $Z$. Here is an example of such a table.

How do we use this table? The first thing to notice is that the normal distribution is symmetrical about the mean, so the probability from $-\infty$ up to the value of the mean is 0.5.

Suppose we have a normal distribution with a mean of $\mu = 2$ and a standard deviation of $\sigma = 0.5$. How would we use this table to calculate the probability of a value greater or equal to e.g. $3$ being real? (that is, any value greater and including 3).

The definition of $Z$ is $Z = \frac{ | x - \mu | }{ \sigma }$

where the modulus in the numerator is so that $Z$ is always positive. With our example, $Z = (3 - 2)/0.5 = 2.0$. So, finding $Z = 2.0$ in the table gives the cumulative probability $P(Z)$ of the value $x$ being between $-\infty$ and $2$ being $P(Z=2) = 0.9772$. So the probability of a value of $x$ from $-\infty \text{ to } 3 \text{ is } 0.9772 \text{ or } 97.72 \%$.

If we are trying to work out the probability of measuring a value of $x > 3$ then we need to remember that the total probability is 1, so the probability of the value of $x > 3 \text{ is } 1-0.9772 = 0.0228$ or $2.28 \%$. Obviously, with our chosen value of $\sigma = 0.5$, a value of $x=3$ is 2-sigma away from the mean ( $Z=2$), so a result quoted as a $2 \sigma$ result (or confidence) means that it has a $2.28 \%$ of being false, and a $97.72 \%$ of being real.

What would we get if we had chosen a value of 1-sigma from the mean, or in other words a value of $x = 2.5$? In this case, $Z = (2.5 - 2)/0.5 = 1$, and so using our table we find $P(Z) = 0.8413$. So the probability of $x$ being equal to or greater than 2.5 is $1 - 0.8413 = 0.1587$ or $15.87\%$. As you can see, a $84.13\%$ chance of a result being real (or a $15.87 \%$ chance of a result being false) is not very good, which is why a $1 \sigma$ detection of a signal is not usually considered good enough to be believed.

What would we get if we had chosen a value of 3-sigma from the mean, or in other words a value of $x = 3.5$? In this case, $Z = (3.5 - 2)/0.5 = 3$, and so using our table we find $P(Z=3) = 0.9987$, so the probability of obtaining a value of equal to or greater than 3.5 is $1 - 0.9987 = 0.0013$ or $0.13\%$. So, when we say that a detection is made at the 3-sigma level, what we are saying is that it is $99.87\%$ certain, or that it has just a $0.13\%$ probability of being false.

Usually in science, a 3-sigma detection is taken as being the minimum to be believed, and quite often 5-sigma is chosen, which is essentially $0\%$ probability of the result being false.

## Summary

The figure below summarises this graphically.

To translate between this figure and what we have calculated above, just note that the percentages to the left of the mean all add up to $50\%$, so if we wanted to work out the chance of a result being greater than $1\sigma$ above the mean we would work out $100\% - (50\% + 34.1\%) = 15.9\%$, just as we had above. For $3 \sigma$ we have $100\% - (50\% + 34.1\% + 13.6\% + 2.1\%) = 0.2\%$ (we got $0.13\%$ before, the difference is due to rounding).

And, here is a table summarising the significances, to two decimal places.

The significance of various levels of $\sigma$ $\sigma$ Confidence that result is real $1 \sigma$ 84.13% $1.5 \sigma$ 93.32% $2 \sigma$ 97.73% $2.5 \sigma$ 99.38% $3 \sigma$ 99.87% $3.5 \sigma$ 99.98% $> 4 \sigma$ 100%

So, going back to the BICEP2 result, they state in their paper that their signal is in excess of the background (noise) signal by $3 \sigma$, which would mean that their signal is real with a $99.87\%$ certainty. But, of course, although there seems to be little doubt that their signal is real, what is still undecided and hotly disputed is whether the signal is nearly entirely due to the CMB or could be mainly due to foreground Galactic dust. We shall have to wait to find out the answer to that question!

## ***UPDATE***

In February 2015 the BICEP2 team withdrew their claim for having discovered primordial B-mode polarisation, and accepted that their detection was of Galactic dust. You can read far more about this fascinating story in my book “The Cosmic Microwave Background – How it Changed Our Understanding of the Universe”. My book “The Cosmic Microwave Background – how it changed our understanding of the Universe” is published by Springer. Follow this link for more details.

### 30 Responses

1. on 26/06/2014 at 10:02 | Reply Phillip Helbig

But these are all probabilities of the data, given the model, right? What one is really interested in is the probability of the model, given the data.

The two are not necessarily the same:
Data: person is pregnant
Model: person is female
Without further information, the probability of the data, given the model, is about 3%. The probability of the model, given the data, is 100%.

• on 26/06/2014 at 10:25 | Reply RhEvans

Good point!

• on 31/01/2019 at 14:21 | Reply Stefan Neagu

After 5 years…
Data: person is not pregnant.
Model: preson is male.
What are the probabilities of data and model at 2 sigma?

• on 08/03/2019 at 11:27 Phillip Helbig

Probability of the data, given the model, essentially 100 per cent.

Probability of the model, given the data, without further information: Without further information, the probability that a woman is not pregnant for a period of 5 years is about 86 per cent, for a man 100 per cent. So I would say that the probability of the model, given the data, is about 100/186. Imagine you start with 100 men and 100 women—as in the general population. After 5 years, 100 men are not pregnant and 86 women are not pregnant. Do the probability that a random person is a man is about 100/186.

2. on 26/06/2014 at 14:27 | Reply ianlib

Wonderful description. I was generally aware of the importance of the various sigma probabilities when Higgs was labelled sigma 5. Thanks for making aware of the math behind the categories.

• on 26/06/2014 at 15:29 | Reply RhEvans

You’re welcome, I’m glad you liked it 🙂

3. on 27/06/2014 at 08:35 | Reply Andrew Blain

That’s fine, but when the probability distribution develops tails that exceed those of a Gaussian, and we continue to base decisions on the standard error function, then we might well end up running screaming from Wall Street into self-catering potato fields.

• on 27/06/2014 at 20:46 | Reply RhEvans

I’d love to know more about self-catering potato fields. The potato fields where I grew up in Pembrokeshire (one of the major sources of “new potatoes” in the DUK) were definitely not self catering!

4. on 27/06/2014 at 20:29 | Reply Matthew Dorey

A very good description of the topic. Thanks. The first commenter raises a very interesting point too!

• on 27/06/2014 at 20:34 | Reply RhEvans

Thanks! Phillip usually does raise interesting points 😛

5. […] cosmic microwave background. I have previously blogged about this story, for example here, here and here. But, just to quickly recap, in March the BICEP2 team announced that they had detected the B-mode […]

6. on 10/07/2015 at 05:34 | Reply dota2hack.org

Heya I don’t know if it’s me or possibly your web blog but it’s launching slow to me, it took me sort of a minute to finally load up still , gmail operates perfectly to
me. However thanks for submitting lovely blog post.
I do believe it really has been incredibly helpful individual who visit here.
I should mention that you actually have done brilliant
job with this plus hope to discover further wonderful content through you.
To obtain more knowledge by posts which you post, I have saved this site.

• on 10/07/2015 at 05:36 | Reply RhEvans

Thank you for liking my site.

7. […] A very high-energy gamma-ray image of RCW 86, a supernova remnant. To understand what 3, 5 or 7 significance means, read my blogpost here. […]

8. […] the normal or gaussian distribution. I blogged about that distribution in this blogpost here “What does a 1-sigma, 3-sigma or 5-sigma detection mean?”. The function which describes the normal distribution has the […]

9. […] to draw a firm conclusion,” according to a report by CERN. This means that there’s still a 1 in 100 probability that the findings don’t really point to a new physics and are simply the result of […]

• on 19/03/2018 at 21:47 | Reply royalringball

\sigma Confidence that result is real
1 \sigma 84.13%
1.5 \sigma 93.32%
2 \sigma 97.73%
2.5 \sigma 99.38%
3 \sigma 99.87%
3.5 \sigma 99.98%
> 4 \sigma 100%

10. […] this constant have diverged towards slightly different values. Today this tension level sits at 3.4 sigma; a difference which is statistically significant and therefore cannot be ignored. According to […]

11. on 05/04/2018 at 19:34 | Reply Sihem

Many thanks for this very clear explanation.

12. on 22/08/2018 at 14:19 | Reply Mr. Science

I’m guessing that, “The normal distribution looks life a ‘bell curve’.” was a Freudian-slip, right?

• on 22/08/2018 at 16:02 | Reply RhEvans

• on 27/08/2018 at 17:54 Mr. Science

Typing “looks life” instead of “looks like” could be a typo or it could represent an unintentional error regarded as revealing subconscious feelings related to the existential irony of a measurable error in a blog about accurately measuring inevitable errors. Or perhaps it was an intentional pun made by a mashup of the phrases “Looks like” and “That’s life”…or perhaps, I’m overthinking this. #absurdism

• on 27/08/2018 at 18:31 RhEvans

Yes, you’re overthinking it. Simply a typo. Not the first, nor the last

13. […] What does a 1-sigma, a 3-sigma or a 5-sigma detection mean?https://thecuriousastronomer.wordpress.com/2014/06/26/what-does-a-1-sigma-3-sigma-or-5-sigma-detecti&#8230; […]

14. […] ¿Qué significa una detección de 1-sigma, 3-sigma o 5-sigma?https://thecuriousastronomer.wordpress.com/2014/06/26/what-does-a-1-sigma-3-sigma-or-5-sigma-detecti&#8230; […]

15. […] What does a 1-sigma, a 3-sigma or a 5-sigma detection mean?https://thecuriousastronomer.wordpress.com/2014/06/26/what-does-a-1-sigma-3-sigma-or-5-sigma-detecti&#8230; […]

16. […] of ‘Oumuamua as it passed through our solar system. This acceleration, detected at 30σ significance, corresponded to a radial (with respect to the sun) acceleration proportional to r-2. This behavior […]

17. on 13/11/2018 at 07:19 | Reply Puma

That’s so clear. Thank u very much

18. […] What does a 1-sigma, a 3-sigma or a 5-sigma detection mean?https://thecuriousastronomer.wordpress.com/2014/06/26/what-does-a-1-sigma-3-sigma-or-5-sigma-detecti&#8230; […]

19. […] less than a 0.02% chance that the values only differ due to random statistical uncertainty. (See here for a great explanation of statistical […]