A few days ago, I blogged about the controversy over the BICEP2 result, and the possibility that their measured signal may actually be dominated by contamination from foreground Galactic dust. As Peter Coles’ blog mentions, their paper has now been published in Physical Review Letters. In the abstract to their paper, the BICEP2 team say
Cross correlating BICEP2 against 100 GHz maps from the BICEP1 experiment, the excess signal is confirmed with significance and its spectral index is found to be consistent with that of the CMB, disfavoring dust at .
What does a phrase like “with significance” actually mean? It is the significance with which scientists believe a result to be real as opposed to a random fluctuation in the background signal (the noise). In order to fully understand why scientists quote results to a particular , and what it means in detail, the first step is to understand something called the normal distribution.
You can read more about the BICEP2 result, and how its conclusions were withdrawn, in my book “The Cosmic Microwave Background – How it Changed Our Understanding of the Universe”. Follow this link for more details.
The Normal Distribution
If you have a large number of independent measurements, then their distribution will tend towards something called the normal distribution. This distribution looks like the following, where on the x-axis we have some variable (such as the the background noise in a signal), and the y-axis represents the frequency with which that variable occurs. Normal distributions are usually normalised so that the total probability (the area under the curve) is unity (1), as the sum of all probabilities is always equal to one. The curve is often referred to as a bell curve for obvious reasons.
The mathematical formula for the normal distribution is given by something called the Gaussian function (and so another name for a normal distribution is a “Gaussian distribution”) and has the form
where is the variable, is the mean of the distribution, and is the standard deviation of the distribution. Usually in statistics we have a mean, a median and a mode, but for a normal distribution they are all equal. The standard deviation is related to the width of the curve. For example, in the figure below we show four normal distributions. The blue, red and orange curves all have the same mean (zero), but different standard deviations, which is related to the curve’s width (the diagram actually quotes the variance, which is just the square of the standard deviation). The green curve has a mean of -2 not 0, and it has a different standard deviation to the other three.
As can be seen from these diagrams, if the total probability under each curve is unity, then the probability of a value being measured depends on what the mean is and what the standard deviation is. The further a measurement is from the mean (i.e. towards either end of the bell curve), the less and less likely it is of being measured at random, or to put it another way the less and less likely the signal is of being due to a fluctuation in the background.
So what does a 3-sigma result mean?
We can work out the probability of a particular measurement once we know the mean and the standard deviation of a normal distribution. There are tables to do this, they give the area under the normal distribution function (which remember is related to probability) in terms of a parameter usually written as . Here is an example of such a table.
How do we use this table? The first thing to notice is that the normal distribution is symmetrical about the mean, so the probability from up to the value of the mean is 0.5.
Suppose we have a normal distribution with a mean of and a standard deviation of . How would we use this table to calculate the probability of a value greater or equal to e.g. being real? (that is, any value greater and including 3).
The definition of is
where the modulus in the numerator is so that is always positive. With our example, . So, finding in the table gives the cumulative probability of the value being between and being . So the probability of a value of from .
If we are trying to work out the probability of measuring a value of then we need to remember that the total probability is 1, so the probability of the value of or . Obviously, with our chosen value of , a value of is 2-sigma away from the mean (), so a result quoted as a result (or confidence) means that it has a of being false, and a of being real.
What would we get if we had chosen a value of 1-sigma from the mean, or in other words a value of ? In this case, , and so using our table we find . So the probability of being equal to or greater than 2.5 is or . As you can see, a chance of a result being real (or a chance of a result being false) is not very good, which is why a detection of a signal is not usually considered good enough to be believed.
What would we get if we had chosen a value of 3-sigma from the mean, or in other words a value of ? In this case, , and so using our table we find , so the probability of obtaining a value of equal to or greater than 3.5 is or . So, when we say that a detection is made at the 3-sigma level, what we are saying is that it is certain, or that it has just a probability of being false.
Usually in science, a 3-sigma detection is taken as being the minimum to be believed, and quite often 5-sigma is chosen, which is essentially probability of the result being false.
Summary
The figure below summarises this graphically.
To translate between this figure and what we have calculated above, just note that the percentages to the left of the mean all add up to , so if we wanted to work out the chance of a result being greater than above the mean we would work out , just as we had above. For we have (we got before, the difference is due to rounding).
And, here is a table summarising the significances, to two decimal places.
Confidence that result is real | |
---|---|
84.13% | |
93.32% | |
97.73% | |
99.38% | |
99.87% | |
99.98% | |
100% |
So, going back to the BICEP2 result, they state in their paper that their signal is in excess of the background (noise) signal by , which would mean that their signal is real with a certainty. But, of course, although there seems to be little doubt that their signal is real, what is still undecided and hotly disputed is whether the signal is nearly entirely due to the CMB or could be mainly due to foreground Galactic dust. We shall have to wait to find out the answer to that question!
***UPDATE***
In February 2015 the BICEP2 team withdrew their claim for having discovered primordial B-mode polarisation, and accepted that their detection was of Galactic dust. You can read far more about this fascinating story in my book “The Cosmic Microwave Background – How it Changed Our Understanding of the Universe”.
But these are all probabilities of the data, given the model, right? What one is really interested in is the probability of the model, given the data.
The two are not necessarily the same:
Data: person is pregnant
Model: person is female
Without further information, the probability of the data, given the model, is about 3%. The probability of the model, given the data, is 100%.
Good point!
After 5 years…
Data: person is not pregnant.
Model: preson is male.
What are the probabilities of data and model at 2 sigma?
Probability of the data, given the model, essentially 100 per cent.
Probability of the model, given the data, without further information: Without further information, the probability that a woman is not pregnant for a period of 5 years is about 86 per cent, for a man 100 per cent. So I would say that the probability of the model, given the data, is about 100/186. Imagine you start with 100 men and 100 women—as in the general population. After 5 years, 100 men are not pregnant and 86 women are not pregnant. Do the probability that a random person is a man is about 100/186.
Wonderful description. I was generally aware of the importance of the various sigma probabilities when Higgs was labelled sigma 5. Thanks for making aware of the math behind the categories.
You’re welcome, I’m glad you liked it 🙂
That’s fine, but when the probability distribution develops tails that exceed those of a Gaussian, and we continue to base decisions on the standard error function, then we might well end up running screaming from Wall Street into self-catering potato fields.
I’d love to know more about self-catering potato fields. The potato fields where I grew up in Pembrokeshire (one of the major sources of “new potatoes” in the DUK) were definitely not self catering!
A very good description of the topic. Thanks. The first commenter raises a very interesting point too!
Thanks! Phillip usually does raise interesting points 😛
[…] cosmic microwave background. I have previously blogged about this story, for example here, here and here. But, just to quickly recap, in March the BICEP2 team announced that they had detected the B-mode […]
Heya I don’t know if it’s me or possibly your web blog but it’s launching slow to me, it took me sort of a minute to finally load up still , gmail operates perfectly to
me. However thanks for submitting lovely blog post.
I do believe it really has been incredibly helpful individual who visit here.
I should mention that you actually have done brilliant
job with this plus hope to discover further wonderful content through you.
To obtain more knowledge by posts which you post, I have saved this site.
Thank you for liking my site.
[…] A very high-energy gamma-ray image of RCW 86, a supernova remnant. To understand what 3, 5 or 7 significance means, read my blogpost here. […]
[…] the normal or gaussian distribution. I blogged about that distribution in this blogpost here “What does a 1-sigma, 3-sigma or 5-sigma detection mean?”. The function which describes the normal distribution has the […]
[…] to draw a firm conclusion,” according to a report by CERN. This means that there’s still a 1 in 100 probability that the findings don’t really point to a new physics and are simply the result of […]
\sigma Confidence that result is real
1 \sigma 84.13%
1.5 \sigma 93.32%
2 \sigma 97.73%
2.5 \sigma 99.38%
3 \sigma 99.87%
3.5 \sigma 99.98%
> 4 \sigma 100%
[…] this constant have diverged towards slightly different values. Today this tension level sits at 3.4 sigma; a difference which is statistically significant and therefore cannot be ignored. According to […]
Many thanks for this very clear explanation.
I’m guessing that, “The normal distribution looks life a ‘bell curve’.” was a Freudian-slip, right?
Was it? Explain please.
Typing “looks life” instead of “looks like” could be a typo or it could represent an unintentional error regarded as revealing subconscious feelings related to the existential irony of a measurable error in a blog about accurately measuring inevitable errors. Or perhaps it was an intentional pun made by a mashup of the phrases “Looks like” and “That’s life”…or perhaps, I’m overthinking this. #absurdism
Yes, you’re overthinking it. Simply a typo. Not the first, nor the last
[…] What does a 1-sigma, a 3-sigma or a 5-sigma detection mean?https://thecuriousastronomer.wordpress.com/2014/06/26/what-does-a-1-sigma-3-sigma-or-5-sigma-detecti… […]
[…] ¿Qué significa una detección de 1-sigma, 3-sigma o 5-sigma?https://thecuriousastronomer.wordpress.com/2014/06/26/what-does-a-1-sigma-3-sigma-or-5-sigma-detecti… […]
[…] What does a 1-sigma, a 3-sigma or a 5-sigma detection mean?https://thecuriousastronomer.wordpress.com/2014/06/26/what-does-a-1-sigma-3-sigma-or-5-sigma-detecti… […]
[…] of ‘Oumuamua as it passed through our solar system. This acceleration, detected at 30σ significance, corresponded to a radial (with respect to the sun) acceleration proportional to r-2. This behavior […]
That’s so clear. Thank u very much
[…] What does a 1-sigma, a 3-sigma or a 5-sigma detection mean?https://thecuriousastronomer.wordpress.com/2014/06/26/what-does-a-1-sigma-3-sigma-or-5-sigma-detecti… […]
[…] less than a 0.02% chance that the values only differ due to random statistical uncertainty. (See here for a great explanation of statistical […]
[…] something, you need to be very sure, in fact, a ‘fact’ in science needs to pass the 5 sigma level, (99.9999% sure). To anyone who has ever worked in research, you know the lengths that scientists […]
[…] ) and in the local universe (72.04 ± 2.67 km/s/Mpc). Up to now, this discrepancy was less than 3σ and could still be explained by a statistical fluke. In today’s paper, however, Adam Riess and […]
[…] While the experiment needs to be verified at a higher confidence (the current confidence is 3-sigma), their results could indicate that we all owe our existence to these tiny, ghostly […]
[…] While the experiment needs to be verified at a higher confidence (the current confidence is 3-sigma), their results could indicate that we all owe our existence to these tiny, ghostly […]
[…] look at this observation for what it is. Phosphine was detected at 15-sigma. That means it’s a very, very clear detection. Assuming the astronomers did their due diligence, it looks like phosphine might really be in the […]
[…] look at this observation for what it is. Phosphine was detected at 15-sigma. That means it’s a very, very clear detection. Assuming the astronomers did their due diligence, it looks like phosphine might really be in the […]
[…] look at this observation for what it is. Phosphine was detected at 15-sigma. That means it’s a very, very clear detection. Assuming the astronomers did their due diligence, it looks like phosphine might really be in the […]
Very good explanation very helpful for understanding this…