Not all implementations of statistical tests return p-values. In some cases, you must use alternatives, such as critical values. In addition, critical values are used when estimating the expected intervals for observations from a population, such as in tolerance intervals.
In this tutorial, you will discover critical values, why they are important, how they are used, and how to calculate them in Python using SciPy.
After completing this tutorial, you will know:
Examples of statistical hypothesis tests and their distributions from which critical values can be calculated and used.
How exactly critical values are used on one-tail and two-tail statistical hypothesis tests.
How to calculate critical values for the Gaussian, Student’s t, and Chi-Squared distributions.
Kick-start your project with my new book Statistics for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.
Let’s get started.
A Gentle Introduction to Critical Values for Statistical Hypothesis Testing Photo by Steve Bittinger, some rights reserved.
Tutorial Overview
This tutorial is divided into 4 parts; they are:
Why Do We Need Critical Values?
What Is a Critical Value?
How to Use Critical Values
How to Calculate Critical Values
Need help with Statistics for Machine Learning?
Take my free 7-day email crash course now (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
Why Do We Need Critical Values?
Many statistical hypothesis tests return a p-value that is used to interpret the outcome of the test.
Some tests do not return a p-value, requiring an alternative method for interpreting the calculated test statistic directly.
A statistic calculated by a statistical hypothesis test can be interpreted using critical values from the distribution of the test statistic.
Some examples of statistical hypothesis tests and their distributions from which critical values can be calculated are as follows:
Z-Test: Gaussian distribution.
Student t-Test: Student’s t-distribution.
Chi-Squared Test: Chi-Squared distribution.
ANOVA: F-distribution.
Critical values are also used when defining intervals for expected (or unexpected) observations in distributions. Calculating and using critical values may be appropriate when quantifying the uncertainty of estimated statistics or intervals such as confidence intervals and tolerance intervals.
What Is a Critical Value?
A critical value is defined in the context of the population distribution and a probability.
An observation from the population with a value equal to or lesser than a critical value with the given probability.
We can express this mathematically as follows:
1
Pr[X <= critical value] = probability
Where Pr is the calculation of probability, X are observations from the population, critica_value is the calculated critical value, and probability is the chosen probability.
Critical values are calculated using a mathematical function where the probability is provided as an argument. For most common distributions, the value cannot be calculated analytically; instead it must be estimated using numerical methods. Historically it is common for tables of pre-calculated critical values to be provided in the appendices of statistics textbooks for reference purposes.
Critical values are used in statistical significance testing. The probability is often expressed as a significance, denoted as the lowercase Greek letter alpha (a), which is the inverted probability.
1
probability = 1 - alpha
Standard alpha values are used when calculating critical values, chosen for historical reasons and continually used for consistency reasons. These alpha values include:
1% (alpha=0.01)
5% (alpha=0.05)
10% (alpha=0.10)
Critical values provide an alternative and equivalent way to interpret statistical hypothesis tests to the p-value.
How to Use Critical Values
Calculated critical values are used as a threshold for interpreting the result of a statistical test.
The observation values in the population beyond the critical value are often called the “critical region” or the “region of rejection“.
Critical Value: A value appearing in tables for specified statistical tests indicating at what computed value the null hypothesis can be rejected (the computed statistic falls in the rejection region).
A one-tailed test has a single critical value, such as on the left or the right of the distribution.
Often, a one-tailed test has a critical value on the right of the distribution for non-symmetrical distributions (such as the Chi-Squared distribution).
The statistic is compared to the calculated critical value. If the statistic is less than or equal to the critical value, we fail to reject the null hypothesis (e.g. no effect). Otherwise it is rejected.
We can summarize this interpretation as follows:
Test Statistic <= Critical Value: Fail to reject the null hypothesis of the statistical test.
Test Statistic > Critical Value: Reject the null hypothesis of the statistical test.
Two-Tailed Test
A two-tailed test has two critical values, one on each side of the distribution, which is often assumed to be symmetrical (e.g. Gaussian and Student-t distributions.).
When using a two-tailed test, a significance level (or alpha) used in the calculation of the critical values must be divided by 2. The critical value will then use a portion of this alpha on each side of the distribution.
To make this concrete, consider an alpha of 5%. This would be split to give two alpha values of 2.5% on either side of the distribution with an acceptance area in the middle of the distribution of 95%.
We can refer to each critical value as the lower and upper critical values for the left and right of the distribution respectively. Test statistic values more than or equal to the lower critical value and less than or equal to the upper critical value indicate the failure to reject the null hypothesis. Whereas test statistic values less than the lower critical value and more than the upper critical value indicate rejection of the null hypothesis for the test.
We can summarize this interpretation as follows:
Lower CR <= Test Statistic <= Upper CR: Failure to reject the null hypothesis of the statistical test.
Test Statistic < Lower CR OR Test Statistic > Upper CR: Reject the null hypothesis of the statistical test.
If the distribution of the test statistic is symmetric around a mean of zero, then we can shortcut the check by comparing the absolute (positive) value of the test statistic to the upper critical value.
|Test Statistic| <= Upper Critical Value: Failure to reject the null hypothesis of the statistical test.
Where |Test Statistic| is the absolute value of the calculated test statistic.
How to Calculate Critical Values
Density functions return the probability of an observation in the distribution. Recall the definitions of the PDF and CDF as follows:
Probability Density Function (PDF): Returns the probability for an observation having a specific value from the distribution.
Cumulative Density Function (CDF): Returns the probability for an observation equal to or lesser than a specific value from the distribution.
In order to calculate a critical value, we require a function that, given a probability (or significance), will return the observation value from the distribution.
Specifically, we require the inverse of the cumulative density function, where given a probability, we are given the observation value that is less than or equal to the probability. This is called the percent point function (PPF), or more generally the quantile function.
Percent Point Function (PPF): Returns the observation value for the provided probability that is less than or equal to the provided probability from the distribution.
Specifically, a value from the distribution will equal or be less than the value returned from the PPF with the specified probability.
Let’s make this concrete with three distributions from which it is commonly required to calculate critical values. Namely, the Gaussian distribution, Student’s t-distribution, and the Chi-squared distribution.
We can calculate the percent point function in SciPy using the ppf() function on a given distribution. It should also be noted that you can also calculate the ppf() using the inverse survival function called isf() in SciPy. This is mentioned as you may see use of this alternate approach in third party code.
Gaussian Critical Values
The example below calculates the percent point function for 95% on the standard Gaussian distribution.
1
2
3
4
5
6
7
8
9
10
# Gaussian Percent Point Function
from scipy.stats import norm
# define probability
p=0.95
# retrieve value <= probability
value=norm.ppf(p)
print(value)
# confirm with cdf
p=norm.cdf(value)
print(p)
Running the example first prints the value that marks 95% or less of the observations from the distribution of about 1.65. This value is then confirmed by retrieving the probability of the observation from the CDF, which returns 95%, as expected.
We can see that the value 1.65 aligns with our expectation with regard to the number of standard deviations from the mean that cover 95% of the distribution in the 68–95–99.7 rule.
1
2
1.6448536269514722
0.95
Student’s t Critical Values
The example below calculates the percentage point function for 95% on the standard Student’s t-distribution with 10 degrees of freedom.
1
2
3
4
5
6
7
8
9
10
11
# Student t-distribution Percent Point Function
from scipy.stats importt
# define probability
p=0.95
df=10
# retrieve value <= probability
value=t.ppf(p,df)
print(value)
# confirm with cdf
p=t.cdf(value,df)
print(p)
Running the example returns the value of about 1.812 or less that covers 95% of the observations from the chosen distribution. The probability of the value is then confirmed (with minor rounding error) via the CDF.
1
2
1.8124611228107335
0.949999999999923
Chi-squared Critical Values
The example below calculates the percentage point function for 95% on the standard Chi-Squared distribution with 10 degrees of freedom.
1
2
3
4
5
6
7
8
9
10
11
# Chi-Squared Percent Point Function
from scipy.stats import chi2
# define probability
p=0.95
df=10
# retrieve value <= probability
value=chi2.ppf(p,df)
print(value)
# confirm with cdf
p=chi2.cdf(value,df)
print(p)
Running the example first calculates the value of 18.3 or less that covers 95% of the observations from the distribution. The probability of this observation is confirmed by using it as input to the CDF.
1
2
18.307038053275143
0.95
Further Reading
This section provides more resources on the topic if you are looking to go deeper.
the numbers in Alex’s image is for two-tailed test.
When we calculate alpha for two-tail we divide it, so alpha = 0.05 means p=0.025 and 0.975,
So your probability is between both ends of the distribution:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
from scipy.stats importt
# define probability
p=0.025
df=10
# retrieve value <= probability
value=t.ppf(p,df)
print(value)
# confirm with cdf
p=t.cdf(value,df)
print(p)
p=0.975
df=10
# retrieve value <= probability
value=t.ppf(p,df)
print(value)
# confirm with cdf
p=t.cdf(value,df)
print(p)
if run this code, you can see that both values are symmetrical.
Jason, I believe that it would make your examples much clearer if you were to specify that all the ppf functions return a one-tailed critical value. That is why you can put it directly back in the CDF function and return the same percentage you started with. I’m sure you know this, but it would help clarify your examples that it is a *one-sided* 95% confidence interval of the standard normal, student-t, and chi-squared distributions.
Sorry to bother you but i think there is an error in the 1 Tailed Distribution section. You say in text that:
“The statistic is compared to the calculated critical value. If the statistic is less than or equal to the critical value, the null hypothesis of the statistical test is failed to be rejected. Otherwise it is rejected.”
but in the summary of that same section you say:
Test Statistic Critical Value: Reject the null hypothesis of the statistical test.
which is it? Should the test statistic be greater than or greater than or equal to the critical value for the null hypothesis to be rejected?
Hi! I’ve got a question. Look, you told that
p = 0.95
df = 10
value = t.ppf(p, df)
returns the Student’s t Critical Value
But if we look at the correspondent table http://referati-besplatno.ru/wp-content/uploads/2011/10/kzs.png we can see the difference (remember that alpha=1-p,sp p=0.95 is equal to alpha=0.05)
Where is the true?
Hi Alex, I don’t know how the examples from your book were calculated.
Hi Alex and Jason,
the numbers in Alex’s image is for two-tailed test.
When we calculate alpha for two-tail we divide it, so alpha = 0.05 means p=0.025 and 0.975,
So your probability is between both ends of the distribution:
if run this code, you can see that both values are symmetrical.
@Jason
thanks for the article!
Nice!
Jason, I believe that it would make your examples much clearer if you were to specify that all the ppf functions return a one-tailed critical value. That is why you can put it directly back in the CDF function and return the same percentage you started with. I’m sure you know this, but it would help clarify your examples that it is a *one-sided* 95% confidence interval of the standard normal, student-t, and chi-squared distributions.
Thanks, that is a great suggestion. I hope to make time for this update.
Thank you thank you thank you.
@Jason – It’s been 7 months. Please “make time” for this update. It’s critical to understanding.
Thanks for the suggestion Vicki.
hi,
your tutorials helped me a lot understanding the mystery of machine learning.
Could you please create a tutorial on 2 proportion z test. That would be helpful.
Dear M. jason Brownlee,
I would compute critical value for ANOVA test (on way).
Thank you very much.
Nice.
Sorry to bother you but i think there is an error in the 1 Tailed Distribution section. You say in text that:
“The statistic is compared to the calculated critical value. If the statistic is less than or equal to the critical value, the null hypothesis of the statistical test is failed to be rejected. Otherwise it is rejected.”
but in the summary of that same section you say:
Test Statistic Critical Value: Reject the null hypothesis of the statistical test.
which is it? Should the test statistic be greater than or greater than or equal to the critical value for the null hypothesis to be rejected?
The test statistic must exceeds the critical value in a right-tailed. An equal value is a failure to reject.
Thanks, I have updated the post to be consistent.