Wiki Page, Math 13 - Perez Sandbox : Module 20 - Introduction to Hypothesis Testing (2b of 5)

Wiki Page
2026-01-17T21:15:30-08:00

How Small Does the P-Value Have to Be to Reject the Null Hypothesis?

A small P-value indicates that it is unlikely that the actual sample data came from the population described by the null hypothesis. More precisely, a small P-value says that there is only a small chance that we will randomly select a sample with results at least as extreme as the data if is true. The smaller the P-value, the stronger the evidence against .

But how small does the P-value have to be in order to reject ?

In practice, we often compare the P-value to 0.05. We reject the null hypothesis in favor of the alternative if the P-value is less than (or equal to) 0.05.

Why compare to 0.05?

It is just a convention to define a cut-off to identify results that are highly unlikely to occur when H₀ is true. By comparing the P-value to 0.05, we are saying that 5% of the time when H₀ is true, sampling variability will produce results at least as extreme as the data we observed in our study.

It is possible that a random sample will have this much error when H₀ is true, but is highly unlikely. Therefore, when a real life study has a result that is highly unlikely to occur in random sampling from a described by the null hypothesis, statisticians are comfortable concluding that is (probably) not true.

The number 0.05 is called the significance level for the test.

Are other significance levels used in statistical practice?

Occasionally, researchers use significance levels other than 0.05. In particular, if rejecting will be controversial or expensive, we may require stronger evidence. In this case, a smaller significance level, such as 0.01, is used. As with the hypotheses, we should choose the significance level before collecting data. It is treated as an agreed-upon benchmark prior to conducting the hypothesis test. In this way, we can avoid arguments about the strength of the data.

The symbol we use for the significance level is (the lowercase Greek letter alpha). We sometimes refer to the significance level as the -level. We call this value the significance level because if the P-value is less than the significance level, we say the results of the test showed a statistically significant difference; the difference is not attributed to chance variability.

If the P-value , we reject the null hypothesis in favor of the alternative hypothesis.

If the P-value , we fail to reject the null hypothesis.

Statistical Significance: Another Way to Describe Unlikely Results

When the P-value is less than (or equal to) , we also say that the difference between the actual sample statistic and the assumed parameter value is statistically significant.

In the previous example, the P-value is less than 0.05, so we say the difference between the sample mean (75 MB) and the assumed mean from the null hypothesis (62 MB) is statistically significant.

You will also see this described as a significant difference. A significant difference is an observed difference that is too large to attribute to chance. In other words, it is a difference that is unlikely when we consider sampling variability alone. If the difference is statistically significant, we reject .

Other Observations about Stating Conclusions in a Hypothesis Test

We have to be very careful in how we state the conclusion. There are only two possibilities.

We have enough evidence to reject the null hypothesis and support the alternative hypothesis.

We do not have enough evidence to reject the null hypothesis, so there is not enough evidence to support the alternative hypothesis.

If the P-value in the previous example was greater than 0.05, then we would not have enough evidence to reject and accept . In this case our conclusion would be that “there is not enough evidence to show that the mean amount of data used by teens with smart phones has increased.”

Notice that this conclusion answers the original research question. It focuses on the alternative hypothesis. It does not say “the null hypothesis is true.” We never accept the null hypothesis or state that it is true. When there is not enough evidence to reject , the conclusion will say, in essence, that “there is not enough evidence to support .” But of course we will state the conclusion in the specific context of the situation we are investigating.

Mark this page as done, then click the "Next" or > button to continue.

Content by the Open Learning Initiative and licensed under CC BY.