Wiki Page, Math 13 - Perez Sandbox : Module 20 - Introduction to Hypothesis Testing (4b of 5)

Wiki Page
2026-01-17T21:15:30-08:00

More on Drawing a Conclusion in a Hypothesis Test Using a Level of Significance and the P-Value

EXAMPLE White male support for Obama

Let’s return to the hypothesis test about white male support for Obama.

: The proportion of white males voting for Obama in 2012 is 0.40.

: The proportion of white males voting for Obama in 2012 is less than 0.40.

Suppose that our random sample of 200 white male voters has 35% planning to vote for Obama in 2012. Clearly 35% is less than 40%, but is the difference statistically significant or due to chance?

The probability (P-value) is about 0.078. What can we conclude? A small P-value indicates that the data are unlikely to occur in random sampling from a population in which the null hypothesis is true. So the smaller the P-value, the stronger the evidence is against the null hypothesis.

Do you view something that happens 7.8% of the time as “highly unlikely”?

This is a judgment call. There is no right or wrong answer. Because this is a judgment call, we will often agree to a definition of “highly unlikely” before we run the test. This is the significance level, . The significance level is a benchmark for how small the P-value must be in order for us to say results are statistically significant. It gives us a cutoff point for rejecting the null hypothesis.

Here are the decision-making rules that we gave earlier.

If the P-value , we reject the null hypothesis in favor of the alternative hypothesis.

If the P-value , we fail to reject the null hypothesis.

Why do these “rules” make sense? Visualizing a simulation of the sampling distribution is helpful in understanding these rules.

Keep in mind that all of the sample proportions in the simulation did actually occur when we selected random samples from a population with P = 0.40. Results less than 0.40 support the alternative hypothesis but to varying degrees. To provide enough evidence to reject the null hypothesis and accept the alternative hypothesis, results have to be smaller than 0.40 and “highly unlikely.”

When we set the significance level at 5% ( = 0.05), we agree to view results that occur less than 5% of the time as “highly unlikely” when sampling from the population described by the null. In this case, the observed sample deviates enough from the hypothesized population proportion that we reject the null hypothesis and accept the alternative.

If we set the significance level at 10% ( = 0.10), we have changed the definition of highly unlikely. As you can see, different significance levels can lead to different conclusions.

Here are our conclusions for the two different levels of significance:

At the 5% level, our poll results are not statistically significant (P-value = 0.078). We conclude that white male support for Obama will not be less than 40% in 2012. (Note: This statement says we do not have enough evidence to accept . Because is related to the claim or hunch that motivated our investigation, we state our conclusion in terms of .)

At the 10% level, our poll results are statistically significant (P-value = 0.078). We conclude that white male support for Obama will be less than 40% in 2012. (Note: This statement says we have enough evidence to accept . Again, the conclusion is stated in terms of .)

How Do You Choose a Level of Significance?

Remember that the purpose of the hypothesis test is to describe the degree of evidence that the sample provides against the null hypothesis. The P-value does this. How small a P-value is convincing evidence? It depends on the situation and the opinions of the people who use the hypothesis test to make a decision. If rejecting the null hypothesis will be controversial or expensive, then the users of the test results may want to use a smaller level of significance than we did. For this reason, we always report the P-value with our conclusions so that the people who use the results of the test can determine if the P-value is strong enough evidence for their purpose.

Later we discuss other considerations for choosing a level of significance.

Mark this page as done, then click the "Next" or > button to continue.

Content by the Open Learning Initiative and licensed under CC BY.