Wiki Page, Math 13 - Perez Sandbox : Module 21 - Hypothesis Test for a Population Proportion (3b of 3)

Wiki Page
2026-01-17T21:15:34-08:00

More about P-Values and the Significance Level

You may wonder why 5% is often selected as the significance level in hypothesis testing and why 1% is also a commonly used level.

The use of and is largely due to convenience and tradition. When Ronald Fisher (one of the founders of modern statistics) published one of his tables (back in the days before technology), he used a mathematically convenient scale that included 5% and 1%. Later, these same 5% and 1% levels were used by other people, in part just because Fisher was so highly esteemed. But mostly, these are arbitrary levels.

The idea of selecting some sort of relatively small cutoff was historically important in the development of statistics. But it’s important to remember that there is really a continuous range of increasing confidence toward the alternative hypothesis, not a single all-or-nothing value. There isn’t much meaningful difference, for instance, between the P-values 0.049 and 0.051, and it would be foolish to declare one case definitely a "real" effect and the other case definitely a "random" effect. In either case, the study results are roughly 5% likely by chance if there’s no actual effect.

Whether such a P-value is sufficient for us to reject a particular null hypothesis ultimately depends on the risk of making the wrong decision and the extent to which the hypothesized effect might contradict our prior experience or previous studies.

EXAMPLE Sample Size and Hypothesis Testing

Consider our earlier example about teenagers and Internet access. According to the Kaiser Family Foundation, 84% of U.S. children ages 8 to 18 had Internet access at home as of August 2009. Researchers wonder if this number has changed since then. The hypotheses we tested were:

:

:

The original sample consisted of 500 children, and 86% of them had Internet access at home. The P-value was about 0.22, which was not strong enough to reject the null hypothesis. There was not enough evidence to show that the proportion of all U.S. children ages 8 to 18 have Internet access at home.

Suppose we sampled 2,000 children and the sample proportion was still 86%.

Our test statistic would be Z 2.44, and our P-value would be about 0.015. The larger sample size would allow us to reject the null hypothesis even though the sample proportion was the same.

This is illustrated below for a sampling distribution of samples of 500 and for a sampling distribution of samples of 2,000. In both distributions, the sample proportions with more than 2% error are colored red. You can see that with samples of 2,000 children, it is much less likely that we see error exceeding 2%. For the samples of 500, the two-tailed P-value is 0.22, which for samples of 2,000, the P-value is only 0.015.

Similarly, the corresponding two-tailed area in the Standard Normal Distribution is much smaller for samples of 2,000. The z-scores for samples of 500 are -1.22 and 1.22, and for samples of 2,000, -2.44 and 2.44. The curve for samples of 500 has its z-scores much closer to the center of the curve than for samples of 2,000, where the z-scores are farther away, which shows the much smaller P-value for samples of 2,000.

Why does this happen?

Larger samples vary less, so a sample proportion of 0.86 is more unusual with larger samples than with smaller samples if the population proportion is really 0.84. This means that if the alternative hypothesis is true, a larger sample size will make it more likely that we reject the null. Therefore, we generally prefer a larger sample as we have seen previously.

Another way to say this: A given sample proportion will provide much stronger evidence against the null (and in favor of the alternative) when the sample size is larger.

Summary

In general if we view the null hypothesis as a statement of "no effect," then statisticians may interpret a "fail to reject" conclusion as a problem with sample size. The sample was not large enough to detect the "effect" described in the alternative hypothesis.

A larger sample will produce stronger evidence against the null. This means that if the sample is large enough, we will reject the null and accept the alternative hypothesis and conclude there is an "effect," and the proportions from large samples that differ only a small amount from hypothesized population proportion will be statistically significant. For this reason, statistically significant differences are not necessarily large differences, and can even be so small as to be of no practical importance in everyday decision-making.

EXAMPLE Drugs and Side Effects

The following is an excerpt from cancerguide.org, a website started by Steve Dunn when he discovered he had advanced kidney cancer at the age of 32. His goal for the website is to help cancer patients understand the technical medical information. Much of his website focuses on making sense of the statistical information reported with medical trials.

Here is an excerpt from Dunn's website that highlights the previous point that statistically significant results may not be of practical importance:

“It’s also important to realize that ‘statistically significant difference’ does not mean ‘big difference.’ If two treatments are very similar in outcome, but not exactly the same, you can find a statistically significant difference by just testing enough people. In general, the more people you include in a trial, the smaller a difference is needed before that difference proves to be statistically significant. So if in some trial, treatment A has a cure rate of 52% and treatment B 54%, then even if they tested enough patients to make this difference ‘statistically significant,’ you are not likely to decide that B is really much better, and very likely other characteristics of A and B such as side effects would guide your choice between them.”

Mark this page as done, then click the "Next" or > button to continue.

Content by the Open Learning Initiative and licensed under CC BY.