Wiki Page, Math 13 - Perez Sandbox : Module 20 - Introduction to Hypothesis Testing (5c of 5)

Wiki Page
2026-01-17T21:15:31-08:00

Beware of Type I Error in Multiple Hypothesis Tests

In general, if the null hypothesis is true, the significance level gives the probability of making a Type I error.

If we conduct a large number of hypothesis tests using the same null hypothesis, then, a Type I error will occur in a predictable percentage (α %) of the hypothesis tests. This is a problem!

If we run one hypothesis test and the data is significant at the 5% level, we have reasonably good evidence that the alternative hypothesis is true. If we run 20 hypothesis tests and the data in one of the tests is significant at the 5% level, it doesn’t tell us anything! We expect 5% of the tests (1 in 20) to show significant results just due to chance.

This is an important point! Beware of situations in which a hypothesis test has been performed a large number of times until a statistically significant result is obtained. This statistically significant result is probably a Type I error.

EXAMPLE Cell phones and brain cancer

The following is an excerpt from a 1999 New York Times article titled “Cell phones: questions but no answers,” as referenced by David S. Moore in Basic Practice of Statistics (4th ed., New York: W. H. Freeman, 2007):

A hospital study that compared brain cancer patients and a similar group without brain cancer found no statistically significant association between cell phone use and a group of brain cancers known as gliomas. But when 20 types of glioma were considered separately, an association was found between cell phone use and one rare form. Puzzlingly, however, this risk appeared to decrease rather than increase with greater mobile phone use.

This is an example of a probable Type I error. Notice that the hypothesis test is performed 20 times. Suppose we conducted 20 hypotheses tests with the null hypothesis “Cell phone use is not associated with cancer” at the 5% level. We expect 1 in 20 (5%) to give significant results by chance alone when there is no association between cell phone use and cancer. So the conclusion that this one type of cancer is related to cell phone use is probably just a result of random chance and not an indication of an association.

This cartoon from xkcd.com illustrates this same idea.

Mark this page as done, then click the "Next" or > button to continue.

Content by the Open Learning Initiative and licensed under CC BY.