Wiki Page, Math 13 - Perez Sandbox : Module 21 - Hypothesis Test for a Population Proportion (3c of 3)

Wiki Page
2026-01-17T21:15:34-08:00

Cautions about Conclusions from Hypothesis Tests

It is not uncommon to see debate over the conclusions and implications of statistical studies. When we read the summary of a statistical study, it is important to evaluate whether the conclusions are reasonable in light of the study design.

Here are some examples of common pitfalls:

1. The study makes an inference based on nonrandom data.
If the data come from a sample that is not randomly selected or from groups that are not randomly assigned, we should not use the data in inference procedures.
Why? Because all inference procedures are based on probability. We can make probability statements only about random events. For this reason we build probability models from repeated random sampling or repeated random assignment. Without a random sample, we cannot use a hypothesis test to draw a conclusion about a population. Similarly, without random assignment of subjects to treatments, we cannot draw a conclusion about the relationship between two variables.

2. The study makes inappropriate cause-and-effect conclusions.
We can make cause-and-effect conclusions only with data from a randomized comparative experiment. If data comes from a single observational study, we cannot make cause-and-effect conclusions.

3. The study overgeneralizes its conclusions.

In the case of an experiment, if researchers randomly assign individuals to one of two treatments, statistically significant results suggest the treatment is effective. This is an appropriate causal conclusion if the experiment is well designed. But if the original group of individuals is not randomly selected, then we should be cautious about generalizing this conclusion to a broader population. For example, a well-designed experiment may provide evidence that aspirin lowers the incidence of heart attacks; but if the subjects are all white males, we cannot conclude that the aspirin effect will hold for black males or for women.

In the case of an observational study, the sample must be randomly selected from the population of interest. For example, if we want to draw a conclusion about teenagers nationwide, we cannot use a random sample of teenagers from our community.

The following graphic summarizes these ideas.

Summary

If we want to draw a conclusion about a population, we have to have a random sample of individuals from that population.

If we want to draw a cause-and-effect conclusion about the relationship between two variables, we have to conduct an experiment that uses random assignment of individuals to treatments.

In an experiment if we have both random selection of individuals and random assignment of those individuals to treatments, then we can draw a cause-and-effect conclusion about the treatment AND we can generalize that conclusion to a broader population.

Mark this page as done, then click the "Next" or > button to continue.

Content by the Open Learning Initiative and licensed under CC BY.