8.2: Hypothesis Testing Framework

In this section we will outline the hypothesis testing process in full.

This process combines the ideas of both Ronald Fisher (who emphasized using data to test a ruling-out hypothesis, \( H_0 \)) and Jerzy Neyman and Egon Pearson (who structured hypothesis testing as a formal decision-making process based on Type I and Type II errors).

The following five-step plan will be used for every hypothesis test.

An Outline of the Five Steps of a Hypothesis Test

1. State the hypotheses
   Clearly define the null hypothesis (\( H_0 \)) and the alternative hypothesis (\( H_A \)).
   • \( H_0 \): often represents "no effect," "no difference," or "status quo." The approach we will take usually require an "equals" statement.
   • \( H_A \): represents all other cases from the null hypothesis. This will be some kind of inequality (>, <, or \(\neq\)).
   • Many fields have specific requirements on how the null hypotheses are structured. When in doubt, use a two-tailed test with an "equals" statement for \(H_0\) and an \(\neq\) statement for \(H_A\)
2. Select the appropriate test statistic and distribution
   Choose a statistical test that matches the structure of your question, and define the relevant test statistic \( T \), then calculate it using the sample data.
   • Is this a test about one mean? Use a z or t test for a single sample mean.
   • Is this comparing two means? Use a two-sample t test.
   • Is this a test about one proportion or rate? Use a z test for proportions.
   • Is this comparing two proportions? Use a two-proportion z test.
   • For other types of tests see the List of test statistics.
   • Your choice depends on: the type of variable, the number of groups, the statistics we can calculate, and the sample size.
3. Choose a significance level (\( \alpha \))
   Set the allowable probability of making a Type I Error, the risk you're willing to take of rejecting the null when it's actually true. Many industries set a specific standard for this value.

   Common values: \( \alpha = 0.05, 0.01, 0.10 \)

4. Calculate the test statistic and p-value
   Determine how extreme your observed result is under the assumption that \( H_0 \) is true, either via a p-value or by comparing to a critical value from a distribution table.
5. Make a decision and state your conclusion
   Use your comparison between the observed test statistic and the distribution under \( H_0 \) to decide:
   • Reject \( H_0 \) (if the evidence against \(H_0\) is statistically significant, meaning your p-value falls below your \(\alpha\)
   • Fail to reject \( H_0 \) (if the evidence against \(H_0\) isn't strong enough)

   Always frame your decision in the context of the original research question.

---

Example: "Winters are warmer today than when I was a kid"

A friend says, "I swear winters are warmer now than when we were kids." Can we test this statistically?

You decide to gather historical winter temperatures from 20 years ago and compare them to modern-day winters in your region. You calculate the average January temperature today and compare it to 32°F, the known average from the past.

Let's walk through the structure:

• Step 1: \[ H_0: \mu = 32^\circ\text{F} \quad \text{(no warming)} \\ H_A: \mu > 32^\circ\text{F} \quad \text{(today's winters are warmer)} \]
• Step 2: Use a one-sample test for a population mean. Since the population standard deviation is unknown, we would use a t-test; if \( \sigma \) were known, a z-test would apply instead.
• Step 3: Choose \( \alpha = 0.05 \). Since there is no excessive worry of the Type I error here (accidentally concluding that winters are warmer when they aren't), we will go with the default \( \alpha = 0.05 \).
• Step 4: Calculate the observed test statistic from your sample data (mean, standard deviation, sample size). If we assume that the current mean temperature is still 32°F, how likely or unlikely is our observed sample data?
• Step 5: Compare the test statistic to the appropriate distribution and make your conclusion. If the probability of our results is below \( \alpha = 0.05 \), we can conclude that winters are indeed warmer. If the probability of our results is higher than that, we don't have strong enough evidence to say winters are warmer.

This structure helps us test the belief ("it's warmer!") using real evidence from data.

Think About It:

• In the winters example, why is this a right-tailed test rather than a two-tailed test? When would a two-tailed test be more appropriate?
• What would it mean to make a Type I error in the winters context? What real-world consequences might that have?
• Before reading the next sections, can you think of a research question from your own life that could be framed as a hypothesis test? Which of the five steps feels most straightforward? Which feels most uncertain?

---

Related Video

---

What Comes Next

Now that we've outlined the general process, we're going to build specific examples using data. We'll start with hypothesis tests involving a single mean, and later move to tests comparing two means or working with proportions.

In each case, the framework we've outlined stays the same — we only change the specific test statistic and distribution used.