8.2: Hypothesis Testing Framework

Now that we've built a foundation for understanding what a hypothesis is — and what it means to be wrong — we’re ready to outline the hypothesis testing process in full.

This process combines the ideas of both Ronald Fisher (who emphasized using data to test a ruling-out hypothesis, \( H_0 \)) and Jerzy Neyman and Egon Pearson (who structured hypothesis testing as a formal decision-making process based on Type I and Type II errors).

Here’s a five-step plan we’ll use for every hypothesis test. Whether we’re studying proportions, means, or comparing two groups, this structure remains the same — what will change is the type of test statistic we use.

An Outline of the Five Steps of a Hypothesis Test

1. State the hypotheses
   Clearly define the null hypothesis (\( H_0 \)) and the alternative hypothesis (\( H_A \)).
   • \( H_0 \): often represents “no effect,” “no difference,” or “status quo.” This is always an "equals" statement.
   • \( H_A \): what you are trying to find evidence for. This will be some kind of inequality (>,<, or \(\neq\)).
2. Select the appropriate test statistic and distribution
   Choose a statistical test that matches the structure of your question, and define the relevant test statistic \( T \), then calculate it using the sample data.
   • Is this a test about one mean? Use a z or t test for a single sample mean.
   • Is this comparing two means? Use a two-sample t test.
   • Is this a test about one proportion or rate? Use a z test for proportions.
   • For other types of tests see - List of test statistics
   • Your choice depends on: the type of variable, the number of groups, what statistics we can calculate and the sample size.
3. Choose a significance level (α)
   Set the allowable probability of making a Type I Error — in other words, the risk you're willing to take of rejecting the null when it's actually true.

   Common values: \( \alpha = 0.05, 0.01, 0.10 \)

4. Determine where it falls in the sampling distribution (Calculate the p-value)
   Then determine how extreme that value is under the assumption \( H_0 \) is true (either via a p-value or using a critical value from a distribution table).
5. Make a decision and state your conclusion
   Use your comparison between \( t_{obs} \) and the distribution under \( H_0 \) to decide:
   • Reject \( H_0 \) (if the evidence against \(H_0\) is strong)
   • Fail to reject \( H_0 \) (if the evidence against \(H_0\) isn't strong enough)

   Always frame your decision in the context of the original research question.

---

Example: “Winters are hotter today than when I was a kid”

A friend says, “I swear winters are warmer now than when we were kids.” Can we test this statistically?

You decide to gather historical winter temperatures from 20 years ago and compare them to modern-day winters in your region. You calculate the average January temperature today and compare it to 32 degrees F, the known average from the past.

Let’s walk through the structure:

• Step 1: \[ H_0: \mu = 32^\circ\text{F} \quad \text{(no warming)} \\ H_A: \mu > 32^\circ\text{F} \quad \text{(today’s winters are warmer)} \]
• Step 2: Use a one-sample t-test for a population mean (since you're comparing sample data to a known mean).
• Step 3: Choose \( \alpha = 0.05 \). Since there is no excessive worry of the Type I error here (accidentally concluding that winters are warmer when they aren't) we will go with the default \( \alpha = 0.05 \).
• Step 4: Calculate \( t_{obs} \) from your sample data (mean, standard deviation, sample size). If we assume that the current mean temperature is still 32 degrees F, how likely or unlikely is our observed sample data?
• Step 5: Compare \( t_{obs} \) to t-distribution and make your conclusion. If the probability of our results is below \( \alpha = 0.05 \) we can conclude that winters are indeed warmer. If the probability of our results is higher than that, we don't have strong enough evidence to say winters are warmer.

This structure helps us test the belief (“it’s hotter!”) using real evidence from data.

---

What Comes Next

Now that we’ve outlined the general process, we’re going to build specific examples using data. We’ll start with hypothesis tests involving a single mean, and later move to tests comparing two means or working with proportions.

In each case, the framework we've outlined stays the same — we only change the specific test statistic and distribution used.