11.2: Hypothesis Testing with Analysis of Variance

So far we have seen what ANOVA is used for, why we use it, and how we use it. Now we can turn to the formal hypotheses we will be testing. As with before, we have a null and an alternative hypothesis to lay out. Our null hypothesis is still the idea of “no difference” in our data. Because we have multiple group means, we simply list them out as equal to each other:

\(
\begin{aligned}
\qquad H_0&: \text{There is no difference in the group means} \\
\qquad H_0&: \mu_1 = \mu_2 = \mu_3
\end{aligned}
\)

We list as many \(\mu\) parameters as groups we have. In the example above, we have three groups to test, so we have three parameters in our null hypothesis. If we had more groups, say, four, we would simply add another \(\mu\) to the list and give it the appropriate subscript, giving us:

\(
\begin{aligned}
\qquad H_0&: \text{There is no difference in the group means} \\
\qquad H_0&: \mu_1 = \mu_2 = \mu_3 = \mu_4
\end{aligned}
\)

Notice that we do not say that the means are all equal to zero; we only say that they are equal to one another. It does not matter what the actual value is, so long as it holds for all groups equally.

Our alternative hypothesis for ANOVA is a little bit different. Let’s take a look at it and then dive deeper into what it means:

\(
\begin{aligned}
\qquad H_A: \text{At least one mean is different}
\end{aligned}
\)

The first difference is obvious: there is no mathematical statement of the alternative hypothesis in ANOVA. This is due to the second difference: we are not saying which group is going to be different, only that at least one will be. Because we do not hypothesize about which mean(s) will be different, there is no way to write it mathematically. Similarly, we do not have directional hypotheses (greater than or less than) like we did in Unit 2. Due to this, our alternative hypothesis is always exactly the same: at least one mean is different.

In Unit 2, we saw that, if we reject the null hypothesis, we can adopt the alternative, and this made it easy to understand what the differences looked like. In ANOVA, we will still adopt the alternative hypothesis as the best explanation of our data if we reject the null hypothesis. However, when we look at the alternative hypothesis, we can see that it does not give us much information. We will know that a difference exists somewhere, but we will not know where that difference is. Is only Group 1 different, but Groups 2 and 3 are the same? Is only Group 2 different? Are all three of them different? Based on just our alternative hypothesis, there is no way to be sure. We will come back to this issue later and see how to find out specific differences. For now, just remember that we are testing for any difference in group means, and it does not matter where that difference occurs.

Now that we have our hypotheses for ANOVA, let’s work through an example. We will continue to use the data from the previous section for continuity.

Video: A Gentle Introduction to ANOVA – The Problem of Probability Pyramiding

A Gentle Introduction to ANOVA – The Problem of Probability Pyramiding on YouTube.

Example: Scores on Job-Application Tests

Our data come from three groups of 10 people each, all of whom applied for a single job opening: those with no college degree, those with a college degree that is not related to the job opening, and those with a college degree from a relevant field. We want to know if we can use this group membership to account for our observed variability and, by doing so, test if there is a difference between our three group means. We will start, as always, with our hypotheses.

Step 1: State the Hypotheses

Our hypotheses are concerned with the means of groups based on education level, so:

\[
\begin{aligned}
\qquad H_0&: \text{There is difference between the means of the education groups} \\
\qquad H_0&: \mu_1 = \mu_2 = \mu_3 \\[2.5ex]
\qquad H_A&: \text{At least one mean is different}
\end{aligned}
\nonumber \]

Again, we phrase our null hypothesis in terms of what we are actually testing, and we use a number of population parameters equal to our number of groups. Our alternative hypothesis is always exactly the same.

Step 2: Find the Critical Values

Our test statistic for ANOVA, as we saw above, is F. Because we are using a new test statistic, we will get a new table: the F distribution table, a portion of which is shown in Table \(\PageIndex{1}\). (The complete F table can be found in section 16.3.)

The F table only displays critical values for \(\alpha=.05\). This is because other significance levels are uncommon, so it is not worth it to use up the space to present them. There are now two degrees of freedom we must use to find our critical value: numerator and denominator. These correspond to the numerator and denominator of our test statistic, which, if you look at the ANOVA table presented earlier, are our Between and Within rows, respectively. The df_B is the “df: Numerator (Between)” because it is the degrees of freedom value used to calculate the Mean Square Between, which in turn is the numerator of our F statistic. Likewise, the df_W is the “df: Denominator (Within)” because it is the degrees of freedom value used to calculate the Mean Square Within, which is our denominator for F.

The formula for df_B is k − 1; remember that k is the number of groups we are assessing. In this example, k = 3, so our df_B = 2. This tells us that we will use the second column, the one labeled 2, to find our critical value. To find the proper row, we simply calculate the df_W, which was N − k. The original prompt told us that we have “three groups of 10 people each,” so our total sample size is 30. This makes our value for df_W = 27. If we follow the second column down to the row for 27, we find that our critical value is 3.35. We use this critical value the same way as we did before: it is our criterion against which we will compare our obtained test statistic to determine statistical significance.

Table \(\PageIndex{1}\): Snippet from critical values for F (F table).

	1	2	3	4	5	6	7
1	161	200	216	225	230	234	237
2	18.51	19	19.16	19.25	19.3	19.33	19.35
3	10.13	9.55	9.28	9.12	9.01	8.94	8.89
4	7.71	6.94	6.59	6.39	6.26	6.16	6.09
5	6.61	5.79	5.41	5.19	5.05	4.95	4.88
6	5.99	5.14	4.76	4.53	4.39	4.28	4.21
7	5.59	4.74	4.35	4.12	3.97	3.87	3.79
8	5.32	4.46	4.07	3.84	3.69	3.58	3.5
9	5.12	4.26	3.86	3.63	3.48	3.37	3.29
10	4.96	4.1	3.71	3.48	3.33	3.22	3.14
11	4.84	3.98	3.59	3.36	3.2	3.09	3.01
12	4.75	3.89	3.49	3.26	3.11	3	2.91
13	4.67	3.81	3.41	3.18	3.03	2.92	2.83
14	4.6	3.74	3.34	3.11	2.96	2.85	2.76
15	4.54	3.68	3.29	3.06	2.9	2.79	2.71
16	4.49	3.63	3.24	3.01	2.85	2.74	2.66
17	4.45	3.59	3.2	2.96	2.81	2.7	2.61
18	4.41	3.55	3.16	2.93	2.77	2.66	2.58
19	4.38	3.52	3.13	2.9	2.74	2.63	2.54
20	4.35	3.49	3.1	2.87	2.71	2.6	2.51
21	4.32	3.47	3.07	2.84	2.68	2.57	2.49
22	4.3	3.44	3.05	2.82	2.66	2.55	2.46
23	4.28	3.42	3.03	2.8	2.64	2.53	2.44
24	4.26	3.4	3.01	2.78	2.62	2.51	2.42
25	4.24	3.39	2.99	2.76	2.6	2.49	2.4
26	4.23	3.37	2.98	2.74	2.59	2.47	2.39
27	4.21	3.35	2.96	2.73	2.57	2.46	2.37

Step 3: Calculate the Test Statistic and Effect Size

Now that we have our hypotheses and the criteria we will use to test them, we can calculate our test statistic. To do this, we will fill in the ANOVA table, working our way from left to right and filling in each cell to get our final answer. We will assume that we are given \(SS_B=8246\) and \(SS_W=3020\). These may seem like random numbers, but remember that they are based on the distances between the groups themselves and within each group.

Figure \(\PageIndex{1}\)shows the plot of the data with the group means and grand mean included. If we wanted to, we could use this information, combined with our earlier information that each group has 10 people, to calculate the between-groups sum of squares by hand. However, doing so would take some time, and without the specific values of the data points, we would not be able to calculate our within-groups sum of squares, so we will trust that these values are the correct ones.

Figure \(\PageIndex{1}\): Means. (“Job Test Scores Group Means” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

Completing the ANOVA Summary Table

• We were given the sums of squares values for our first two rows, so we can use those to calculate the total sum of squares, \(SS_T=SS_B+SS_W=8,246+3,030=11,266\).
• We also calculated our degrees of freedom earlier, so we can fill in those values. Additionally, we know that the total degrees of freedom is N − 1, which is 29. This value of 29 is also the sum of the other two degrees of freedom, so everything checks out.
• Now we have everything we need to calculate our mean squares. Our MS values for each row are just the SS divided by the df for that row, giving us \(MS_B=SS_B/df_B=8,246/2=4,123\) and \(MS_W=SS_W/df_W=3,020/27=111.85\).
• Remember that we do not calculate a Total Mean Square, so we leave that cell blank. Finally, we have the information we need to calculate our test statistic. F is calculated as \(F=MS_B/MS_W=4,123/111.85=36.86\) our MSB divided by MSW.
• The completed ANOVA table is shown in Table \(\PageIndex{2}\).

Table \(\PageIndex{2}\): ANOVA Summary Table

Source	SS	df	MS	F
Between	8,246	2	4,123	36.86
Within	3,020	27	111.85
Total	11,266	29

So, working our way through the table, given only two SS values and the sample size and group size from before, we calculate our test statistic to be F_obt = 36.86, which we will compare to the critical value in Step 4.

Video: How To Calculate and Understand Analysis of Variance (ANOVA) F Test

How To Calculate and Understand Analysis of Variance (ANOVA) F Test on YouTube.

Effect Size: Variance Explained

Recall that the purpose of ANOVA is to take observed variability and see if we can explain those differences based on group membership. To that end, our effect size will be just that: the variance explained. You can think of variance explained as the proportion or percent of the differences we are able to account for based on our groups. We know that the overall observed differences are quantified as the total sum of squares, and that our observed effect of group membership is the between-groups sum of squares. Our effect size, therefore, is the ratio of these two sums of squares. Specifically:

\[
\eta^2=\frac{SS_B}{SS_T}
\nonumber \]

The effect size \(\eta^2\) is called “eta-squared” and represents variance explained. For our example, our values give an effect size of:

\[
\eta^2=\frac{8246}{11266}=.73
\nonumber \]

So, we are able to explain 73% of the variance in job-test scores based on education. This is, in fact, a huge effect size, and most of the time we will not explain nearly that much variance. Our guidelines for the size of our effects are:

• \(\eta^2 \ge .01\): small
• \(\eta^2 \ge .09\): medium
• \(\eta^2 \ge .25\): large

So, we found that not only do we have a statistically significant result, but that our observed effect was very large! However, we still do not know specifically which groups are different from each other. It could be that they are all different, or that only those job seekers who have a relevant degree are different from the others, or that only those who have no degree are different from the others. To find out which is true, we need to do a special analysis called a post hoc test.

Step 4: Make the Decision

Our obtained test statistic was calculated to be F_obt = 36.86, and our critical value was found to be F* = 3.35. Our obtained statistic is larger than our critical value, so we can reject the null hypothesis.

Reject \(H_0\). The results of the ANOVA indicated that there were significant differences in job skills test scores for applicants in each of the three education groups, and the effect size was large, F(2, 27) = 36.86, p < .05, \(\eta^2\) = .73. Post hoc tests (see the next section) were performed to determine where the differences were.

Notice that when we report F, we include both degrees of freedom. We always report the numerator and then the denominator, separated by a comma. We must also note that, because we were only testing for any difference, we cannot yet conclude which groups are different from the others. To do so, we need to perform a post hoc test.

Test Your Knowledge

Question \(\PageIndex{1}\)

Question \(\PageIndex{2}\)

Question \(\PageIndex{3}\)