13.4: When Should You Conduct Post-Hoc Pairwise Comparisons?
- Page ID
- 19500
The pairwise comparison calculations for a factorial design are the same as any pairwise comparison after any significant ANOVA. Instead of reviewing them here (because you can review them in the prior two chapters), we are going to discuss when (and why) you would or would not conduct pairwise comparisons in a factorial design.
Long Answer
We’ll start by refreshing our member on what we’ve done before. Let’s start with t-test, from oh-so long ago!
Exercise \(\PageIndex{1}\)
Did we conduct pairwise comparisons when we retained the null hypothesis when comparing two groups with a t-test? Why or why not?
- Answer
-
We did not conduct pairwise comparisons when the null hypothesis was retained with a t-test. We didn’t need to find which means were difference because the null hypothesis (which we retained) says that all of the means are similar.
Exercise \(\PageIndex{2}\)
Did we conduct pairwise comparisons when we rejected the null hypothesis when comparing two groups with a t-test? Why or why not?
- Answer
-
I know that it was a long time ago, but no, we did not conduct pairwise comparisons with t-tests. Even when we rejected the null hypothesis (which said that the means were similar, so we are saying that they are probably different), we only had two means. The t-test was our “pairwise” comparison. In other words, because there were only two means, so we knew that if the means were statistically different from each other that the bigger one was statistically significantly bigger.
What about an ANOVA that compared three groups? To answer these questions, it doesn’t matter if the ANOVA was BG or RM, just that there one was IV with three (or more) groups.
Exercise \(\PageIndex{3}\)
Did we conduct pairwise comparisons when we retained the null hypothesis when comparing three groups with an ANOVA? Why or why not?
- Answer
-
No, we did not conduct pairwise comparisons with ANOVAS with three groups if we retained the null hypothesis. With any retained null hypothesis, we are agreeing that the means are similar, so we wouldn’t spend time looking for any pairs of means that are different.
Exercise \(\PageIndex{4}\)
Did we conduct pairwise comparisons when we rejected the null hypothesis when comparing three groups with an ANOVA? Why or why not?
- Answer
-
Yes, this is when we would conduct pairwise comparisons. The null hypothesis says that all of the means are similar, but when we reject that we are only saying that at least one mean is different from one other mean (one pair of means differs). When we have three or more groups, we need to figure out which means differ from which other means. In other words, a significant ANOVA shows us that at least one of the means is different from at least one other mean, but we don’t know which means are different from which other means. We have to do pairwise mean comparisons to see which means are significantly different from which other means.
Finally, on to factorial designs! If you’ve been answering the Exercises as you go, these should be pretty easy.
Exercise \(\PageIndex{5}\)
Do we conduct pairwise comparisons when we retain the null hypothesis for main effects in a factorial design? Why or why not?
- Answer
-
No. When we retain a null hypothesis, we are saying that all of the means are similar. Let’s not waste time looking for a difference when we just said that there wasn’t one.
Okay, this one might be a little challenging, so we'll walk through it together.
Example \(\PageIndex{1}\)
Do we conduct pairwise comparisons when we reject the null hypothesis for main effects in a factorial design?
Solution
It depends!
If we only have two means, we don’t have to conduct pairwise comparisons because (just like with a t-test) rejecting the null hypothesis for the main effect means that we know that the bigger mean is statistically significantly bigger.
But if our IV has more than two groups, then we would need to conduct pairwise comparisons (just like an in ANOVA) to find which means are different from which other means.
Back to an easier one on null hypotheses and post-hoc tests.
Exercise \(\PageIndex{6}\)
Do we conduct pairwise comparisons when we retain the null hypothesis for an interaction in a factorial design? Why or why not?
- Answer
-
No. The null hypothesis says that all of the means are similar. If we retain the null hypothesis, then we are saying that all of the means are probably similar. Why would we look for a difference between pairs of means that we think are similar?
This one should be clear if you understand the reasoning for when we do and do not conduct post-hoc pairwise comparisons.
Example \(\PageIndex{2}\)
Do we conduct pairwise comparisons when we reject the null hypothesis for an interaction in a factorial design? Why or why not?
Solution
Yes! The smallest factorial design is a 2x2, which means that we have our means representing the combination of the two IVs. Rejecting the null hypothesis for the interaction says that at least one of those means is different from at least one other mean. We should use pairwise comparisons to find which combination of IV levels has a different mean from which other combination.
Short Answer
|
|
Only Two Groups |
Three or More Groups or Two or More IVs |
|---|---|---|
|
Retain the Null Hypothesis |
No- means are similar |
No- means are similar |
|
Reject the Null Hypothesis |
No- The bigger group is statistically bigger |
Yes- Find which mean is different from each other mean by comparing each pair of means. |
Time to practice next!