12.3: - Interpreting the F-test and Post-Hoc Comparisons

All the F-test will tell you is that something is happening. But it will not tell you exactly what is happening. If you have a significant F-test, you know there are differences between the groups, but do not know which groups are responsible for the significant F-test.

At this point, we introduce post–hoc comparisons. Post-hoc means after the fact. We only do the post-hoc tests after we have a significant F-test, or for any significant overall statistical result. Statistical tests such as the F-test are overall tests; they indicate that somewhere among all of your groups or variables, there is a significant effect. Consider post-hoc tests as specific tests that determine where the significance is. For an ANOVA, the post-hoc comparisons indicate where the effect is, or which groups are different. If the F-test is not significant, there is no need to do the post-hoc comparisons.

The purpose of the post-hoc is to determine which two groups are significant. You can only compare two groups' means at a time. When you compare two groups at a time, you use a post-hoc test, which, in this case, is actually a version of the t-test. Recall that a t-test examines the means between two groups. Hence, we use a t-test here to compare the means among all the groups, one pair of groups at a time.

Why do we use post-hoc t-tests? Why not use a series of t-tests? We use the post-hoc t-test instead of the regular, or independent, t-test because we need the post-hoc test to adjust for the Type I error, or account for the experimenter-wise error rate.

Recall that a Type I error is when you find a significant effect, but in reality, you find a null effect or nothing. In what situations would you encounter a Type I error? One situation is what we call “fishing. Recall that in fishing, you continually cast your fishing line into the lake, over and over again. If you keep fishing, eventually you will catch something. But catching something after repeated tries does not mean that you can claim to be a fisherman, which is a significant finding, because in reality, you really are not a fisherman, and catching the fish was just a random lucky effect.

Where is the “fishing” in this situation? Conducting multiple t-tests is the same as casting your fishing line into the lake over and over again. Eventually, you will find a significant effect because you kept running t-tests after t-tests. Something will happen eventually. But that does not mean you found a significant effect. You got lucky because you kept running test after test. It is like dating. If you keep asking someone out on a date, eventually they will say yes (at least in theory, I have never found that to be the case, sigh).

Instead of just multiple t-tests and the basic independent t-tests, statisticians devised a series of post-hoc t-tests to adjust the p-value for the experimenter-wise error, or the increased likelihood of committing a Type I error due to conducting several t-tests. The common ones you will see associated with an F-test are Tukey, Sidak, and Bonferroni. These t-tests were devised to handle anomalies across the groups, such as unequal sample sizes or different distributions across the groups. The mathematical differences among the t-tests are beyond the scope of this textbook. One way to use these t-tests is to select them as your post-hoc tests, and if all three post-hoc tests give you the same answer, then you have confidence in your results. If one says significance and the other two do not, or two say significance, and one does not, then consult with a statistician, because something is off.

Once you get a significant F-test, all you can say is that something is significant. Then, you proceed to interpret the post-hoc tests to determine which group is significantly higher or lower than the other.

Sometimes, not all groups might be different; sometimes, only one group is causing the F-test to be significant. It is possible to have a significant F-test, without any significant post-hoc tests. There are no clear, consistent reasons for these scenarios. If these situations happen, consult a statistician.