11.4: T-test Assumptions

All statistical tests have assumptions. Assumptions in the context of statistical tests are characteristics or requirements that need to be met for the test results to be valid or correct. There is only one result from a statistical test: significant or not significant. By valid or correct, we mean that the results are likely a good estimate of whether or not your finding is significant. If the assumptions are not met, then it is possible that your finding of significant or not significant might not be a good estimate. If you conducted a different test or did the test again with a different sample, you might get a different result. If the assumptions are not met, then it's best to get a consultation because there are considerations before you make a decision to proceed with your test.

11.4.1: Homogeneity of Variance

The first assumption is homogeneity of variance between the two groups. This assumption means that both groups have the same variance, or spread of scores, and the spread of scores is a normal distribution. The variances, or the normal distribution, of both groups should be the same.

Figure 1 presents what the homogeneity of variance between the two groups should look like. The one on the left is homogeneity. The one on the right is heterogeneity. Does this matter? The one on the left has two clearly different groups because their spread of scores does not overlap. The one on the right has two groups with different variances. The first group has a normal distribution, but the second group has a wider distribution than the first group. The problem is that you cannot tell if the second group has a mean score that makes them different from the first group, or if the second group had a wider variation, or spread of scores, that made them different from the first group. Does this situation matter? Groups are supposed to be mutually exclusive or fundamentally different from each other. If the spread of scores of one group overlaps with the other, the groups may not be so different from each other.

Figure 1: Homogeneity of Variance Between Two Groups

Check this assumption by using Levene’s test for Equality of Variance. If this test is not significant, the variances are homogenous and equal. Therefore, report the t-test as is. That means you can trust the result. If this test IS significant, the variances are NOT homogenous, NOT equal. Therefore, DO NOT report the t-test as is. That means you CANNOT accept the results as accurate. Then you report the adjusted t-test as part of the t-test results.

Double check – if both the as is t-test and the adjusted t-test give the same result, meaning that both t-tests say that the result is significant or not significant, report the “s is t-test because it’s simpler. But, if the as is t-test and the adjusted t-test give DIFFERENT results, meaning that one t-test says that the result is significant or not significant, BUT the other t-test gives the OPPOSITE result, then CALL ME. Because something is going on.

11.4.2: Robust Test

It is important to know that the t-test is considered a robust test. Robust means you will get the same answer, whether the test is significant or not significant, even if the assumption is violated. If the homogeneity of variance assumption is not met, the t-test will likely give you the same result if the assumption were met. The preceding discussion matters if, at the end of the day, the t-test will give us an answer no matter what. The preceding discussion does matter because there might be something else going on besides the homogeneity of variance. Think of this assumption as a red flag, but it is not the only red flag that will make or break your statistical results.

11.4.3: Sample Size

Based on guidelines for a normal distribution, having 30 participants per group is ideal for obtaining a stable result. Simulation studies determined that when a sample size is n > 20 for each group, and if both groups are of an equal size, the t-test should produce robust, stable statistical results.

Of note is that the sample sizes for both groups should be equal. How equal is equal? If group A has 40, and group B has 50, that is equal enough. The problem arises when one group, particularly the group of interest, has a sample size that is below 20 and is not equal to the other group. If one group is male elementary grade boys with behavioral problems, and the second group is elementary grade females with behavioral problems, it is quite likely that the boys will have an n of 30, while the girls will have an n of 11. This imbalance likely occurs because, on average, there are more young boys with behavioral problems than young girls. Is this a problem? From a population standpoint, this imbalance is expected. From a statistical standpoint, it can be a problem because it is hard to establish a mean score for a group of just 11. What to do? You could recruit more participants, in this case, more young girls with behavioral problems. Or you may have to scrap the t-test and use another statistical test. If so, consult with a statistician.