2.11: Chapter summary

In this chapter, we reviewed basic statistical inference methods in the context of a two-sample mean problem using linear models and the lm function. You were introduced to using R to do enhanced visualizations (pirate-plots), permutation testing, and generate bootstrap confidence intervals as well as obtaining parametric \(t\)-test and confidence intervals. You should have learned how to use a for loop for doing the nonparametric inferences and the lm and confint functions for generating parametric inferences. In the examples considered, the parametric and nonparametric methods provided similar results, suggesting that the assumptions were not too violated for the parametric procedures. When parametric and nonparametric approaches disagree, the nonparametric methods are likely to be more trustworthy since they have less restrictive assumptions but can still make assumptions and can have problems.

When the noted conditions are violated in a hypothesis testing situation, the Type I error rates can be inflated, meaning that we reject the null hypothesis more often than we have allowed to occur by chance. Specifically, we could have a situation where our assumed 5% significance level test might actually reject the null when it is true 20% of the time. If this is occurring, we call a procedure liberal (it rejects too easily) and if the procedure is liberal, how could we trust a small p-value to be a “real” result and not just an artifact of violating the assumptions of the procedure? Likewise, for confidence intervals we hope that our 95% confidence level procedure, when repeated, will contain the true parameter 95% of the time. If our assumptions are violated, we might actually have an 80% confidence level procedure and it makes it hard to trust the reported results for our observed data set. Statistical inference relies on a belief in the methods underlying our inferences. If we don’t trust our assumptions, we shouldn’t trust the conclusions to perform the way we want them to. As sample sizes increase and/or violations of conditions lessen, then the procedures will perform better. In Chapter 3, some new tools for doing diagnostics are introduced to help us assess how and how much those validity conditions are violated.

It is good to review how to report hypothesis test conclusions and compare those for when we have strong, moderate, or weak evidence. Suppose that we are doing parametric inferences with lm for differences between groups A and B, are extracting the \(t\)-statistics, have 15 degrees of freedom, and obtain the following test statistics and p-values:

• \(t_{15} = 3.5\), p-value = 0.0016:

  There is strong evidence against the null hypothesis of no difference in the true means of the response between A and B (\(t_{15} = 3.5\), p-value = 0.0016), so we would conclude that there is a difference in the true means.

• \(t_{15} = 1.75\), p-value = 0.0503:

  There is moderate evidence against the null hypothesis of no difference in the true means of the response between A and B (\(t_{15} = 1.75\), p-value = 0.0503), so we would conclude that there is likely⁵⁸ a difference in the true means.

• \(t_{15} = 0.75\), p-value = 0.232:

  There is weak evidence against the null hypothesis of no difference in the true means of the response between A and B (\(t_{15} = 0.75\), p-value = 0.232), so we would conclude that there is likely not a difference in the true means.

The last conclusion also suggests an action to take when we encounter weak evidence against null hypotheses – we could potentially model the responses using the null model since we couldn’t prove it was wrong. We would take this action knowing that we could be wrong, but the “simpler” model that the null hypothesis suggests is often an attractive option in very complex models, such as what we are going to encounter in the coming chapters, especially in Chapters 5 and 8.