5.5: Conclusion

This chapter has served as a critical bridge, transforming the deterministic framework of Ordinary Least Squares (OLS) into a powerful engine for statistical inference. We began with the solid mathematical foundation established in the previous chapter — a structure built entirely upon our chosen definition of "best" as minimizing the sum of squared residuals. To this purely algebraic edifice, we introduced a single, pivotal probabilistic assumption:

\begin{equation*}
\varepsilon_i \stackrel{\text{iid}}{\sim} N(0,\ \sigma^2)
\end{equation*}

This (overly) concise statement — that the unobserved error terms are independently and identically drawn from a Normal distribution with zero mean and constant variance — proved to be a keystone. From the marriage of our least-squares mathematics and this normality assumption, the entire inferential architecture of the chapter arose. The assumption endowed our previously abstract estimators with precise, known probability distributions: the regression coefficients \(b_i\) inherited a Normal distribution, while the quadratic form of the residuals led to a Chi-square distribution for the estimator of the error variance, \(\hat{\sigma}^2\).

These derived distributions were the raw material from which we forged our tools of inquiry. By constructing what MATH 322 students will call "pivotal quantities" (but the rest of us will just call ratios where the parameters of interest were separated from their standard errors), we arrived at test statistics with fully known, tabulated distributions. Specifically, we saw how the t-distribution emerged naturally for testing hypotheses about individual regression parameters. (In the future, we will see something similar about the F-distribution for comparing nested models.) This logical progression from assumption to distribution to test statistic is the very core of classical statistical inference (a.k.a. "Fisherian statistics"). Once a test statistic's distribution is known under a null hypothesis, the gates to hell inference are opened wide: we can calculate p-values to weigh the evidence against the null, and we can construct confidence intervals to express the plausible range of a parameter with a specified degree of certainty.

A particularly elegant application of this framework was the development of the Working-Hotelling confidence bands for the entire regression line. Unlike an interval for a single predicted value, these bands provide a simultaneous confidence region for the mean response at all levels of the predictor variables (a.k.a. creating bounds on the entire regression line), illustrating how our probability statements can be extended to entire functions derived from the model.

The power of this process is undeniable, but its validity rests entirely on the foundation we built. The difficulty in statistics often lies not in the calculations themselves, but in satisfying the conditions required to ensure that a test statistic follows its presumed theoretical distribution. Here, the assumptions of independence, identical distribution (iid), homoskedasticity, and Normality were the indispensable keys that unlocked these known distributions. They allowed us to move from "this is our best estimate" to "this is how confident we can be in that estimate, and here is how we can test hypotheses about it."

This dependency, however, directs our attention to a crucial and unavoidable question:

What happens when these foundational assumptions are not met?

The reliability of every p-value and confidence interval we have constructed is conditional upon their truth. In the next chapter, we will therefore turn from application to diagnosis. We will develop methods — both graphical and numerical — to test these very assumptions. Our task will be to learn how to scrutinize the residuals (the proxies for our unobservable errors) to assess whether or not the assumptions are reasonable, given the data and our model. This process of model criticism is not merely a technical step; it is a fundamental safeguard in statistical inference, ensuring that the elegant machinery we have just built is applied to a world of data that conforms (at least approximately) to the reality our models assume.