10.1: Ordinary Least Squares

This page serves as a (very) concise review of the core assumptions underlying ordinary least squares (OLS) regression. It revisits the fundamental premise that the model's errors are independent and identically distributed (i.i.d.) normal random variables with a constant variance. The section succinctly restates this assumption in both symbolic forms and interprets its key implication: the covariance matrix of the errors is a scalar matrix, where the diagonal represents the constant variance and the off-diagonal zeros confirm the errors are uncorrelated. This recapitulation sets the stage for the chapter's exploration of alternative least squares methods that relax one or more of these classic OLS conditions.

  

Learning Objectives

By the end of this section, you will be able to:

1. State the key ordinary least squares (OLS) assumption that the errors are independent and identically distributed (i.i.d.) with a mean of zero and constant variance, often expressed as \(\varepsilon \sim N(\mathbf{0}, \sigma^2 \mathbf{I})\).
2. Interpret the covariance matrix of the errors, \(\sigma^2 \mathbf{I}\), understanding that the diagonal entries represent the constant error variance (\(\sigma^2\)) and the off-diagonal entries of zero represent the assumption of no correlation between errors.
3. Explain that this assumption is fundamental for the standard OLS results regarding the distribution of estimators and the validity of hypothesis tests and confidence intervals.

 

✦•················• ✦ •··················•✦

 

First, let us review ordinary least squares (OLS). When formulating OLS estimation of the classical linear model (CLM), we made the assumption that the residuals are independent and identically distributed Normal with constant zero expected value and variance.

In symbols, this is written as either

\begin{equation}
\varepsilon_i \stackrel{\text{iid}}{\sim} N\left(0;\, \sigma^2\right)
\end{equation}

or as

\begin{equation}
\mathbf{E} \sim N_n\left(\mathbf{0};\, \sigma^2\mathbf{I}\right)
\end{equation}

The two statements are different ways of saying the exact same thing.

 

Note that the covariance matrix of \(\mathbf{E}\) is \(\sigma^2\mathbf{I}\):

\begin{equation}
V[\mathbf{E}]\ =\ \sigma^2\ \left[
\begin{matrix}
1 & 0 & 0 & \cdots & 0 \\
0 & 1 & 0 & \cdots & 0 \\
0 & 0 & 1 & & 0 \\
\vdots & \vdots & & \ddots & \vdots \\
0 & 0 & 0 & \cdots & 1 \\
\end{matrix}
\right]\ =\ \left[
\begin{matrix}
\sigma^2 & 0 & 0 & \cdots & 0 \\
0 & \sigma^2 & 0 & \cdots & 0 \\
0 & 0 & \sigma^2 & & 0 \\
\vdots & \vdots & & \ddots & \vdots \\
0 & 0 & 0 & \cdots & \sigma^2 \\
\end{matrix}
\right]
\end{equation}

The values along the diagonal represent the variances of each residual in the population. That they are the same value, \(\sigma^2\), indicates that the variance of the residuals is constant.

 

The values off the diagonal represent the covariance between the residuals. For instance, the value at position 1,2 is the covariance between \(\varepsilon_1\) and \(\varepsilon_2\), which we symbolized as \(\sigma_{1,2}\) in The Appendix of Statistics. Since that value is \(0\), we are specifying that the two are linearly uncorrelated (a.k.a. independent). Thus, the covariance matrix above specifies that the variances of the residuals are constant and that the residuals are independent of each other. If this requirement is met, then we should use ordinary least squares regression. However, not always is this requirement met.