Multiple Linear Regression

A response variable \(Y\) is linearly related to \(p\) different explanatory variables \(X^{(1)},\ldots,X^{(p-1)}\) (where \(p \geq2\)). The regression model is given by

\[Y_i = \beta_0 + \beta_1 X_i^{(1)} + \cdots + \beta_p X_i^{(p-1)} + \varepsilon_i, \qquad i=1,\ldots,n \qquad \label{1}\]

where \(\varepsilon_i\) have mean zero, variance \(\sigma^2\) and are uncorrelated. The Equation \ref{1} can be expressed in matrix notations as

\[Y = \mathbf{X} \beta + \varepsilon,\]

where

\[ Y = \begin{bmatrix} Y_1 \\ Y_2 \\ \cdot\\ Y_n \end{bmatrix}, \qquad \varepsilon = \begin{bmatrix} \varepsilon_1 \\ \varepsilon_2 \\ \cdot\\ \varepsilon_n \end{bmatrix},\]

\[\mathbf{X} = \begin{bmatrix} 1 & X_1^{(1)} & X_1^{(2)} & \cdots & X_1^{(p-1)} \\ 1 & X_2^{(1)} & X_2^{(2)} & \cdots & X_2^{(p-1)} \\ \cdot & \cdot & \cdot & \cdots & \cdot\\ 1 & X_n^{(1)} & X_n^{(2)} & \cdots & X_n^{(p-1)} \end{bmatrix}, \qquad\mbox{and} \qquad \beta = \begin{bmatrix} \beta_0\\ \beta_1 \\ \cdot \\ \beta_{p-1} \end{bmatrix} .\]

So \(\mathbf{X}\) is an \(n \times p\) matrix.

Estimation Problem

Note that \(\beta\) is estimated by the least squares procedure. That is minimizing the sum of squared errors \(\sum_{i=1}^n (Y_i - \beta_0 - \beta_1 X_i^{(1)} - \cdots - \beta_{p-1} X_i^{(p-1)})^2\). The latter quantity can be expressed in matrix notations as \(\parallel Y - \mathbf{X}\beta\parallel^2\). Minimization with respect to the parameter \(\beta\) (a \(p \times 1\) vector) gives rise to the normal equations:

\[\begin{eqnarray*} b_0 n + b_1\sum_i X_i^{(1)} + b_2 \sum_i X_i^{(2)} + \cdots + b_{p-1} \sum_i X_i^{(p-1)} &=& \sum_i Y_i \\ b_0 \sum_i X_i^{(1)} + b_1 \sum_i (X_i^{(1)})^2 + b_2 \sum_i X_i^{(1)} X_i^{(2)} + \cdots + b_{p-1} \sum_i X_i^{(1)} X_i^{(p-1)} &=& \sum_i X_i^{(1)} Y_i \\ \cdots \qquad \cdots \qquad \cdots \qquad \cdots &=& \cdot \\ b_0 \sum_i X_i^{(p-1)} + b_1 \sum_i X_i^{(p-1)}X_i^{(1)} + b_2 \sum_i X_i^{(p-1)} X_i^{(2)} + \cdots + b_{p-1} \sum_i (X_i^{(p-1)})^2 &=& \sum_i X_i^{(p-1)} Y_i \end{eqnarray*}\]

Observe that we can express this system of \(p\) equations in \(p\) variables \(b_0,b_1,\ldots,b_{p-1}\) as \(\begin{equation}\label{eq:normal} \mathbf{X}^T\mathbf{X} \mathbf{b} = \mathbf{X}^T Y, \end{equation}\) where \(\mathbf{b}\) is a \(p \times 1\) vector with \(\mathbf{b}^T = (b_0,b_1,\ldots,b_{p-1})\).

If the \(p \times p\) matrix \(\mathbf{X}^T\mathbf{X}\) is nonsingular (as we shall assume for the time being), then the solution to this system is given by \(\widehat \beta = \mathbf{b} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T Y .\) This is the least squares estimate of \(\beta\).

Expected value and variance of random vectors

For an \(m \times 1\) vector \(\mathbf{Z}\), with coordinates \(Z_1,\ldots,Z_m\), the expected value (or mean), and variance of \(\mathbf{Z}\) are defined as

\[E(\mathbf{Z}) = E \begin{bmatrix} Z_1 \\ Z_2 \\ \cdot\\ Z_m \end{bmatrix} = \begin{bmatrix} E(Z_1) \\ E(Z_2)\\ \cdot\\ E(Z_m)\) \(\begin{bmatrix} \mbox{Var}(Z_1) & \mbox{Cov}(Z_1,Z_2) & \cdot & \mbox{Cov}(Z_1,Z_m) \\ \mbox{Cov}(Z_2,Z_1) & \mbox{Var}(Z_2) & \cdot & \mbox{Cov}(Z_2,Z_m) \\ \cdot & \cdot & \cdots & \cdot \\ \mbox{Cov}(Z_m,Z_1) & \mbox{Cov}(Z_m,Z_2) & \cdot & \mbox{Var}(Z_m) \end{bmatrix}.\]

Observe that Var\((\mathbf{Z})\) is an \(m\times m\) matrix. Also, since Cov\((Z_i,Z_j)\) = Cov\((Z_j,Z_i)\) for all \(1\leq i,j \leq m\), Var\((\mathbf{Z})\) is a symmetric matrix. Moreover, it can be checked, using the relationship that Cov\((Z_i,Z_j) = E(Z_iZ_j) - E(Z_i)E(Z_j)\), that Var\((\mathbf{Z}) = E(\mathbf{Z}\mathbf{Z}^T) - (E(\mathbf{Z}))(E(\mathbf{Z}))^T\).

Contributors

• Agnes Oshiro