Multiple Linear Regression

A response variable $Y$ is linearly related to $p$ different explanatory variables $X^{(1)},\ldots,X^{(p-1)}$ (where $p \geq2$). The regression model is given by

$$Y_i = \beta_0 + \beta_1 X_i^{(1)} + \cdots + \beta_p X_i^{(p-1)} + \varepsilon_i, \qquad i=1,\ldots,n \qquad \label{1}$$

where $\varepsilon_i$ have mean zero, variance $\sigma^2$ and are uncorrelated. The Equation $\ref{1}$ can be expressed in matrix notations as

$$Y = \mathbf{X} \beta + \varepsilon,$$

where

$$Y = \begin{bmatrix} Y_1 \\ Y_2 \\ \cdot\\ Y_n \end{bmatrix}, \qquad \varepsilon = \begin{bmatrix} \varepsilon_1 \\ \varepsilon_2 \\ \cdot\\ \varepsilon_n \end{bmatrix},$$

$$\mathbf{X} = \begin{bmatrix} 1 & X_1^{(1)} & X_1^{(2)} & \cdots & X_1^{(p-1)} \\ 1 & X_2^{(1)} & X_2^{(2)} & \cdots & X_2^{(p-1)} \\ \cdot & \cdot & \cdot & \cdots & \cdot\\ 1 & X_n^{(1)} & X_n^{(2)} & \cdots & X_n^{(p-1)} \end{bmatrix}, \qquad\mbox{and} \qquad \beta = \begin{bmatrix} \beta_0\\ \beta_1 \\ \cdot \\ \beta_{p-1} \end{bmatrix} .$$

So $\mathbf{X}$ is an $n \times p$ matrix.

Estimation Problem

Note that $\beta$ is estimated by the least squares procedure. That is minimizing the sum of squared errors $\sum_{i=1}^n (Y_i - \beta_0 - \beta_1 X_i^{(1)} - \cdots - \beta_{p-1} X_i^{(p-1)})^2$. The latter quantity can be expressed in matrix notations as $\parallel Y - \mathbf{X}\beta\parallel^2$. Minimization with respect to the parameter $\beta$ (a $p \times 1$ vector) gives rise to the normal equations:

$$\begin{eqnarray*} b_0 n + b_1\sum_i X_i^{(1)} + b_2 \sum_i X_i^{(2)} + \cdots + b_{p-1} \sum_i X_i^{(p-1)} &=& \sum_i Y_i \\ b_0 \sum_i X_i^{(1)} + b_1 \sum_i (X_i^{(1)})^2 + b_2 \sum_i X_i^{(1)} X_i^{(2)} + \cdots + b_{p-1} \sum_i X_i^{(1)} X_i^{(p-1)} &=& \sum_i X_i^{(1)} Y_i \\ \cdots \qquad \cdots \qquad \cdots \qquad \cdots &=& \cdot \\ b_0 \sum_i X_i^{(p-1)} + b_1 \sum_i X_i^{(p-1)}X_i^{(1)} + b_2 \sum_i X_i^{(p-1)} X_i^{(2)} + \cdots + b_{p-1} \sum_i (X_i^{(p-1)})^2 &=& \sum_i X_i^{(p-1)} Y_i \end{eqnarray*}$$

Observe that we can express this system of $p$ equations in $p$ variables $b_0,b_1,\ldots,b_{p-1}$ as $\begin{equation}\label{eq:normal} \mathbf{X}^T\mathbf{X} \mathbf{b} = \mathbf{X}^T Y, \end{equation}$ where $\mathbf{b}$ is a $p \times 1$ vector with $\mathbf{b}^T = (b_0,b_1,\ldots,b_{p-1})$.

If the $p \times p$ matrix $\mathbf{X}^T\mathbf{X}$ is nonsingular (as we shall assume for the time being), then the solution to this system is given by $\widehat \beta = \mathbf{b} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T Y .$ This is the least squares estimate of $\beta$.

Expected value and variance of random vectors

For an $m \times 1$ vector $\mathbf{Z}$, with coordinates $Z_1,\ldots,Z_m$, the expected value (or mean), and variance of $\mathbf{Z}$ are defined as

$$E(\mathbf{Z}) = E \begin{bmatrix} Z_1 \\ Z_2 \\ \cdot\\ Z_m \end{bmatrix} = \begin{bmatrix} E(Z_1) \\ E(Z_2)\\ \cdot\\ E(Z_m)\) \(\begin{bmatrix} \mbox{Var}(Z_1) & \mbox{Cov}(Z_1,Z_2) & \cdot & \mbox{Cov}(Z_1,Z_m) \\ \mbox{Cov}(Z_2,Z_1) & \mbox{Var}(Z_2) & \cdot & \mbox{Cov}(Z_2,Z_m) \\ \cdot & \cdot & \cdots & \cdot \\ \mbox{Cov}(Z_m,Z_1) & \mbox{Cov}(Z_m,Z_2) & \cdot & \mbox{Var}(Z_m) \end{bmatrix}.$$

Observe that Var$(\mathbf{Z})$ is an $m\times m$ matrix. Also, since Cov$(Z_i,Z_j)$ = Cov$(Z_j,Z_i)$ for all $1\leq i,j \leq m$, Var$(\mathbf{Z})$ is a symmetric matrix. Moreover, it can be checked, using the relationship that Cov$(Z_i,Z_j) = E(Z_iZ_j) - E(Z_i)E(Z_j)$, that Var$(\mathbf{Z}) = E(\mathbf{Z}\mathbf{Z}^T) - (E(\mathbf{Z}))(E(\mathbf{Z}))^T$.

Contributors

• Agnes Oshiro