# Multiple Linear Regression

### Multiple Linear Regression

A response variable $$Y$$ is linearly related to $$p$$ different explanatory variables $$X^{(1)},\ldots,X^{(p-1)}$$  (where $$p \geq2$$). The regression model is given by
$Y_i = \beta_0 + \beta_1 X_i^{(1)} + \cdots + \beta_p X_i^{(p-1)} + \varepsilon_i, \qquad i=1,\ldots,n \qquad (1)$
where $$\varepsilon_i$$ have mean zero, variance $$\sigma^2$$ and are uncorrelated. The equation (1) can be expressed in matrix notations as
$$Y = \mathbf{X} \beta + \varepsilon, \qquad \mbox{where} \qquad Y = \begin{bmatrix} Y_1 \\ Y_2 \\ \cdot\\ Y_n \end{bmatrix}, \qquad \varepsilon = \begin{bmatrix} \varepsilon_1 \\ \varepsilon_2 \\ \cdot\\ \varepsilon_n \end{bmatrix},$$
$$\mathbf{X} = \begin{bmatrix} 1 & X_1^{(1)} & X_1^{(2)} & \cdots & X_1^{(p-1)} \\ 1 & X_2^{(1)} & X_2^{(2)} & \cdots & X_2^{(p-1)} \\ \cdot & \cdot & \cdot & \cdots & \cdot\\ 1 & X_n^{(1)} & X_n^{(2)} & \cdots & X_n^{(p-1)} \end{bmatrix}, \qquad\mbox{and} \qquad \beta = \begin{bmatrix} \beta_0\\ \beta_1 \\ \cdot \\ \beta_{p-1} \end{bmatrix} .$$

So $$\mathbf{X}$$ is an $$n \times p$$ matrix.

### Estimation problem

Note that $$\beta$$ is estimated by the $$\textit{least squares}$$ procedure. That is minimizing the sum of squared errors $$\sum_{i=1}^n (Y_i - \beta_0 - \beta_1 X_i^{(1)} - \cdots - \beta_{p-1} X_i^{(p-1)})^2$$. The latter quantity can be expressed in matrix notations as $$\parallel Y - \mathbf{X}\beta\parallel^2$$. Minimization with respect to the parameter $$\beta$$ (a $$p \times 1$$ vector) gives rise to the $${\bf normal equations}$$ :

$$\begin{eqnarray*} b_0 n + b_1\sum_i X_i^{(1)} + b_2 \sum_i X_i^{(2)} + \cdots + b_{p-1} \sum_i X_i^{(p-1)} &=& \sum_i Y_i \\ b_0 \sum_i X_i^{(1)} + b_1 \sum_i (X_i^{(1)})^2 + b_2 \sum_i X_i^{(1)} X_i^{(2)} + \cdots + b_{p-1} \sum_i X_i^{(1)} X_i^{(p-1)} &=& \sum_i X_i^{(1)} Y_i \\ \cdots \qquad \cdots \qquad \cdots \qquad \cdots &=& \cdot \\ b_0 \sum_i X_i^{(p-1)} + b_1 \sum_i X_i^{(p-1)}X_i^{(1)} + b_2 \sum_i X_i^{(p-1)} X_i^{(2)} + \cdots + b_{p-1} \sum_i (X_i^{(p-1)})^2 &=& \sum_i X_i^{(p-1)} Y_i \end{eqnarray*}$$

Observe that we can express this system of $$p$$ equations in $$p$$ variables $$b_0,b_1,\ldots,b_{p-1}$$ as $$$$\label{eq:normal} \mathbf{X}^T\mathbf{X} \mathbf{b} = \mathbf{X}^T Y,$$$$ where $$\mathbf{b}$$ is a $$p \times 1$$ vector with $$\mathbf{b}^T = (b_0,b_1,\ldots,b_{p-1})$$.

If the $$p \times p$$ matrix $$\mathbf{X}^T\mathbf{X}$$ is nonsingular (as we shall assume for the time being), then the solution to this system is given by $$\widehat \beta = \mathbf{b} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T Y .$$ This is the least squares estimate of $$\beta$$.

### Expected value and variance of random vectors

For an $$m \times 1$$ vector $$\mathbf{Z}$$, with coordinates $$Z_1,\ldots,Z_m$$, the expected value (or mean), and variance  of $$\mathbf{Z}$$ are defined as $$E(\mathbf{Z}) = E \begin{bmatrix} Z_1 \\ Z_2 \\ \cdot\\ Z_m \end{bmatrix} = \begin{bmatrix} E(Z_1) \\ E(Z_2)\\ \cdot\\ E(Z_m)$$ $$\begin{bmatrix} \mbox{Var}(Z_1) & \mbox{Cov}(Z_1,Z_2) & \cdot & \mbox{Cov}(Z_1,Z_m) \\ \mbox{Cov}(Z_2,Z_1) & \mbox{Var}(Z_2) & \cdot & \mbox{Cov}(Z_2,Z_m) \\ \cdot & \cdot & \cdots & \cdot \\ \mbox{Cov}(Z_m,Z_1) & \mbox{Cov}(Z_m,Z_2) & \cdot & \mbox{Var}(Z_m) \end{bmatrix}.$$ Observe that Var$$(\mathbf{Z})$$ is an $$m\times m$$ matrix. Also, since Cov$$(Z_i,Z_j)$$ = Cov$$(Z_j,Z_i)$$ for all $$1\leq i,j \leq m$$, Var$$(\mathbf{Z})$$ is a symmetric matrix. Moreover, it can be checked, using the relationship that Cov$$(Z_i,Z_j) = E(Z_iZ_j) - E(Z_i)E(Z_j)$$, that Var$$(\mathbf{Z}) = E(\mathbf{Z}\mathbf{Z}^T) - (E(\mathbf{Z}))(E(\mathbf{Z}))^T$$.

### Contributors

• Agnes Oshiro
(Source: Spring 2012 STA108 Handout 12    )