20.6: Statistics in Matrices

While the formulas you learned in introductory statistics are perfectly suited for calculation, a matrix formulation offers a more general and powerful perspective that scales to advanced methods like linear models. This optional section re-expresses fundamental sample statistics — such as the mean, variance, and covariance — using matrix notation.

The purpose is twofold: to provide essential practice with matrix operations, and to reveal the elegant underlying structure that unifies these concepts. In what follows, we will represent a data sample of size \(n\) as a column vector \(\mathbf{Y}\). Through this lens, familiar quantities like the sum of squares emerge naturally as specific quadratic forms, building a critical bridge between basic statistics and the linear algebra that drives modern data analysis.

✦•················• ✦ •··················•✦

Lemma \(\PageIndex{1}\)

The sample mean using matrices: \(\overline{Y} = \frac{1}{n} \mathbf{j}^\prime \mathbf{Y}\).

Proof.

\begin{align}
\frac{1}{n} \mathbf{j}^\prime \mathbf{Y} &= \frac{1}{n} \left[ 1\ 1\ 1\ \cdots\ 1\ \right] \left[ \begin{matrix}Y_1 \\ Y_2 \\ Y_3 \\ \vdots \\ Y_n \end{matrix} \right] \\[1em]
&= \frac{1}{n} \left( 1Y_1 + 1Y_2 + 1Y_3 + \cdots + 1Y_n\right) \\[1em]
&= \frac{1}{n} \sum_{i=1}^n Y_i \\[1em]
&= \overline{Y}
\end{align}
\(\blacksquare\)

Theorem \(\PageIndex{2}\)

The sum of squared values using matrices: \(\mathbf{Y}^\prime\mathbf{Y}\).

Proof.

\begin{align}
\mathbf{Y}^\prime \mathbf{Y} &= \left[ y_1\ y_2\ y_3\ \cdots\ y_n\ \right] \left[ \begin{matrix}y_1 \\ y_2 \\ y_3 \ \vdots \\ y_n \end{matrix} \right] \\[1em]
&= \left( y_1y_1 + y_2y_2 + y_3y_3 + \cdots + y_ny_n\right) \\[1em]
&= \sum_{i=1}^n y_i^2
\end{align}
\(\blacksquare\)

Lemma \(\PageIndex{3}\)

The sample variance using matrices: \(s_y^2 = \frac{1}{n-1} \left( \mathbf{Y}-\overline{Y}\mathbf{j}\right)^\prime\left( \mathbf{Y}-\overline{Y}\mathbf{j}\right)\).

Proof.

\begin{align}
& \frac{1}{n-1} \left( \mathbf{Y}-\overline{Y}\mathbf{j}\right)^\prime\left( \mathbf{Y}-\overline{Y}\mathbf{j}\right)\\[1em]
&= \frac{1}{n-1}\ \left[ y_1-\overline{Y},\ y_2-\overline{Y},\ y_3-\overline{Y},\ \cdots,\ y_n-\overline{Y},\ \right] \left[ \begin{matrix} y_1-\overline{Y} \\ y_2-\overline{Y} \\ y_3-\overline{Y} \\ \cdots \\ y_n-\overline{Y} \\ \end{matrix} \right] \\[1em]
&= \frac{1}{n-1}\ (y_1-\overline{Y})(y_1-\overline{Y}) + (y_2-\overline{Y})(y_2-\overline{Y}) + \cdots + (y_n-\overline{Y})(y_n-\overline{Y}) \\[1em]
&= \frac{1}{n-1}\ \sum_{i=1}^n (y_i-\overline{Y})^2
\end{align}
\(\blacksquare\)

Lemma \(\PageIndex{4}\)

The sample covariance using matrices: \(s_{xy} = \frac{1}{n-1} \left( \mathbf{Y}-\overline{Y}\mathbf{j}\right)^\prime\left( \mathbf{X}-\bar{x}\mathbf{j}\right)\).

Proof.

This proof echoes the previous proof.

\begin{align}
& \frac{1}{n-1} \left( \mathbf{Y}-\overline{Y}\mathbf{j}\right)^\prime\left( \mathbf{X}-\bar{x}\mathbf{j}\right)\\[1em]
&= \frac{1}{n-1}\ \left[ y_1-\overline{Y},\ y_2-\overline{Y},\ y_3-\overline{Y},\ \cdots,\ y_n-\overline{Y},\ \right] \left[ \begin{matrix} x_1-\bar{x} \\ x_2-\bar{x} \\ x_3-\bar{x} \\ \cdots \\ x_n-\bar{x} \\ \end{matrix} \right] \\[1em]
&= \frac{1}{n-1}\ (y_1-\overline{Y})(x_1-\bar{x}) + (y_2-\overline{Y})(x_2-\bar{x}) + \cdots + (y_n-\overline{Y})(x_n-\bar{x}) \\[1em]
&= \frac{1}{n-1}\ \sum_{i=1}^n (y_i-\overline{Y})(x_i-\bar{x})
\end{align}
\(\blacksquare\)

Note that this notation leads to the synonym \(s^2_y = s_{yy}\). It also leads to a nice proof that the covariance matrix is symmetric.

Definition: Variance-Covariance Matrix

Let \(\mathbf{Y}\) be a random vector (a column vector whose elements are random variables). The quantity \(V[\mathbf{Y}]\) is called the variance-covariance matrix of \(\mathbf{Y}\).

Note that it is often just called the covariance matrix of \(\mathbf{Y}\).

Lemma \(\PageIndex{5}\)

Let \(\mathbf{Y} \in \mathscr{M}_{r,c}\). If \(\mathbf{Y}^\prime = \left[ \mathbf{Y_1},\ \mathbf{Y_2},\ \mathbf{Y_3},\ \cdots, \mathbf{Y_r} \right]\), then the elements of \(V[\mathbf{Y}]\) are \(\left[ \sigma_{ij}\right]\), where \(\sigma_{ij}\) is the covariance between \(Y_i\) and \(Y_j\) and \(\sigma_{i,i}\) is the variance of \(Y_i\).

Lemma \(\PageIndex{6}\)

Covariance matrices are symmetric.

Lemma \(\PageIndex{7}\)

If \(\mathbf{Y}\) is a random vector and \(\mathbf{X}\) is not, then

\begin{equation}
V[\mathbf{X}^\prime\mathbf{Y}] = \mathbf{X}^\prime V[\mathbf{Y}]\mathbf{X}
\end{equation}

This assumes that the multiplication makes sense (the matrices are commensurate).