20.5: Consequences

The formal definitions of sample statistics, their distributions, and their relationship to population parameters are not merely academic distinctions — they are the very engine of statistical inference. From these foundational ideas flow consequences that govern how we learn from data. These consequences directly underpin the theory, diagnostics, and interpretation of linear models.

This section explores the most important of these implications, including the behavior of estimators, the decomposition of variance, the derivation of standard errors, and the theoretical justification for key assumptions. Understanding these logical outcomes is what transforms simple calculation into reliable and defensible statistical analysis.

✦•················• ✦ •··················•✦

Basic Matrix Results

First, when taking the transpose of a product, you switch the order of the multiplication:

\begin{equation}
\left( \mathbf{A}\mathbf{B}\right)^\prime = \mathbf{B}^\prime \mathbf{A}^\prime
\end{equation}

A similar result holds with taking the transpose of inverses. The only difference is that all three inverses must exist:

\begin{equation}
\left( \mathbf{A}\mathbf{B}\right)^{-1} = \mathbf{B}^{-1} \mathbf{A}^{-1}
\end{equation}

Lemma \(\PageIndex{1}\)

For any matrix \(\mathbf{X}\), the matrix \(\mathbf{X}^\prime \mathbf{X}\) is symmetric.

Proof.
I leave the proof as an exercise for you.

If you know the determinant of a matrix, you can easily calculate the determinant of a scalar multiple of that matrix.

Lemma \(\PageIndex{2}\)

If \( c \in \) ℝ and \(\mathbf{A} \in \mathcal{M}_n\), then \(\det c\mathbf{A} = c^n \det \mathbf{A}\).

There is a similar result with the determinant of a trace of a matrix.

Lemma \(\PageIndex{3}\)

If \( c \in \) ℝ and \(\mathbf{A} \in \mathcal{M}_n\), then \(\text{tr}\ c\mathbf{A} = c\ \text{tr}\ \mathbf{A}\).

Positive Definite Matrices

There are a lot of interesting properties of positive definite (pd) matrices. So, let us break these into a separate subsection.

Lemma \(\PageIndex{4}\)

The diagonal elements of a pd matrix are all positive. That is, let \(\mathbf{A} \in \mathscr{M}_{n}\) be positive definite, then \(a_{ii} > 0,\ \forall i \in \set{1, 2, \ldots, n}\).

Proof.
This can easily be shown by letting the \(\mathbf{q}^\prime\) row vector be \(\mathbf{e_i}\). The quadratic form \(\mathbf{q}^\prime \mathbf{A} \mathbf{q}\) would therefore equal the diagonal element at position \(i\). Since \(\mathbf{A}\) is positive definite, that element must be greater than 0.

\(\blacksquare\)

Note that the converse is not true, in general. Just because the diagonal elements are all positive does not mean that the matrix is positive definite. For an example, note

\begin{equation}
\mathbf{A} = \left[ \begin{matrix} 1&1\\ 1&1\\ \end{matrix} \right]
\end{equation}

is not positive definite. To see this, its determinant is 0.

A few additional results:

• The inverse of a pd matrix is also pd. That is, if \(\mathbf{A}\) is positive definite, then so is \(\mathbf{A}^{-1}\).
• Since the determinant of a pd matrix is positive, all of the eigenvalues are positive. And, since all of the diagonal elements of a pd matrix are positive, then the trace is positive. Since the trace is used to calculate the degrees of freedom, the matrix must be positive definite.
• If \(\mathbf{X}\) is a full rank matrix, even if not square, \(\mathbf{X}^\prime\mathbf{X}\) is positive definite.

• And, most importantly, the covariance matrix is positive definite if the design matrix, \(\mathbf{X}\), is full rank. Otherwise, it is positive semi-definite and has a determinant of zero.