5.2: Test Statistics and Hypothesis Testing

Building on the solid mathematical foundation established throughout this chapter, we now possess the essential tools to move from pure derivation to practical inference. The preceding section formalized the sampling distributions of our key estimators — specifically, showing that under standard assumptions, the estimated regression coefficients follow a multivariate normal distribution, and the error variance estimate, appropriately scaled, follows a chi-square distribution. This critical step transforms our algebraic results into probabilistic ones.

With these exact distributions in hand — and armed with our foundational knowledge of probability theory — we can now construct rigorous statistical procedures to test individual hypotheses. This allows us to interrogate specific, targeted questions about our model, such as whether a particular predictor variable, \(\beta_i\), has a statistically significant partial relationship with the outcome after accounting for all other variables in the model.

✦•················• ✦ •··················•✦

For this section, we rely heavily on the definition of Student's t distribution. Please review it before continuing.

Definition: Student's t Distribution

If we let \(Z \sim N(0,\ 1)\) and \(V \sim \chi^2_{\nu}\), with \(Z\) and \(V\) independent, then

\begin{equation}
T \stackrel{\text{def}}{=} \frac{Z}{\sqrt{V/\nu}} \label{eqn:lm3-tform}
\end{equation}

follows a Student's t distribution with \(\nu\) degrees of freedom.

You have, most likely, come across this ratio in your elementary statistics course when you were investigating hypotheses about a single population mean (assuming that the data came from a Normal distribution).

Theorem \(\PageIndex{1}\): Distribution of T(\(b_1\))

The quantity

\begin{equation}
T = \frac{\displaystyle b_1 - \beta_1}{\displaystyle \sqrt{\ \mathrm{MSE}/S_{xx}\ }}
\end{equation}

follows a Student's t distribution with \(n-p\) degrees of freedom.

Proof.
To prove this statement, one must show that it can be written in the form of Equation \(\ref{eqn:lm3-tform}\). First, let us look at the numerator.

\begin{align}
b_1 &\sim N \left( \beta_1,\ \sigma^2/S_{xx} \right) \\[1em]
\implies \qquad \frac{ b_1 - \beta_1}{\sqrt{\sigma^2/S_{xx}}} &\sim N(0,\ 1)
\end{align}

Now for the denominator we use a previous theorem (The MSE Theorem):

\begin{align}
\frac{(n-p)\ \mathrm{MSE}}{\sigma^2} &\sim \chi^2_{n-p}
\end{align}

Next, we put these two together

\begin{align}
T &= \frac{b_1 - \beta_1}{\sqrt{\mathrm{MSE}/S_{xx}}} \\[1em]
&= \frac{\quad \displaystyle \frac{ b_1 - \beta_1}{\sqrt{\sigma^2/S_{xx}}} \quad }{ \displaystyle\sqrt{ \frac{\mathrm{MSE}}{\sigma^2}} } \\[1em]
&= \frac{ \frac{ b_1 - \beta_1}{\sqrt{\sigma^2/S_{xx}}} }{ \sqrt{ \frac{(n-p)\ \mathrm{MSE}}{\sigma^2}/(n-p)} } \label{eq:lm3-blah}
\end{align}

Note that the numerator of Equation \(\ref{eq:lm3-blah}\) follows a standard Normal distribution, while the denominator is the square-root of a chi-square distribution divided by its degrees of freedom. Thus, by the definition of Student's t distribution, the quantity \(T\) follows a Student's t distribution with \(n-p\) degrees of freedom.

\begin{align}
T &\sim t_{n-p}
\end{align}

\(\blacksquare\)

Note

This result is important for two reasons:

First, it allows us to test hypotheses regarding the \(\beta_1\) parameter.

Second, this result allows us to calculate confidence intervals for \(\beta_1\) (see Section 5.3: Confidence Intervals).

This parameter is usually of most interest to researchers as it provides "the effect of the independent variable on the dependent variable."

Since we know the distribution of this ratio, we can calculate p-values for any hypothesis about \(\beta_1\) using the same rules as from your elementary statistics course (see the Table below).

Table \(\PageIndex{1}: How to calculate p-values, given the alternative hypothesis.

Hypothesis Testing
\(H_{0}: \beta_{1} = \beta_{1,0}\)	\( H_{A}: \beta_1 \neq \beta_{1,0}\)	p-value = \( P[t \le -|T|] \times 2 \)
\(H_{0}: \beta_{1} \le \beta_{1,0}\)	\(H_{A}: \beta_1 \gt \beta_{1,0}\)	p-value = \( P[t \ge T] \)
\(H_{0}: \beta_{1} \ge \beta_{1,0}\)	\(H_{A}: \beta_1 \lt \beta_{1,0}\)	p-value = \( P[t \le T] \)

Technically, we do need to show that \(b_1\) and \(MSE\) are independent. If they are not, then Theorem 1 above theorem is not valid. For the proof, you will want to investigate Cochran's Theorem and its uses (Bapat 2000, Cochrane 1934).

Theorem \(\PageIndex{2}\): Distribution of T(\(b_0\))

The ratio

\begin{equation}
T = \frac{b_0 - \beta_0}{ \sqrt{\ \mathrm{MSE}\ \left( \frac{1}{n}+\frac{\bar{x}^2}{S_{xx}}\right)\ } }
\end{equation}

follows a Student's t distribution with \(n-p\) degrees of freedom.

Proof. I leave this as an exercise.

This theorem allows us to easily prove the next.

Theorem \(\PageIndex{3}\): Distribution of T(\(\hat{Y}\))

The ratio

\begin{equation}
T = \frac{\hat{y} - \hat{y}_0}{ \sqrt{\ \mathrm{MSE}\ \left( \frac{1}{n}+\frac{(x-\bar{x})^2}{S_{xx}}\right)\ } }
\end{equation}

follows a Student's t distribution with \(n-p\) degrees of freedom.

Proof. I leave this as an exercise, too.

That sure is a lot of exercise.

How are those abs doing? Sore yet?