22.5: Distributions of Sample Statistics

Now, let us synthesize the ideas from the previous sections. We have established that sample statistics are random variables, and that each population parameter is a fixed target. The true power of statistical inference emerges when we connect these concepts by knowing the distributions of our sample statistics. This knowledge — understanding how a statistic like the sample mean or variance behaves across all possible samples — is what allows us to move from a single, observed data point to broader conclusions. It enables us to quantify uncertainty, construct confidence intervals, and test hypotheses.

Ultimately, it is through the known distribution of a statistic that we can rigorously draw conclusions about an unknown population parameter using only the information contained in a random sample.

✦•················• ✦ •··················•✦

Sample Mean

The distribution of the sample mean is one of the first sampling distributions you would have come across in your introductory statistics course. It usually serves as the link between the Normal distribution and confidence intervals.

Theorem \(\PageIndex{1}\): Distribution of the Sample Mean

If \(Y_i \sim N(\mu,\ \sigma^2)\), and we collect an independent and identically distributed (iid) sample of size \(n\), then

\begin{equation}
\overline{Y} \sim N\left(\mu,\ \frac{\sigma^2}{n} \right)
\end{equation}

Proof.
This proof proceeds in three parts. The first part notes that a linear combination of independent Normal random variables is also a Normal random variable. (The proof of this can be found at Theorem: Sum of Normals.)

Since the Normal distribution has two parameters, mean and variance, the second and third parts determine the expected value and variance of that random variable.

\begin{align}
E\left[\overline{Y}\right] &= E\left[ \frac{1}{n} \sum_{i=1}^n\ Y_i \right] \\[1em]
&= \frac{1}{n} \sum_{i=1}^n\ E[Y_i] \\[1ex]
&= \frac{1}{n} \sum_{i=1}^n\ \mu \\[1ex]
&= \frac{1}{n} n \mu \\[1ex]
&= \mu
\end{align}

Thus, the expected value of \(\overline{Y}\) is \(\mu\). Now for the variance:

\begin{align}
V\left[\overline{Y}\right] &= V\left[ \frac{1}{n} \sum_{i=1}^n\ Y_i \right] \\[1em]
&= \frac{1}{n^2} V\left[ \sum_{i=1}^n\ Y_i \right]\\[1ex]
&= \frac{1}{n^2} \sum_{i=1}^n\ V[ Y_i ] \\[1ex]
&= \frac{1}{n^2} \sum_{i=1}^n\ \sigma^2 \\[1ex]
&= \frac{1}{n^2} n \sigma^2 \\[1ex]
&= \frac{\sigma^2}{n}
\end{align}

Putting these parts together gives us our conclusion.

\(\blacksquare\)

Note

Notice the procedure in the previous proof. The first step is to determine the distribution. The remaining steps determine the values of the distribution's parameters. We needed to determine the values of \(\mu\) and \(\sigma^2\) because the Normal distribution uses both as parameters.

Note

That the expected value of \(\overline{Y}\) is \(\mu\) means that the samle mean is an unbiased estimator of the population mean.

Sample Variance

The distribution of the sample variance is rarely covered in introductory statistics. However, it is needed when determining some important test statistics.

Theorem \(\PageIndex{2}\): Distribution of the Sample Variance

If \(Y_i \stackrel{\text{iid}}{\sim} N(\mu,\ \sigma^2)\), and we collect a random sample of size \(n\), then

\begin{equation}
\frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{\nu=n-1}
\end{equation}

Proof.
Since we need to show that it follows a Chi-square distribution, we simply use the definition of that distribution, performing some algebra to ensure it is in the right form.

First, since we will need it in the future of this proof, please recall

\begin{align}
S^2_y = \frac{1}{n-1} \sum_{i=1}^n\ \left(Y_i - \overline{Y}\right)^2
\end{align}

This is equivalent to

\begin{align}
(n-1)S_y^2 = \sum_{i=1}^n\ \left(Y_i - \overline{Y}\right)^2
\end{align}

Now, remember the definition of a Chi-square random variable:

\begin{align}
\sum_{i=1}^n \left( \frac{Y_i - \mu}{\sigma} \right)^2 \sim \chi^2_n \label{eq:appS-chisq2}
\end{align}

Note that this is very similar to our definition of \(S^2\). The difference is the presence of \(\mu\). So, for reasons that will become obvious later, let us subtract and add \(\overline{Y}\) to the numerator of \(\ref{eq:appS-chisq2}\), then expand the square:

\begin{align}
\chi^2_n &\sim \sum_{i=1}^n \left( \frac{Y_i - \overline{Y} + \overline{Y} - \mu}{\sigma} \right)^2 \\[1em]
&= \sum_{i=1}^n \left( \frac{(Y_i - \overline{Y}) + (\overline{Y} - \mu)}{\sigma} \right)^2 \\[1em]
&= \sum_{i=1}^n \left( \frac{Y_i - \overline{Y}}{\sigma}\right)^2 + \sum_{i=1}^n \left( \frac{\overline{Y} - \mu}{\sigma}\right)^2 + 2 \sum_{i=1}^n \left( \frac{(Y_i - \overline{Y})(\overline{Y}-\mu)}{\sigma} \right)
\end{align}

Note that the third term is zero:

\begin{align}
2\ \sum_{i=1}^n \left( \frac{(Y_i - \overline{Y})(\overline{Y}-\mu)}{\sigma} \right) &= 2\ \left( \frac{(\overline{Y}-\mu)}{\sigma} \right) \sum_{i=1}^n (Y_i - \overline{Y}) \\
&= 2\ \left( \frac{(\overline{Y}-\mu)}{\sigma} \right) \ 0 \\
&= 0
\end{align}

With that, we have:

\begin{align}
\chi^2_n &\sim \sum_{i=1}^n \left( \frac{Y_i - \overline{Y}}{\sigma}\right)^2 + \sum_{i=1}^n \left( \frac{\overline{Y} - \mu}{\sigma}\right)^2 \\
&= \frac{\sum_{i=1}^n (Y_i - \overline{Y})}{\sigma^2} + n \left( \frac{\overline{Y} - \mu}{\sigma}\right)^2 \\
&= \frac{(n-1) S^2}{\sigma^2} + n \left( \frac{\overline{Y} - \mu}{\sigma}\right)^2 \\
\text{And:} \qquad \chi^2_n &\sim \frac{(n-1) S^2}{\sigma^2} + \left( \frac{\overline{Y} - \mu}{\sigma/\sqrt{n}}\right)^2 \\
\end{align}

The second term is the square of a standard Normal distribution; that is, it follows a \(\chi^2_1\) distribution. Since the sum of two Chi-square distributions is another Chi-square distribution (with \(\nu\) being the sum of the degrees of freedom), we have our result:

\begin{equation}
\frac{(n-1) S^2}{\sigma^2} \sim \chi^2_{n-1}
\end{equation}

Note

Do you remember from the Chi-square section that the expected value of a Chi-square random variable is \(\nu\)? Use that to see why we needed to divide by \(n-1\) in the formula for the sample variance to ensure \(S^2\) is an unbiased estimator of \(\sigma^2\).

That is, you should be able to prove \(E[S^2] = \sigma^2\).

By the way, there is another proof using algebra, but it gives little insight into probability. Because of this, I forgo it. However, it is a really nice exercise in high school algebra and statistics definitions.

Exercise \(\PageIndex{1}\)

Prove \(E[S^2] = \sigma^2\).