14.4: The Gaussian Distribution

To illustrate what we did in the previous sections, let us apply what we know to the Gaussian distribution, and show that it is exponential class. This will allow us to determine the canonical link, the expected value, and the variance. Hopefully, the results will not surprise us.

 

Learning Objectives

By the end of this section, you will be able to:

1. Demonstrate that the Gaussian (normal) distribution is a member of the exponential family by rewriting its probability density function in the standard GLM form.
2. Identify the components of the Gaussian distribution within this framework: the natural parameter, the dispersion parameter, the cumulant function, and the term.
3. Derive the canonical link function for the Gaussian distribution as the identity link (\(g(\mu) = \mu\)), and confirm that the expected value and the variance match the known properties of the Normal distribution.
4. Recognize that the GLM framework with the Gaussian distribution and identity link reproduces the Classical Linear Model, illustrating how GLM generalizes the CLM to handle other types of response variables.

 

✦•················• ✦ •··················•✦

 

The Gaussian is Exponential Class

We start with the probability density function (pdf) of the Gaussian.

\begin{equation}
f(y) = \frac{1}{\sqrt{2 \pi \sigma^2}}\ \exp \left[ - \frac{(y - \mu)^2}{2 \sigma^2} \right] \\[1em]
\end{equation}

Now, to write this in standard form, we use algebra and some logarithm rules.

\begin{align}
f(y) & = \exp \left[ - \frac{(y - \mu)^2}{2 \sigma^2} + \log \left( \frac{1}{\sqrt{2 \pi \sigma^2}} \right) \right] \\[2em]
& = \exp \left[ - \frac{y^2 -2y\mu + \mu^2}{2 \sigma^2} + \log \left( \frac{1}{\sqrt{2 \pi \sigma^2}} \right) \right] \\[2em]
& = \exp \left[ - \frac{y^2}{2 \sigma^2} + \frac{y\mu}{\sigma^2} - \frac{\mu^2}{2 \sigma^2} + \log \left( \frac{1}{\sqrt{2 \pi \sigma^2}} \right) \right] \\[2em]
& = \exp \left[ \frac{y\mu -\frac{1}{2} \mu^2}{\sigma^2} + \log \left( \frac{1}{\sqrt{2 \pi \sigma^2}} \right) - \frac{y^2}{2 \sigma^2} \right]
\end{align}

Recall from the previous section that the "standard form" is

\begin{equation}
f(y) = \exp \left[ \frac{y \theta - b(\theta)}{a(\phi)} + c(y,\phi) \right]
\end{equation}

Comparing the two distributions gives us the following:

• \(y = y\)
• \(\theta = \mu\)
• \(a(\phi)=\sigma^2\)
• \(b(\theta) = \frac{1}{2} \mu^2 = \frac{1}{2} \theta^2\)
• \(c(y, \phi)= \log \left( \frac{1}{\sqrt{2 \pi \sigma^2}} \right) - \frac{y^2}{2 \sigma^2}\)

 

Thus, according to this list, the canonical link is \(g(\mu) = \mu\), also known as the identity function.

 

The dispersion parameter is \(a(\phi) = \sigma^2\).

 

Also note that the expected value is

\begin{align}
E[Y] & = b^\prime\!(\theta) \\[1em]
&= \frac{\text{d}}{\text{d}\theta} \left( \frac{1}{2} \theta^2 \right) \\[1em]
&= \theta \\[1em]
&= \mu
\end{align}

Hopefully, this is as we expect.

 

Finally, note that the variance is

\begin{align}
V[Y] & = b^{\prime\prime}\!(\theta) a(\phi) \\[1em]
&= \frac{\text{d}^2}{\text{d}\theta^2} \left( \frac{1}{2} \theta^2 \sigma^2 \right)\\[1em]
&= \frac{\text{d}}{\text{d}\theta} \left( \theta \sigma^2 \right)\\[1em]
&= \sigma^2
\end{align}

Also as we expect, hopefully.

 

Other Link Functions

While the canonical link is the identity function (\(\eta = \mu\)), it is not the only allowable link function. In Section 8.1: The Issue of Boundedness, we transformed the continuous dependent variable because it was bounded below by (but never equaled) zero. In such a case, the logarithm is an appropriate link function: The dependent variable has a restricted range. The link function converts that range to an unbounded range. The same is true under the GLM framework. Similarly, the logit function is frequently an appropriate link function, as it was in Section 8.1: The Issue of Boundedness.

With that, we start to see that for continuous dependent variables, what we did under the CLM paradigm we can do under the GLM paradigm. This is always true; the GLM paradigm extends the CLM paradigm to handle different classes of dependent variables.