3.3: Our First Assumptions

This was fun. We were able to determine the correct formula for the line of best fit — given our particular definition of "best." Those equations lead to other equations. These are the mathematical results for our given sample.

Cool mule.

✦•················• ✦ •··················•✦

Until this point, we have only required variation in the independent variable. If we make three additional assumptions, we have additional results.

Note

This is how mathematical statistics progresses. Assumptions are made, then we play with the equations to learn about the consequences of those assumptions. Then, when we have exhausted our efforts, we make additional assumptions... ad infinitum

Recall that the data model is

\begin{equation}
y = \beta_0 + \beta_1 x + \varepsilon \label{eq:dataModel}
\end{equation}

With this, here are the three assumptions we will make, all about the residuals:

• The first assumption is that the residuals are realizations of a random variable (\(\varepsilon\) has a distribution).
• The second is that the expected value of the residuals is zero, \(E[\varepsilon]=0\) (the measurements are not systematically biased).
• The third is that the residuals are independent and have a finite and constant variance, \(V[\varepsilon]=\sigma^2 < \infty\) (the residuals are homoskedastic).

The above simple assumptions lead to several additional interesting results. Some are proven here, some are left as exercises.

Theorem \(\PageIndex{1}\)

The OLS estimator for \(\beta_1\) is unbiased. That is,

\begin{equation}
E[b_1] = \beta_1
\end{equation}

Proof:
To prove this, we will start with the formula for \(b_1\) and simplify until we obtain the results.

\begin{align}
E[b_1] &= E\left[ \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}\right] \\[1em]
&= E\left[ \frac{\sum_{i=1}^n (x_i - \bar{x})y_i}{\sum_{i=1}^n (x_i - \bar{x})^2} \right] \\[1em]
&= \frac{\sum_{i=1}^n (x_i - \bar{x})E[y_i]}{\sum_{i=1}^n (x_i - \bar{x})^2} \\[1em]
&= \frac{\sum_{i=1}^n (x_i - \bar{x})\left(\beta_0 + x_i\beta_1 + \varepsilon\right)}{\sum_{i=1}^n (x_i - \bar{x})^2} \\[1em]
%
&= \frac{\sum_{i=1}^n (x_i - \bar{x})\beta_0}{\sum_{i=1}^n (x_i - \bar{x})^2} + \frac{\sum_{i=1}^n (x_i - \bar{x}) x_i\beta_1 }{\sum_{i=1}^n (x_i - \bar{x})^2} + \frac{\sum_{i=1}^n (x_i - \bar{x})E[\varepsilon]}{\sum_{i=1}^n (x_i - \bar{x})^2} \\[1em]
%
&= \frac{\beta_0\sum_{i=1}^n (x_i - \bar{x})}{\sum_{i=1}^n (x_i - \bar{x})^2} + \frac{\beta_1 \sum_{i=1}^n (x_i - \bar{x})x_i}{\sum_{i=1}^n (x_i - \bar{x})^2} + \frac{E[\varepsilon] \sum_{i=1}^n (x_i - \bar{x})}{\sum_{i=1}^n (x_i - \bar{x})^2} \\[1em]
%
&= \beta_0\ \frac{0}{\sum_{i=1}^n (x_i - \bar{x})^2} + \beta_1\ \frac{\sum_{i=1}^n (x_i - \bar{x})(x_i - \bar{x})}{\sum_{i=1}^n (x_i - \bar{x})^2} + 0\ \frac{0}{\sum_{i=1}^n (x_i - \bar{x})^2} \\[1em]
%
&= \beta_1
\end{align}

Thus, the OLS estimator of \(\beta_1\) is unbiased. This is a nice property. It means \(E[b_1] = \beta_1\).

Note

Did you notice where we used these three results?

\begin{align}
& \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) = \sum_{i=1}^n (x_i - \bar{x})y_i \text{,} \\
& \sum_{i=1}^n (x_i - \bar{x})(x_i - \bar{x}) = \sum_{i=1}^n (x_i - \bar{x})x_i \text{, and} \\
& \sum_{i=1}^n (x_i - \bar{x}) = 0
\end{align}

All three are just simple algebra.

───── ⋆⋅☆⋅⋆ ─────

Here are a few other results for you to prove:

Theorem \(\PageIndex{2}\)

\( E[\beta_0] = \beta_0 \)

Theorem \(\PageIndex{3}\)

\( V[b_1] = \frac{\displaystyle \sigma^2}{\displaystyle S_{xx}} \)

Theorem \(\PageIndex{4}\)

\( V[b_0] = \sigma^2\left(\frac{\displaystyle 1}{\displaystyle n} + \frac{\displaystyle \bar{x}}{\displaystyle S_{xx}}\right) \)

Theorem \(\PageIndex{5}\)

\( Cov[b_0, b_1] = -\sigma^2 \frac{\displaystyle \bar{x}}{\displaystyle S_{xx}} \)

───── ⋆⋅☆⋅⋆ ─────

Finally, let us define the mean square error (MSE) in the case of simple linear regression, SLR (one dependent and one independent variable).

Definition: Mean Square Error (MSE)

\( \text{MSE} = \frac{\displaystyle 1}{\displaystyle n-2}\ \displaystyle \sum_{i=1}^n e_i^2 \)

Theorem \(\PageIndex{6}\)

\( E[\text{MSE}] = \sigma^2 \)

In other words, the definition of the MSE provides an unbiased estimator of the variance of the residuals. This is why we define it in this manner.

Note

Be aware that the above definition of the MSE only holds in the case of simple linear regression (SLR); that is, it holds when there is just one dependent and one independent variable. It can be shown that the general definition of the MSE is

\( \text{MSE} = \frac{\displaystyle 1}{\displaystyle n-p}\ \displaystyle \sum_{i=1}^n e_i^2 \)

Here, p is the number of parameters being estimated, which is usually the number of independent variables plus the constant term.