3.4: PRE Measures

We have yet to introduce distribution assumptions on our data. Thus, we can still think about these results as mathematical. In a couple of chapters, we will make assumptions on the distribution of the residuals. However, until we do, we cannot speak about hypothesis testing and confidence intervals.

That will not stop us from continuing our exploration of our mathematical model (theory) and how it relates to the observed data (reality). This section introduces two measures of goodness-of-fit — measures of how well the model describes the reality. In Section 4.3, we will look at another measure and at what is happening in terms of the geometry of the problem.

✦•················• ✦ •··················•✦

Now that we have reality (\(y_i\)) and our errors (\(e_i\)), we can create a measure of how well the model summarizes ("fits") the data. In fact, we will create two of them! Both are termed "proportional reduction in error" measures; that is, they both are measures of how well the model reduces the unexplained variation in the dependent variable. The first is the venerable \(R^2\) ("R-squared") measure. The second is the \(\bar{R}^2\) ("adjusted R-squared" or "R-bar-squared") measure.

Both measure how much the model reduces the variation in the dependent variable. They differ in how that variation is measured. The \(R^2\) measure uses the sum-of-squares; the \(\bar{R}^2\), the variance.

\(R^2\) Measure

The formula for the \(R^2\) measure is

\begin{equation}
R^2 = 1 - \frac{SSE}{TSS}
\end{equation}

Here, \(SSE\) is the sum of the squared errors using the model (the error remaining), and \(TSS\) is the sum of squares without using the model (or with using the null model).

\begin{align}
SSE &= \sum\ (y_i - \hat{y})^2 \\
TSS &= \sum\ (y_i - \bar{y})^2
\end{align}

In this formula, \(\hat{y}\) is the predicted value of \(y\) for each value of \(x_i\) in the data according to the model; \(\bar{y}\) is the predicted value of \(y\) for each value of \(x_i\) in the data absent of the model (the mean of the dependent variable).

Thus:

• the \(SSE\) is a measure of how much variation remains in the model — the residual (unexplained) variation after applying the model.
• the \(TSS\) is a measure of the variation in the original data — the residual variation after applying the "null model."

Note

The "null model" always refers to the model with no independent variable. Thus, it is the model with only the y-intercept (here). The concept of the "null model" is extremely important in statistics, because it allows us to determine how much the model is a improvement over the "lack of" model.

Caution

The \(R^2\) measure only tells us how much of the variation in the dependent variable is described by the model (as compared to how much was there originally). It tells us nothing beyond that.

For instance, an \(R^2\) value of \(0.04\) tells us that the model explains only 4% of the variation in the dependent variable. There is a lot of variation left unexplained by the model. It does not mean that the model is "poor." An \(R^2\) value of \(0.98\) tells us that the model explains 98% of the variation in the dependent variable. It does not tell us that the model is "good."

Note that the formula for \(R^2\) is equivalent to
\begin{equation}
R^2 = 1 - \frac{\ \frac{\displaystyle 1}{\displaystyle n-1}\ SSE\ }{\ \frac{\displaystyle 1}{\displaystyle n-1}\ TSS\ }
\end{equation}

Note that the numerator of the fraction is an estimator for the unexplained variance — but using the wrong number of degrees of freedom (\(n-1\) instead of \(n-p\)). Thus, the \(R^2\) measure is how much the model reduces the unexplained variance, when the variance is estimated using a biased estimator.

\(\bar{R}^2\) Measure

Where \(R^2\) measures how much the model reduces the unexplained variance, when that variance is estimated using a biased estimate of the total variance, \(\bar{R}^2\) measures how much the model reduces the unexplained variance, when that variance is estimated using an unbiased estimate of the total variance. In other words, the adjusted \(R^2\) measure uses the correct degrees of freedom to describe the proportional reduction in error.

\begin{equation}
\bar{R}^2 \ = \ 1 - \frac{\frac{\displaystyle 1}{\displaystyle n-p}\ SSE}{\frac{\displaystyle 1}{\displaystyle n-1}\ TSS} \ = \ 1 - \frac{\displaystyle (n-1)\ SSE}{\displaystyle (n-p)\ TSS}
\end{equation}

Here, \(p\) is the number of parameters estimated in the model. For simple linear regression (one independent variable), \(p=2\), because we are calculating \(b_0\) and \(b_1\) from the data (to estimate \(\beta_0\) and \(\beta_1\)). For the null model, \(p=1\), because we are only calculating \(\bar{y}\) from the data (to estimate \(\beta_0\)).

Note

Usually, one reports the \({R}^2\) value and uses the \(\bar{R}^2\) to help with model selection (select the model with the larger \(\bar{R}^2\)). This, I believe, needs to change. It is the adjusted R-squared value that better estimates the proportion reduction in error. This is because the adjusted R-squared measure uses unbiased estimators of the variances.

One strength of the \({R}^2\) measure is that it ranges from \(0\) to \(1\). The \(\bar{R}^2\) measure can be less than \(0\). However, it is only less than \(0\) when the model describes very little of the variability in the dependent variable.

Other PREs

These are the two most frequently used PRE measures in linear regression. They are not the only ones, however. There is an entire class of PRE measures called "pseudo-\(R^2\)" measures. These are all genuine measures of how well the model helps reduce unexplained variation in the dependent variable. Their formulas tend to follow the structure of

\begin{equation*}
\text{PRE} = 1 - \frac{\text{variation in the dependent variable with\phantom{out} the model}}{\text{variation in the dependent variable with\emph{out} the model}}
\end{equation*}

Note

In the future, you will be introduced to "pseudo-\(R^2\) measures" for several different modeling schemes. Because these follow the same scheme as above, they have the same interpretation. They are measures of how well the model reduces the uncertainty in the dependent variable.

The differences are in how that uncertainty is measured.