8.4: Inference for Linear Regression

In this section, we discuss uncertainty in the estimates of the slope and y-intercept for a regression line. Just as we identified standard errors for point estimates in previous chapters, we first discuss standard errors for these new estimates.

Midterm elections and unemployment

Elections for members of the United States House of Representatives occur every two years, coinciding every four years with the U.S. Presidential election. The set of House elections occurring during the middle of a Presidential term are called . In America’s two-party system, one political theory suggests the higher the unemployment rate, the worse the President’s party will do in the midterm elections.

To assess the validity of this claim, we can compile historical data and look for a connection. We consider every midterm election from 1898 to 2018, with the exception of those elections during the Great Depression. Figure [unemploymentAndChangeInHouse] shows these data and the least-squares regression line:

\[\begin{aligned} &\text{\% change in House seats for President's party} \\ &\qquad\qquad= -7.36 - 0.89 \times \text{(unemployment rate)}\end{aligned}\]

We consider the percent change in the number of seats of the President’s party (e.g. percent change in the number of seats for Republicans in 2018) against the unemployment rate.

Examining the data, there are no clear deviations from linearity, the constant variance condition, or substantial outliers. While the data are collected sequentially, a separate analysis was used to check for any apparent correlation between successive observations; no such correlation was found.

The data for the Great Depression (1934 and 1938) were removed because the unemployment rate was 21% and 18%, respectively. Do you agree that they should be removed for this investigation? Why or why not?

There is a negative slope in the line shown in Figure [unemploymentAndChangeInHouse]. However, this slope (and the y-intercept) are only estimates of the parameter values. We might wonder, is this convincing evidence that the “true” linear model has a negative slope? That is, do the data provide strong evidence that the political theory is accurate, where the unemployment rate is a useful predictor of the midterm election? We can frame this investigation into a statistical hypothesis test:

• \(\beta_1 = 0\). The true linear model has slope zero.
• \(\beta_1 \neq 0\). The true linear model has a slope different than zero. The unemployment is predictive of whether the President’s party wins or loses seats in the House of Representatives.

We would reject \(H_0\) in favor of \(H_A\) if the data provide strong evidence that the true slope parameter is different than zero. To assess the hypotheses, we identify a standard error for the estimate, compute an appropriate test statistic, and identify the p-value.

Understanding regression output from software

Just like other point estimates we have seen before, we can compute a standard error and test statistic for \(b_1\). We will generally label the test statistic using a \(T\), since it follows the \(t\)-distribution.

We will rely on statistical software to compute the standard error and leave the explanation of how this standard error is determined to a second or third statistics course. Figure [midtermUnempRegTable] shows software output for the least squares regression line in Figure [unemploymentAndChangeInHouse]. The row labeled unemp includes the point estimate and other hypothesis test information for the slope, which is the coefficient of the unemployment variable.

	Estimate	Std. Error	t value	Pr(\(>\)\(|\)t\(|\))
(Intercept)	-7.3644	5.1553	-1.43	0.1646
unemp	-0.8897	0.8350	-1.07	0.2961

What do the first and second columns of Figure [midtermUnempRegTable] represent? The entries in the first column represent the least squares estimates, \(b_0\) and \(b_1\), and the values in the second column correspond to the standard errors of each estimate. Using the estimates, we could write the equation for the least square regression line as

\[\begin{aligned} \hat{y} = -7.3644 - 0.8897 x \end{aligned}\]

where \(\hat{y}\) in this case represents the predicted change in the number of seats for the president’s party, and \(x\) represents the unemployment rate.

We previously used a \(t\)-test statistic for hypothesis testing in the context of numerical data. Regression is very similar. In the hypotheses we consider, the null value for the slope is 0, so we can compute the test statistic using the T (or Z) score formula:

\[\begin{aligned} T = \frac{\text{estimate} - \text{null value}}{\text{SE}} = \frac{-0.8897 - 0}{0.8350} = -1.07\end{aligned}\]

This corresponds to the third column of Figure [midtermUnempRegTable].

Use the table in Figure [midtermUnempRegTable] to determine the p-value for the hypothesis test. The last column of the table gives the p-value for the two-sided hypothesis test for the coefficient of the unemployment rate: 0.2961. That is, the data do not provide convincing evidence that a higher unemployment rate has any correspondence with smaller or larger losses for the President’s party in the House of Representatives in midterm elections.

Inference for regression We usually rely on statistical software to identify point estimates, standard errors, test statistics, and p-values in practice. However, be aware that software will not generally check whether the method is appropriate, meaning we must still verify conditions are met.

Examine Figure [elmhurstScatterWLSROnly], which relates the Elmhurst College aid and student family income. How sure are you that the slope is statistically significantly different from zero? That is, do you think a formal hypothesis test would reject the claim that the true slope of the line should be zero? [overallAidIncomeInfAssessOfRegrLineSlope] While the relationship between the variables is not perfect, there is an evident decreasing trend in the data. This suggests the hypothesis test will reject the null claim that the slope is zero.

Figure [rOutputForIncomeAidLSRLineInInferenceSection] shows statistical software output from fitting the least squares regression line shown in Figure [elmhurstScatterWLSROnly]. Use this output to formally evaluate the following hypotheses.

• The true coefficient for family income is zero.
• The true coefficient for family income is not zero.

	Estimate	Std. Error	t value	Pr(\(>\)\(|\)t\(|\))
(Intercept)	24319.3	1291.5	18.83	\(<\)0.0001
familyincome	-0.0431	0.0108	-3.98	0.0002

Confidence interval for a coefficient

Similar to how we can conduct a hypothesis test for a model coefficient using regression output, we can also construct a confidence interval for that coefficient.

Compute the 95% confidence interval for the coefficient using the regression output from Table [rOutputForIncomeAidLSRLineInInferenceSection]. The point estimate is -0.0431 and the standard error is \(SE = 0.0108\). When constructing a confidence interval for a model coefficient, we generally use a \(t\)-distribution. The degrees of freedom for the distribution are noted in the regression output, \(df = 48\), allowing us to identify \(t_{48}^{\star} = 2.01\) for use in the confidence interval.

We can now construct the confidence interval in the usual way:

\[\begin{aligned} \text{point estimate} \pm t_{48}^{\star} \times SE \qquad\to\qquad -0.0431 \pm 2.01 \times 0.0108 \qquad\to\qquad (-0.0648, -0.0214) \end{aligned}\]

We are 95% confident that with each dollar increase in , the university’s gift aid is predicted to decrease on average by $0.0214 to $0.0648.

Confidence intervals for coefficients Confidence intervals for model coefficients can be computed using the \(t\)-distribution:

\[\begin{aligned} b_i \ \pm\ t_{df}^{\star} \times SE_{b_{i}} \end{aligned}\]

where \(t_{df}^{\star}\) is the appropriate \(t\)-value corresponding to the confidence level with the model’s degrees of freedom.

On the topic of intervals in this book, we’ve focused exclusively on confidence intervals for model parameters. However, there are other types of intervals that may be of interest, including prediction intervals for a response value and also confidence intervals for a mean response value in the context of regression. These two interval types are introduced in an online extra that you may download at

---

1. Formally, we can compute the correlation for observations \((x_1, y_1)\), \((x_2, y_2)\), ..., \((x_n, y_n)\) using the formula

   \[\begin{aligned} R = \frac{1}{n-1} \sum_{i=1}^{n} \frac{x_i-\bar{x}}{s_x}\frac{y_i-\bar{y}}{s_y} \end{aligned}\]

   where \(\bar{x}\), \(\bar{y}\), \(s_x\), and \(s_y\) are the sample means and standard deviations for each variable.↩

2. There are applications where the sum of residual magnitudes may be more useful, and there are plenty of other criteria we might consider. However, this book only applies the least squares criterion.↩