Skip to main content
Statistics LibreTexts

Simple Linear Regression (with one predictor)

Model

\(X\) and \(Y\) are the predictor and response variables, respectively. Fit the model,
\[ Y_i = \beta_0+\beta_1X_i+\epsilon_i,      x = 1,2,...,n \]       
where \(  \epsilon_1 ,..., \epsilon_n \)  are uncorrelated,  \( E(\epsilon_1)=0, VAR(\epsilon_1)=\sigma^2 \).
 

Interpretation

Look at the scatter plot of \(Y\) (vertical axis) versus \(X\) (horizontal axis). Consider narrow vertical strips around the different values of \(X\): 
  1. Means (measure of center) of the points falling in the vertical strips lie (approximately) on a straight line with slope \(\beta_1\) and intercept \(\beta_0\).
  2. Standard deviations (measure of spread) of the points falling in each vertical strip are (roughly) the same.

Estimation of \(\beta_0 \) and \( \beta_1 \)

We employ the method of least squares to estimate \(\beta_0\) and \(\beta_1\).
This means, we minimize the sum of squared errors : \(Q(\beta_0,\beta_1) = \sum_{i=1}^n(Y_i-\beta_0-\beta_1X_i)^2\).
This involves differentiating \(Q(\beta_0,\beta_1)\) with respect to the parameters \(\beta_0\) and \(\beta_1\) and setting the derivatives to zero. This gives us the normal equations:
\[nb_0 + b_1\sum_{i=1}^nX_i = \sum_{i=1}^nY_i\]
\[b_0\sum_{i=1}^nX_i+b_1\sum_{i=1}^nX_i^2 = \sum_{i=1}^nX_iY_i\]
Solving these equations, we have:
\[b_1=\frac{\sum_{i=1}^nX_iY_i-n\overline{XY}}{\sum_{i=1}^nX_i^2-n\overline{X}^2} = \frac{\sum_{i=1}^n(X_i-\overline{X})(Y_i-\overline{y})}{\sum_{i=1}^n(X_i-\overline{X})^2},                  b_0 = \overline{Y}-b_1\overline{X}\]
 
\(b_0\) and \(b_1\) are the estimates of \(\beta_0\) and \(\beta_1\), respectively, and are sometimes denoted as \(\widehat\beta_0\) and \(\widehat\beta_1\).

Prediction

The fitted regression line is given by the equation:
\[\widehat{Y} = b_0 + b_1X\]
and is used to predict the value of \(Y\) given a value of \(X\).

Residuals

These are the quantities \(e_i = Y_i - \widehat{Y}_i = Y_i - (b_0 + b_1X_i)\), where \(\widehat{Y}_i = b_0 + b_1X_i\). Note that \(\epsilon_i = Y_i - \beta_0 - \beta_1X_i\). This means that \(e_i\)'s estimate \(\epsilon_i\)'s. Some properties of the regression line and residuals are :
  1. \(\sum_{i}e_i = 0\).
  2. \(\sum_{i}e_i^2 \leq \sum_{i}(Y_i - u_0 - u_1X_i)^2\) for any \((u_0, u_1)\) (with equality when \((u_0, u_1)\) = \((b_0, b_1)\)).
  3. \(\sum_{i}Y_i = \sum_{i}\widehat{Y}_i\).
  4. \(\sum_{i}X_ie_i = 0\).
  5. \(\sum_{i}\widehat{Y}_ie_i = 0\).
  6. Regression line passes through the point \((\overline{X},\overline{Y})\)
  7. The slope \(b_1\) of the regression line can be expressed as \(b_1 = r_{XY}\frac{sy}{sx}\), where \(r_{XY}\) is the correlation coefficient between \(X\) and \(Y\) and \(s_X\) and \(s_Y\) are the standard deviations of \(X\) and \(Y\).
 
Error sum of squares, deonted \(SSE\), is given by
\[SSE = \sum_{i=1}^ne_i^2 = \sum_{i=1}^n(Y_i - \overline{Y})^2 - b_1^2\sum_{i=1}^n(X_i-\overline{X})^2.\]

Estimation of \(\sigma^2\) 

It can be shown that \(E(SSE) = (n-2)\sigma^2.\) Therefore, \(\sigma^2\) is estimated by the mean squared error, i.e., \(MSE = \frac{SSE}{n-2}.\) Note also that this justifies the statement that the degree of freedom of the errors is \(n-2\) which is  sample size \((n)\) minus the number of regression coefficients (\(\beta_0\) and \(\beta_1\)) being estimated.

Contributors

  • Debashis Paul (UCD)
  • Scott Brunstein (UCD)