Skip to main content
Statistics LibreTexts

27: The General Linear Model in R

  • Page ID
    7666
  • 27 The General Linear Model in R

    27.1 Linear regression (Section 26.1)

    To perform linear regression in R, we use the lm() function. Let’s generate some data and use this function to compute the linear regression solution.

    file113.png

    We can then apply lm() to these data:

    ## 
    ## Call:
    ## lm(formula = y ~ x, data = regression_data)
    ## 
    ## Residuals:
    ##     Min      1Q  Median      3Q     Max 
    ## -1.5563 -0.3042 -0.0059  0.3804  1.2522 
    ## 
    ## Coefficients:
    ##             Estimate Std. Error t value Pr(>|t|)    
    ## (Intercept)   9.9761     0.0580  172.12  < 2e-16 ***
    ## x             0.3725     0.0586    6.35  6.6e-09 ***
    ## ---
    ## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
    ## 
    ## Residual standard error: 0.58 on 98 degrees of freedom
    ## Multiple R-squared:  0.292,  Adjusted R-squared:  0.284 
    ## F-statistic: 40.4 on 1 and 98 DF,  p-value: 6.65e-09

    We should see three things in the lm() results:

    • The estimate of the Intercept in the model should be very close to the intercept that we specified
    • The estimate for the x parameter should be very close to the slope that we specified
    • The residual standard error should be roughly similar to the noise standard deviation that we specified

    27.2 Model criticism and diagnostics (Section 26.5)

    Once we have fitted the model, we want to look at some diagnostics to determine whether the model is actually fitting properly. We can do this using the autoplot() function from the ggfortify package.

    file114.png

    The left panel in this plot shows the relationship between the predicted (or “fitted”) values and the residuals. We would like to make sure that there is no clear relationship between these two (as we will see below). The right panel shows a Q-Q plot, which helps us assess whether the residuals from the model are normally distributed. In this case, they look reasonably normal, as the points don’t differ too much from the unit line.

    27.3 Examples of problematic model fit

    Let’s say that there was another variable at play in this dataset, which we were not aware of. This variable causes some of the cases to have much larger values than others, in a way that is unrelated to the X variable. We play a trick here using the seq() function to create a sequence from zero to one, and then threshold those 0.5 (in order to obtain half of the values as zero and the other half as one) and then multiply by the desired effect size:

    ## 
    ## Call:
    ## lm(formula = y2 ~ x, data = regression_data)
    ## 
    ## Residuals:
    ##     Min      1Q  Median      3Q     Max 
    ## -2.3324 -0.9689 -0.0939  1.0421  2.2591 
    ## 
    ## Coefficients:
    ##             Estimate Std. Error t value Pr(>|t|)    
    ## (Intercept)   10.978      0.117   93.65   <2e-16 ***
    ## x              0.270      0.119    2.27    0.025 *  
    ## ---
    ## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
    ## 
    ## Residual standard error: 1.2 on 98 degrees of freedom
    ## Multiple R-squared:  0.0501, Adjusted R-squared:  0.0404 
    ## F-statistic: 5.17 on 1 and 98 DF,  p-value: 0.0252

    One thing you should notice is that the model now fits overall much worse; the R-squared is about half of what it was in the previous model, which reflects the fact that more variability was added to the data, but it wasn’t accounted for in the model. Let’s see if our diagnostic reports can give us any insight:

    file115.png

    The residual versus fitted graph doesn’t give us much insight, but we see from the Q-Q plot that the residuals are diverging quite a bit from the unit line.

    Let’s look at another potential problem, in which the y variable is nonlinearly related to the X variable. We can create these data by squaring the X variable when we generate the Y variable:

    ## 
    ## Call:
    ## lm(formula = y3 ~ x, data = regression_data)
    ## 
    ## Residuals:
    ##    Min     1Q Median     3Q    Max 
    ## -1.610 -0.568 -0.065  0.359  3.266 
    ## 
    ## Coefficients:
    ##             Estimate Std. Error t value Pr(>|t|)    
    ## (Intercept)  10.5547     0.0844  125.07   <2e-16 ***
    ## x            -0.0419     0.0854   -0.49     0.62    
    ## ---
    ## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
    ## 
    ## Residual standard error: 0.84 on 98 degrees of freedom
    ## Multiple R-squared:  0.00245,    Adjusted R-squared:  -0.00773 
    ## F-statistic: 0.241 on 1 and 98 DF,  p-value: 0.625

    Now we see that there is no significant linear relationship between X2X^2 and Y/ But if we look at the residuals the problem with the model becomes clear:

    file116.png

    In this case we can see the clearly nonlinear relationship between the predicted and residual values, as well as the clear lack of normality in the residuals.

    As we noted in the previous chapter, the “linear” in the general linear model doesn’t refer to the shape of the response, but instead refers to the fact that model is linear in its parameters — that is, the predictors in the model only get multiplied the parameters (e.g., rather than being raised to a power of the parameter). Here is how we would build a model that could account for the nonlinear relationship:

    ## 
    ## Call:
    ## lm(formula = y3 ~ x + x_squared, data = regression_data)
    ## 
    ## Residuals:
    ##     Min      1Q  Median      3Q     Max 
    ## -1.4101 -0.3791 -0.0048  0.3908  1.4437 
    ## 
    ## Coefficients:
    ##             Estimate Std. Error t value Pr(>|t|)    
    ## (Intercept)  10.1087     0.0739   136.8   <2e-16 ***
    ## x            -0.0118     0.0600    -0.2     0.84    
    ## x_squared     0.4557     0.0451    10.1   <2e-16 ***
    ## ---
    ## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
    ## 
    ## Residual standard error: 0.59 on 97 degrees of freedom
    ## Multiple R-squared:  0.514,  Adjusted R-squared:  0.504 
    ## F-statistic: 51.2 on 2 and 97 DF,  p-value: 6.54e-16

    Now we see that the effect of X2X^2 is significant, and if we look at the residual plot we should see that things look much better:

    file117.png Not perfect, but much better than before!

    27.4 Extending regression to binary outcomes.

    Let’s say that we have a blood test (which is often referred to as a biomarker) and we want to know whether it predicts who is going to have a heart attack within the next year. We will generate a synthetic dataset for a population that is at very high risk for a heart attack in the next year.

    ## Observations: 1,000
    ## Variables: 3
    ## $ biomarker   <dbl> 1.15, 0.68, 1.21, -0.72, -1.00, -0.12…
    ## $ risk        <dbl> 1.054, -0.529, 0.675, -0.474, -1.398,…
    ## $ heartattack <lgl> TRUE, FALSE, TRUE, FALSE, FALSE, TRUE…

    Now we would like to build a model that allows us to predict who will have a heart attack from these data. However, you may have noticed that the heartattack variable is a binary variable; because linear regression assumes that the residuals from the model will be normally distributed, and the binary nature of the data will violate this, we instead need to use a different kind of model, known as a logistic regression model, which is built to deal with binary outcomes. We can fit this model using the glm() function:

    ## 
    ## Call:
    ## glm(formula = heartattack ~ biomarker, family = binomial(), data = disease_df)
    ## 
    ## Deviance Residuals: 
    ##     Min       1Q   Median       3Q      Max  
    ## -2.1301  -1.0150   0.0305   1.0049   2.1319  
    ## 
    ## Coefficients:
    ##             Estimate Std. Error z value Pr(>|z|)    
    ## (Intercept) -0.00412    0.06948   -0.06     0.95    
    ## biomarker    0.99637    0.08342   11.94   <2e-16 ***
    ## ---
    ## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
    ## 
    ## (Dispersion parameter for binomial family taken to be 1)
    ## 
    ##     Null deviance: 1386.3  on 999  degrees of freedom
    ## Residual deviance: 1201.4  on 998  degrees of freedom
    ## AIC: 1205
    ## 
    ## Number of Fisher Scoring iterations: 3

    This looks very similar to the output from the lm() function, and it shows us that there is a significant relationship between the biomarker and heart attacks. The model provides us with a predicted probability that each individual will have a heart attack; if this is greater than 0.5, then that means that the model predicts that the individual is more likely than not to have a heart attack.
    We can start by simply comparing those predictions to the actual outcomes.

    ## 
    ##  
    ##    Cell Contents
    ## |-------------------------|
    ## |                       N |
    ## |           N / Col Total |
    ## |-------------------------|
    ## 
    ##  
    ## Total Observations in Table:  1000 
    ## 
    ##  
    ##                       | disease_df$heartattack 
    ## disease_df$prediction |     FALSE |      TRUE | Row Total | 
    ## ----------------------|-----------|-----------|-----------|
    ##                 FALSE |       332 |       157 |       489 | 
    ##                       |     0.664 |     0.314 |           | 
    ## ----------------------|-----------|-----------|-----------|
    ##                  TRUE |       168 |       343 |       511 | 
    ##                       |     0.336 |     0.686 |           | 
    ## ----------------------|-----------|-----------|-----------|
    ##          Column Total |       500 |       500 |      1000 | 
    ##                       |     0.500 |     0.500 |           | 
    ## ----------------------|-----------|-----------|-----------|
    ## 
    ## 

    This shows us that of the 500 people who had heart attacks, the model corrected predicted a heart attack for 343 of them. It also predicted heart attacks for 168 people who didn’t have them, and it failed to predict a heart attack for 157 people who had them. This highlights the distinction that we mentioned before between statistical and practical significance; even though the biomarker shows a highly significant relationship to heart attacks, it’s ability to predict them is still relatively poor. As we will see below, it gets even worse when we try to generalize this to a new group of people.

    27.5 Cross-validation (Section 26.6.1)

    Cross-validation is a powerful technique that allows us to estimate how well our results will generalize to a new dataset. Here we will build our own crossvalidation code to see how it works, continuing the logistic regression example from the previous section.

    In cross-validation, we want to split the data into several subsets and then iteratively train the model while leaving out each subset (which we usually call folds) and then test the model on that held-out fold Let’s write our own code to do this splitting; one relatively easy way to this is to create a vector that contains the fold numbers, and then randomly shuffle it to create the fold assigments for each data point.

    Now let’s look at the performance of the model:

    ## 
    ##  
    ##    Cell Contents
    ## |-------------------------|
    ## |                       N |
    ## |           N / Col Total |
    ## |-------------------------|
    ## 
    ##  
    ## Total Observations in Table:  1000 
    ## 
    ##  
    ##                   | disease_df$heartattack 
    ## disease_df$CVpred |     FALSE |      TRUE | Row Total | 
    ## ------------------|-----------|-----------|-----------|
    ##             FALSE |       416 |       269 |       685 | 
    ##                   |     0.832 |     0.538 |           | 
    ## ------------------|-----------|-----------|-----------|
    ##              TRUE |        84 |       231 |       315 | 
    ##                   |     0.168 |     0.462 |           | 
    ## ------------------|-----------|-----------|-----------|
    ##      Column Total |       500 |       500 |      1000 | 
    ##                   |     0.500 |     0.500 |           | 
    ## ------------------|-----------|-----------|-----------|
    ## 
    ## 

    Now we see that the model only accurately predicts less than half of the heart attacks that occurred when it is predicting to a new sample. This tells us that this is the level of prediction that we could expect if were to apply the model to a new sample of patients from the same population.