8.6: Nonresponse

Imagine that some friends who are all investment bankers are sitting at a trendy restaurant, hanging out, when one of them gets a phone call. They look at the number, they don't recognize it, but they go ahead and answer. The voice that comes through and asks politely if they would be interested in taking a survey about the economic conditions in their city. They decline and go back to talking with their friends. They never even give it another thought.

What happens for the researcher when this happens is more complicated. The researcher had carefully designed a study along with a plan for sampling people at random from the population. Certainly, the researcher could not have forced anyone to take the survey, and so there is nothing they can do. But now that someone has not responded, the sampling plan has been violated. The researcher has other responses, so does it make a difference?

The answer to that question depends on if the reason a participant does not respond to a survey is related to the issues being studied by the survey. The investment banker is a busy young person with many things going on. The survey they decided not to participate in was about economic conditions in the city. What type of person might take part in such a survey?

Investment bankers may be busy, but other people might not be. For example, it is well known that retired people often have more free time at their disposal and are more likely to take surveys. Do you think that these individuals may respond differently to the survey than someone who is too busy to take the survey? Remember, this survey is about economic conditions, and many retired persons are living on a fixed income. They may have very different views of economic conditions than a young investment banker. This creates a problem in that the survey may end up giving a skewed view of economic conditions if some people are overrepresented in the sample.

Earlier we discussed the presidential election of 1936 where we considered the methods used by the Literary Digest magazine. We looked at the political poll that predicted Alf Landon would beat Franklin Roosevelt. The poll was in error partially because the population from which the Literary Digest observed their data was not the same as the population they wanted to study. However, this poll also suffered from non-response. The Literary Digest sent out 10 million surveys, but only 2.4 million individuals participated in the survey.

Non-response is a common problem in surveys, and special studies have historically shown that the middle class tends to be overrepresented in typical surveys. Other research has shown that individuals who have strong feelings about a particular survey subject are more likely to participate. Hence, surveys on controversial issues are likely to have high response rate from individuals who have extreme views on the topic. Personal interviews have historically had better response rates, but non-response can still be a problem. Many modern surveys are implemented over social networks or other internet-based resources, and researchers are studying the demographic profiles of those who respond and do not respond to these surveys.

If non-response is a problem in surveys, what do researchers do to get reasonable results from their surveys? Statistical researchers have developed sophisticated methods to account for this bias. For example, if it is known that certain segments of the population tend not to respond to surveys, then respondents from those segments may be weighted more heavily in the results of the survey. The researchers need to be careful to figure out how much additional weight to give these responses. Similarly, even when an individual elects not to participate in a survey, some information may be known about them, such as demographic information. Researchers can use this information along with the remaining information about those who responded to the survey to attempt to guess how those individuals would have filled out the survey. These methods are known as imputation methods. Once again, these methods must be used very carefully.

Many surveys do not try to correct for non-response at all, and the results of these studies must be interpreted very cautiously. Even when the response rate is relatively high, non-response bias can have a large effect on the results when those who do not respond differ greatly from those who do respond.