4.3: Practicality

Section 4.2 considered a hypothetical study of the differences in income across classifications of immigration status. As the discussion progressed, the definition of the population of interest was refined in the interest of practicality. Practicality is a key issue in statistical studies. Populations can be large, complex, and members of the population may be difficult to contact. Practitioners who study populations also have limited practical means due to many unavoidable constraints such as cost, time, and accessibility. Additionally, as we will see later, it is important that each member of the population is accessible for observation to the researcher, even though the researcher may only look at a small part of a population.

Refining a population definition has a consequential effect on the applicability of the results of the study. In the hypothetical study of the differences in income across classifications of immigration status, it was indicated that the study covered the population of residents in a specific city. When the research study is completed, the results of the study would apply only to those individuals who were in the population of interest. Therefore, if the study concluded that a certain immigration status has a negative effect on income, those conclusions could not automatically be assumed true for other cities—unless it could be argued that the populations and the social and employment environments of the two cities are roughly the same.

Similarly, in the study of the effect a specific teaching approach has on students with various gender identities, the results are narrowly focused on what happened in those two classes. Attempting to generalize this study to the application of these teaching methods is difficult, as changing any of the teaching parameters may influence the student performance. It is quite difficult for a study of this nature to fully attribute any observed differences in the performance of the students to the teaching methodology unless great care is taken to control for as many differences between the two classes as is possible. For example, one could argue that there was a difference in performance because the instructor believed in one of the approaches more than the other and did a more competent job for that class. Therefore, attempting to imply that the results support the general notion that a change in teaching method would result in better student performance is tenuous at best, unless a more substantial justification accompanies the analysis.

With the applicability of the results in mind, there is a fundamental interest in considering as large a population as is possible so that the results of the corresponding study will have the widest applicability. For example, in the hypothetical example of income and immigration status, a study that considered the entire population of the United States would have wide applicability. If that type of study were possible, the results could be applicable across the country and could be the basis for legislative and social policy action. In the hypothetical study of teaching styles, a population that is focused on multiple instructors, departments, class subjects, and even universities would have the potential for generalizing the applicability of the results to broader populations.

So, what prevents researchers from always considering the largest possible population with the largest possible applicability for every study? The problem, as we will study in greater detail later, is that researchers need to have the ability to observe data from each of the individuals or items in the population. We will usually not observe all the members of a population, but we need to be able to have that opportunity.

The size and location of a population can influence the potential observability of the members of the population. The hypothetical study of income and immigration status can serve as a good example of the problems that can occur with large populations. First consider the study income and immigration status with the intention of considering the entire population of the United States. What we need to keep in mind is that we would at least, hypothetically, need to be able to observe all the members of this population. So, how could we do that? Do we have a list of everyone employed, their immigration status, and how to get in contact with them? Unless we work for a large government agency, it would be difficult to imagine that we do. Even if we did have an official list of those employed in the United States, would it really include everyone? Again, this is very unlikely, as there are many who work in jobs that are paid in ways that are not easily traceable even by the government. Therefore, we may not really know about these individuals and, most importantly, these workers may be people with certain immigration statuses. Even more importantly, because of the nature of this type of work, the pay may be lower than that of the rest of the population. Unfortunately, this is the very question we are attempting to answer. Hence, if these individuals are systematically left out of the study, then we may get a skewed idea about the incomes and how they relate to immigration status.

Getting in touch with individuals who would be unofficially employed, perhaps with an immigration status that would make them less likely to share personal information, requires a very detailed knowledge of the area and of the people that work in the area. Gathering data from these individuals also requires establishing a trust-based relationship. This type of knowledge and these relationships are time consuming and resource intensive to develop. Hence, unless a researcher has access to local knowledge, along with the resources and relationships required to work with such populations, a study with broad based applicability may not be feasible in practice.

In this case, researchers may elect to work with a smaller population, or at least a more local population to save on resources and time. While this tactic has a fundamental effect on how widely the results of the study can be interpreted, the alternative is often not to do the study at all. In this case localized results may be the best option for obtaining some useful information with regards to the study hypotheses.