4.6: Critical Analysis

When considering a critical analysis of a research study, it is important to first consider the population being studied. A good research study will clearly identify the population, including details on the size of the population, its accessibility to the researchers, and information on why the population is relevant to the research study.

If the researchers are accessing the population themselves, and not relying on an established database, then they should consider the relation between the authors and the population. A small research group considering a large and complex population may be questionable, unless there are specific reasons, such as funding from a government agency, that would allow them to have the resources to conduct such research. Such problems are somewhat alleviated using internet-based surveys, which are very inexpensive and relatively simple to deploy. However, as we shall discuss later, internet-based surveys can be misleading, unless they are carefully designed.

Larger research groups and institutions may have additional resources and experience that allow them to study large and complex questions more easily. That does not necessarily mean that you can automatically trust studies from these types of groups, and you will still need to carefully consider how well the population is defined and the methods used to access the population. Correspondingly, you should not automatically distrust research performed by smaller and perhaps less established research groups. You might find that they have worked very hard to define a population for which they have access. The key is to assess, to the best of your ability, how well the population is defined and whether it appears that the researchers had reasonable access to the population.

If the researchers are using an established dataset, you should first assess whether the population defined by those who obtained the data is well defined. The emphasis in this case is whether the population described by the originators of the data matches the population of interest in the study. If these two population definitions do not match, then the authors of the study are not studying the population that they claim to be studying. In many cases, the population of interest for the study is a subset of the population used to establish the dataset. The researchers will use only a part of the established dataset, excluding cases that are not within their defined population. This type of sub-setting of data is common and is usually used to reduce the number of potential confounding factors in the study. However, when assessing such a study, one should consider what bias might result from using a subset of the population. Subsets of a population can have very different characteristics than the entire population. Hence, the authors should provide a clear explanation as to why a subset of the data has been used, and what limitations and biases may result.

Next, if the observations are based on an established dataset, one should critically assess the source of the data. The source can be very important in establishing a critical analysis of a study. Data obtained from government institutions have usually been gathered using peer-reviewed techniques. Additionally, federal and state governments usually have efficient resources to obtain reliable data. Government agencies also have access to government databases to which private agencies will not have access. Private foundations and companies will also sometimes provide useful data, but one should be careful in assessing these types of studies. You should carefully consider the possibility that a private entity, even if it is a nonprofit organization, might have specific motives for presenting or gathering data in a certain way. It is also worthwhile to check the affiliations of the authors of a study to see if they are in any way connected with the private organization that obtained the data. The main concern with this type of data is assessing whether the source of the data has potential interests in the outcome of the study.

One of the important concepts central to the modern scientific method is reproducibility. That is, other scientists should be able to replicate the results of any experiment on their own. Such reproducibility can be assured by granting public access to data, allowing other researchers to explore the same data and replicate any reported analyses of the data. Recently many research journals began to require either public access to research data, or instructions on how the data can be obtained. Access is not always possible, as some data may be proprietary or may contain sensitive personal information. While you may not be interested in reproducing the results shown in a research article on your own, studies based on publicly available data have an additional level of validity in that the data is readily available to the research community.

Another advantage of assessing studies that use publicly available data is that there may be other research reports that used the same data. One can look for studies that either support or refute the study you are considering. Even if the other studies do not address the same research questions, they may provide additional insight into how the data was obtained and what potential issues may be contained in the data.

In terms of the population, one should also be sure that the conclusions of a study match with the population that is being studied. For example, the results from a study of the economic habits of individuals in a certain neighborhood of a city would generally not be applicable to individuals in other neighborhoods. If the methodology and analysis of the study have been carefully designed and properly executed, the results will be relevant to the population that was used to observe the data. Attempting to make the same conclusions about larger or different populations is an example of extrapolation and is very dangerous.

Definition: Extrapolation

Extrapolation refers to making conclusions for populations that are larger than the population that was observed.

Extrapolation may be warranted when it can be shown that other populations have very similar characteristics to the one that was studied. This is something that the researchers need to carefully document and justify before those conclusions should be taken as reliable.

Most peer-reviewed academic journals insist that researchers include a section that explains the potential limitations of their research. Many of these limitations are often based on what population was observed. These limitations should be carefully considered as they are the key to understanding how universal the results may be, and where further research is required.

In summary, the key questions in critically analyzing the population in a study are:

1. Is the population well defined?
2. Do the researchers have the resources to access the population?
3. Does the population of an established dataset match the researcher's population of interest?
4. Do the sources of the data have potential interests in the outcome of the study?
5. Has a subset of an established set of data been used? Why? Are there potential biases that could result from sub-setting the data?
6. Have the authors clearly stated the limitations of the conclusions of the study based on the definition of the population?
7. Is the data publicly available?
8. Do other research reports use the same data source?
9. What are the conclusions of those research reports?
10. Are the conclusions relevant to the same population where the data was observed? Are the researchers attempting to extrapolate? Is the extrapolation justified?
11. What are the limitations of the study? Where is further research required?