5.4.4: The Supreme Court and Personal Attitudes

Earlier we considered a study of how the potential Supreme Court ruling on same-sex marriage would change individual's personal attitudes about the LGBTQIA+ community (Tankard and Paluck 2017). The research project of interest here tested these ideas in a controlled experimental setting where individuals read a brief article about the likely outcome of the upcoming Supreme Court ruling on same sex marriage. Each participant was randomly assigned to read either a version entitled “Supreme Court Likely to Rule in Favor of Gay Marriage” or a version entitled “Supreme Court Unlikely to Rule in Favor of Gay Marriage.” Participants then responded to a series of questions about demographics and other social views and behaviors (Tankard and Paluck 2017).

The researchers asked several questions, a few of which we will consider here in detail. These first questions dealt with what the individuals perceived as the feelings of society at the time about same-sex marriage. This score was computed by taking the average of the response to the questions, “To what extent do you think Americans oppose or support gay marriage?” and “To what extent do you think Americans oppose or support making gay marriage legal in the United States?” Each question used a scale from –4, which corresponded to being strongly opposed, to 4, which corresponded to being strongly supportive. Each of the questions are measured on an ordinal measurement scale and so the idea of averaging the two responses together should be questioned. For the researchers to average these two responses is must be permissible to add these values, a mathematical operation that is usually not necessarily allowed for ordinal data. To consider this question, we should determine what this value really means.

When a researcher is going to combine two measurements in this way, we need to first make sure that the two scales are comparable. In this case, both questions are measured using the same scale ranging from –4 to 4 with levels of agreement ranging from strongly opposed to strongly support. The next issue is whether the ratings mean the same thing for both questions. That is, does being strongly opposed in the first question have the same level of opposition as being strongly opposed in the other question? There is no way to theoretically know this for sure, and validating this idea would take further research. This is a problem that everyone would have to evaluate on their own, deciding how much trust they would like to put into these calculations.

One way to approach this issue is to imagine what such a combined score really means. Suppose that an individual reported a score of –4, indicating being strongly opposed for both questions, then the average would also be –4, which does indicate strong opposition. Similarly, a reported score of 4 for each question would have an average of 4, indicating strong support for both questions and strong support overall. But what if the response to one score is –4 and the other is 4? Then the average would be 0, indicating neither opposition nor support. Does this really mean the same as if an individual answered 0 to each question? That is, does strong opposition for one question and strong support for the other mean the same thing as neither opposition nor support on both questions? Again, this is an issue that everyone would have to evaluate on their own.

It should be noted that the issue of averaging over the two questions is an attempt to simplify the data used in the study and the corresponding analysis. Simplification will necessarily lose some information from the original data because the researchers are replacing the data from two questions with a single number. The question for the one evaluating the research is whether this simplifies the structure of the data to the point that the results cannot be trusted. In this case there are two relevant issues that might help us settle ourselves as to how we feel about this calculation. The first is that it is likely the responses to the two questions are strongly associated with one another. That is, an individual who gives a low score to the first question is probably also likely to give a low score to the second question as well. In this case one may reasonably assume that knowing how an individual answered one of the questions is almost as good as knowing how the individual answered both questions, and hence it may not be so bad if the two questions are summarized by an average. If this association holds, the case where an individual might answer –4 to the first question and 4 to the second question would be very rare and probably not worth worrying about.

The second relevant issue considers what will be done with the average score in the subsequent analysis. In this case the measurement is taken before, and then after, reading the randomly assigned article. Hence, in this case the researchers are interested in comparing the average score prior to the reading to the average score after the reading. Therefore, they are really interested in how the average might move because of the reading. Given the ordinal original measurement scale of the data, this type of comparison should be valid, as ordinal data does allow for size comparisons. Hence, if we are comfortable with the fact that the two measurements can be averaged, it should follow that the size comparison between the measurements should also be permissible.