8.1: The Role of Probability

In statistics, we want to find that something is significant. Compared to what? A random result. Probability establishes what observations are random results and what observations are not random. Put another way, what observations are significant or likely not random, and which ones are not significant or likely random?

Of note is that significance is an interesting concept because if a result is significant, it has to be significant compared to … what? Recall that a hallmark of statistics is the study of variation. Variation is based on differences. Differences are in this form: this is different than that. The “that” is important because when we say something is significant, it means that the difference between your observations and “that” is likely not a random occurrence. But what is “that?” The “that” is randomness.

However, in the field, we do not usually compare something to randomness. In psychology, we usually compare a treatment group to a control group or a group that does nothing. Or we compare it to usual treatment, such as medication. The main issue is that when something is significant, we must ask the question, “Compared to what?” In one sense, anything can be significant compared to random chance. Anything can be significant compared to doing nothing. Something might not be significant when compared to something established, such as usual treatment care. The issue is a research design issue, not a statistical issue. Research design sets up the comparison of “this compared to that.” Statistics determine if the comparison is significant or if the difference between this and that is no better than random chance. In the end, you want to be cautious when someone says, “This is significant.” You want to respond, “significant compared to what?” A result that is significant compared to random chance is really not that interesting.

It is worth noting that this problem of “compared to what” can be politically sticky. By politically sticky, I mean we can be biased if our results are significant simply by reconfiguring what we are comparing them to. In global warming, we say that our climate is warmer than before. Before what? By following the global temperature across hundreds of years, we can see a warming trend. But others might say, well, compared to the last 20 years, this warming trend is nothing but a mild increase and probably random. Consequently, in a few more years, we will see a cooling trend. Depending on the motives of interested parties, we can make anything “significant” or “not significant” depending on what we choose as our comparison frame. And no, that is not a good practice, so we must be mindful of when someone says, “It is significant,” we should always ask, “Compared to what?” so that we can be thoughtful about the validity of their statement.

What does it mean for something to be significant? What is the thought process for determining that something is a random result versus something is happening? Statistically and technically speaking, we can never be sure that our result is due to something significant. There is always a chance that our result is due to chance. When we say that something is significant, we mean that the probability of the result happening by only random chance is so small that it is more plausible that something must have led to that result because there is very little chance that the result could have occurred on its own by random chance.

A classic example is the poker hand. It is entirely possible that you obtain a full house on the first hand of poker. It is entirely possible that you obtain a full house on your second hand of poker. It is entirely possible that you obtain a full house on your third hand of poker. You can obtain three consecutive full houses in three hands of poker, all due to randomness or just plain luck. However, the probability of getting three consecutive full houses in three hands of poker on plain luck alone is very low. The opposite of chance is a pattern, or “something is going on.” The alternative explanation is that you are doing something to lead to that result because that result would likely not have happened on its own by pure chance. Emphasis on “likely.” Yes, the three consecutive full houses for each poker hand are entirely possible due to chance, but the likelihood is very low. The more plausible explanation is that you did something to lead to that outcome.

Suppose we want to test a new treatment for alcoholism. The average sobriety time after discharge from an alcoholism treatment program is three months. We test your new treatment for alcoholism, and the average sobriety time becomes one year. Is it possible that your set of clients got lucky and just happened to stay sober for one year on their own? It is always a possibility. But the probability of that happening is so low that the more plausible explanation is that your new treatment program is so good that it is likely the treatment program led to a good outcome rather than pure luck.

This issue gets politically sticky. Take the example of global warming. Is the increase in temperature spikes a random event, or is it due to something happening? As of 2024, some believe that increased temperature spikes can occur randomly throughout the history of the planet’s temperature; therefore, what we are experiencing as a temperature spike is nothing more than a random event.

This concept of intentional actions rather than random luck creating specific results is the basis of significance testing. Technically, we can never be certain that what we found is due to our treatment effect or hypothesized association between variables. Technically, anything can happen based on pure luck alone. However, we must weigh the possibility that instead of something happening by random luck, an action must have been taken to cause the outcome because the likelihood of this outcome occurring randomly is low.

Notice we say that the likelihood is low. We didn’t say that the threshold for determining the probability of something happening by pure luck is too low. We did not specify how low a low probability is. This specification is the focus of the remaining chapter.