8.6: Reporting the Results of a Hypothesis Test

When writing up the results of a hypothesis test, there are usually several pieces of information that you need to report, but it varies a fair bit from test to test. Throughout the rest of the book, we'll spend a little time talking about how to report the results of different tests, so that you can get a feel for how it’s usually done. However, regardless of what test you’re doing, the one thing that you always have to do is say something about the p-value, and whether or not the outcome was significant.

The fact that you have to do this is unsurprising; it’s the whole point of doing the test. What might be surprising is the fact that there is some contention over exactly how you’re supposed to do it. Leaving aside those people who completely disagree with the entire framework underpinning null hypothesis testing, there’s a certain amount of tension that exists regarding whether or not to report the exact p-value that you obtained, or if you should state only that p<\(\alpha\).

The Issue

To see why this is an issue, the key thing to recognize is that p values are terribly convenient. In practice, the fact that we can compute a p-value means that we don’t actually have to specify any \(\alpha\) level at all in order to run the test. Instead, what you can do is calculate your p-value and interpret it directly: if you get p=.062, then it means that you’d have to be willing to tolerate a Type I error rate of 6.2% to justify rejecting the null. If you personally find 6.2% intolerable, then you retain the null. Therefore, the argument goes, why don’t we just report the actual p-value and let the reader make up their own minds about what an acceptable Type I error rate is? This approach has the big advantage of “softening” the decision-making process – in fact, if you accept the Neyman definition of the p-value, that’s the whole point of the p-value. We no longer have a fixed significance level of \(\alpha\)=.05 as a bright line separating “accept” from “reject” decisions, and this removes the rather pathological problem of being forced to treat p=.051 in a fundamentally different way to p=.049.

This flexibility is both the advantage and the disadvantage of the p-value. The reason why a lot of people don’t like the idea of reporting an exact p-value is that it gives the researcher a bit too much freedom. In particular, it lets you change your mind about what error tolerance you’re willing to put up with after you look at the data. For instance, consider the ESP experiment. Suppose we ran the test, and ended up with a p-value of .09. Should we accept or reject the null hypothesis? Now, to be honest, we haven’t yet bothered to think about what level of Type I error we're “really” willing to accept. We don’t have an opinion on that topic. But we do have an opinion about whether or not ESP exists, and we definitely have an opinion about whether our research should be published in a reputable scientific journal. And amazingly, now that we’ve looked at the data we're starting to think that a 9% error rate isn’t so bad, especially when compared to how annoying it would be to have to admit to the world that our experiment has failed. So, to avoid looking like we just made it up after the fact, we now say that our \(\alpha\) is .1: a 10% type I error rate isn’t too bad, and at that level our test is significant! We win.

In other words, the worry here is that we might have the best of intentions, and be the most honest of people, but the temptation to just “shade” things a little bit here and there is really, really strong. As anyone who has ever run an experiment can attest, it’s a long and difficult process, and you often get very attached to your hypotheses. It’s hard to let go and admit the experiment didn’t find what you wanted it to find. And that’s the danger here. If we use the “raw” p-value, people will start interpreting the data in terms of what they want to believe, not what the data are actually saying, and if we allow that, well, why are we bothering to do science at all? Why not let everyone believe whatever they like about anything, regardless of what the facts are? Okay, that’s a bit extreme, but that’s where the worry comes from. According to this view, you really must specify your \(\alpha\) value in advance, and then only report whether the test was significant or not. It’s the only way to keep ourselves honest.

Two Proposed Solutions

In practice, it’s pretty rare for a researcher to specify a single \(\alpha\) level ahead of time. Instead, the convention is that scientists rely on three standard significance levels: .05, .01 and .001. When reporting your results, you indicate which (if any) of these significance levels allow you to reject the null hypothesis. This is summarized in Table 8.1. This allows us to soften the decision rule a little bit, since p<.01 implies that the data meet a stronger evidentiary standard than p<.05 would. Nevertheless, since these levels are fixed in advance by convention, it does prevent people from choosing their \(\alpha\) level after looking at the data.

Table 8.1: A commonly adopted convention for reporting p values: in many places it is conventional to report one of four different things (e.g., p<.05) as shown below. I’ve included the “significance stars” notation (i.e., a * indicates p<.05) because you sometimes see this notation produced by statistical software. It’s also worth noting that some people will write n.s. (not significant) rather than p>.05.

Usual notation	Signif. stars	Signif. stars	The null is…
p>.05	NA	The test wasn’t significant	Retained
p<.05	*	The test was significant at $= .05 but not at α=.01 or α=.001.$	Rejected
p<.01	**	The test was significant at α=.05 and α=.01 but not at $= .001	Rejected
p<.001	***	The test was significant at all levels	Rejected

Nevertheless, quite a lot of people still prefer to report exact p values. To many people, the advantage of allowing the reader to make up their own mind about how to interpret p=.06 outweighs any disadvantages. In practice, however, even among those researchers who prefer exact p values, it is quite common to just write p<.001 instead of reporting an exact value for a small p. This is in part because a lot of software doesn’t actually print out the p-value when it’s that small (e.g., SPSS just writes p=.000 whenever p<.001), and in part because a very small p-value can be kind of misleading. The human mind sees a number like .0000000001 and it’s hard to suppress the gut feeling that the evidence in favor of the alternative hypothesis is a near certainty. In practice, however, this is usually wrong. Life is a big, messy, complicated thing: and every statistical test ever invented relies on simplifications, approximations, and assumptions. As a consequence, it’s probably not reasonable to walk away from any statistical analysis with a feeling of confidence stronger than p<.001 implies. In other words, p<.001 is really code for “as far as this test is concerned, the evidence is overwhelming.”

In light of all this, you might be wondering exactly what you should do. There’s a fair bit of contradictory advice on the topic, with some people arguing that you should report the exact p-value, and other people arguing that you should use the tiered approach illustrated in Table 8.1. As a result, the best advice is to suggest that you look at papers/reports written in your field and see what the convention seems to be. If there doesn’t seem to be any consistent pattern, then use whichever method you prefer.