8.9: Steps of the Hypothesis Testing Process

Remember way back in the first chapter when we discussed the scientific method? Figure \(\PageIndex{1}\) shows the model, as a reminder.

Figure \(\PageIndex{1}\): Model of Stages of the Scientific Method. (CC-BY-SA; Michelle Oja via source)

We’re going to use that model to structure our process of null hypothesis significance testing (NHST) into steps. This process can be quite lengthy and involves a lot of pieces, so having a simple five-step model to refer to will help you put some of the puzzle pieces together. These steps will be what we use for the remainder of the textbook. The statistics we use will change, but this process will not.

Step 1: Hypotheses

In real life and real research, you would start with some questions about the world. In most examples in this course, you’ll be provided data or a scenario to work with. Sometimes this scenario will include a prompt or research question. For those working through this five-step Null Hypothesis Significance Testing process, your first step will be to develop research hypotheses and null hypotheses. These hypotheses will usually be stated mathematically in symbols AND in words, with the words explaining what each symbols represents for the specific research scenario. These hypotheses will look different for each different type of statistical analysis used, but we will always state the research hypothesis (what we expect or are predicting) and the null hypothesis to remind us what we are testing (that nothing happened).

Step 2: Collect Data

Most versions of this course focus on analyzing data, designing studies, and collecting data. This step reminds us to identify what variables were collected and who the data was collected on.

Step 3: Analyze Data

Once we have our hypotheses and data, we can do calculations! This step is where the vast majority of differences in future chapters will arise: different tests used for different data are calculated in different ways, but the way we use and interpret them remains the same.

Step 4: Conclusion

To conclude whether or not the research hypothesis is supported, we first need to find a critical value to compare our calculated statistic to in order to decide whether or not the null hypothesis is retained or rejected. Okay, that’s not entirely true. If you are using statistical software, the software provides a p-value (probability that the null hypothesis is true). Typically, we use \(p\)<.05 (less than a 5% probability that nothing is happening) to reject the null hypothesis. Remember:

Reject null: \((\bar{X} \neq \mu) = p<.05\)
Retain null: \( (\bar{X} = \mu) = p>.05\)

Whether we compare our calculated value to a critical value to determine if \(p\)<.05 or \(p\)>.05 (if the probability is higher than 5% or lower than 5% that the null hypothesis is true in the population), or statistical software provides a p-value, we use that information to decide whether the null hypothesis should be retained or rejected.

If the null hypothesis is retained, then we can conclude that the research hypothesis was not supported.

Example \(\PageIndex{1}\)

Why can we conclude that the research hypothesis is automatically not supported if the null hypothesis is retained?

Solution

The null hypothesis always says that nothing is happening, and the research hypothesis (RH) is always saying that something is happening. In research hypotheses with two means, the research hypothesis says that one group’s mean is bigger than another group’s mean, so we can’t support that something is happening (that one group’s mean is bigger than another group’s mean) if we just concluded that nothing is happening (retain the null).

If the null hypothesis is rejected, then we have one more step in making a conclusion. By rejecting the null hypothesis, we’re saying that something happened. But in directional research hypothesis, we expected the results to be in a certain direction. For example, a directional research hypothesis with means would state that the mean number of times that a student re-tries an Example after they don’t get quite the same number is higher for those with growth mindset than those with a fixed mindset. This research hypothesis could look like:

\[ \bar{X}_{GM} > \bar{X}_{FM} \nonumber \]

To support this research hypothesis, the means should actually be in that direction! Rejecting the null hypothesis says that the means are different, but we can’t support a directional research hypothesis unless these means are different in the direction that we hypothesized.

Once you’ve made the decision, share what you’ve learned!

Step 5: Write-Up

Your answer is never just a number in behavioral statistics. In Null Hypothesis Significance Testing, your answer should include information about the research hypothesis and your calculated results.

First, the research hypothesis should be re-stated (in words), and then state your conclusion (from the previous step) about whether the research hypothesis was supported or not. Note that you don’t have to mention the null hypothesis here at all! However, when a directional research hypothesis is not supported, it helps to clarify whether the research hypothesis was not supported because the means were in the opposite direction of what was predicted or because the null hypothesis was retained. When the research hypothesis is supported, most researchers don’t mention the null hypothesis.

Second, your write-up should include your calculated results. When comparing means, this includes stating the mean of each group with the units of the variable that was measured. For example, if you measured final grade percentages, then the unit would be percentages. If you measured the number of classes missed, then the unit would be absences or percentage of classes missed. If you measured scores on a quiz, the units might be points. The other calculated results would be the calculated statistic, e.g., your math results. Each chapter from now on will discuss a different type of statistical analysis, and the results for each formula would be your calculated statistic. You’ll learn more in next chapter about how to format these results, but for now just remember that you’ll need to include that information, too.

Conclusion

These processes are summarized in Table \(\PageIndex{1}\).

Table \(\PageIndex{1}\)- Steps of the NHST Process and What to Do

Step of the Null Hypothesis Significance Testing Process	What To Do
1. Hypotheses	State a directional research hypothesis in words and symbols (include the groups being compared, the DV, and the direction word) State the null hypothesis in words and symbols.
2. Collect Data	You may not need to literally collect data, but you’ll always want to determine: Who is the sample? Who is the population? What is being measured? What are the groups (if any)?
3. Analyze Data	Do the math!
4. Conclusion	Determine whether the research hypothesis is supported or not. To do that, you’ll probably need to find a critical value to see if your null hypothesis is retained or rejected. If the null hypothesis is rejected, then determine if the direction of the actual means is the same as the expected direction in the research hypothesis (Is the mean of the group that you said would be bigger actually the bigger mean?).
5. Write-Up	Share your results with others by: Restating the research hypothesis and stating whether or not it was supported, and Providing the calculated results, which will probably include the actual means (with units) but will definitely include the calculated statistic.

Dr. MO is excited to start putting this five-step Null Hypothesis Significance Testing process into practice! But first, there’s at least one more piece about testing null hypotheses that we should think about: probability errors.