11.5: Testing a Hypothesis with Repeated Measures ANOVA
- Page ID
- 50157
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)
( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\id}{\mathrm{id}}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\kernel}{\mathrm{null}\,}\)
\( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\)
\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\)
\( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)
\( \newcommand{\vectorA}[1]{\vec{#1}} % arrow\)
\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow\)
\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vectorC}[1]{\textbf{#1}} \)
\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)
\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)
\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\(\newcommand{\avec}{\mathbf a}\) \(\newcommand{\bvec}{\mathbf b}\) \(\newcommand{\cvec}{\mathbf c}\) \(\newcommand{\dvec}{\mathbf d}\) \(\newcommand{\dtil}{\widetilde{\mathbf d}}\) \(\newcommand{\evec}{\mathbf e}\) \(\newcommand{\fvec}{\mathbf f}\) \(\newcommand{\nvec}{\mathbf n}\) \(\newcommand{\pvec}{\mathbf p}\) \(\newcommand{\qvec}{\mathbf q}\) \(\newcommand{\svec}{\mathbf s}\) \(\newcommand{\tvec}{\mathbf t}\) \(\newcommand{\uvec}{\mathbf u}\) \(\newcommand{\vvec}{\mathbf v}\) \(\newcommand{\wvec}{\mathbf w}\) \(\newcommand{\xvec}{\mathbf x}\) \(\newcommand{\yvec}{\mathbf y}\) \(\newcommand{\zvec}{\mathbf z}\) \(\newcommand{\rvec}{\mathbf r}\) \(\newcommand{\mvec}{\mathbf m}\) \(\newcommand{\zerovec}{\mathbf 0}\) \(\newcommand{\onevec}{\mathbf 1}\) \(\newcommand{\real}{\mathbb R}\) \(\newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]}\) \(\newcommand{\laspan}[1]{\text{Span}\{#1\}}\) \(\newcommand{\bcal}{\cal B}\) \(\newcommand{\ccal}{\cal C}\) \(\newcommand{\scal}{\cal S}\) \(\newcommand{\wcal}{\cal W}\) \(\newcommand{\ecal}{\cal E}\) \(\newcommand{\coords}[2]{\left\{#1\right\}_{#2}}\) \(\newcommand{\gray}[1]{\color{gray}{#1}}\) \(\newcommand{\lgray}[1]{\color{lightgray}{#1}}\) \(\newcommand{\rank}{\operatorname{rank}}\) \(\newcommand{\row}{\text{Row}}\) \(\newcommand{\col}{\text{Col}}\) \(\renewcommand{\row}{\text{Row}}\) \(\newcommand{\nul}{\text{Nul}}\) \(\newcommand{\var}{\text{Var}}\) \(\newcommand{\corr}{\text{corr}}\) \(\newcommand{\len}[1]{\left|#1\right|}\) \(\newcommand{\bbar}{\overline{\bvec}}\) \(\newcommand{\bhat}{\widehat{\bvec}}\) \(\newcommand{\bperp}{\bvec^\perp}\) \(\newcommand{\xhat}{\widehat{\xvec}}\) \(\newcommand{\vhat}{\widehat{\vvec}}\) \(\newcommand{\uhat}{\widehat{\uvec}}\) \(\newcommand{\what}{\widehat{\wvec}}\) \(\newcommand{\Sighat}{\widehat{\Sigma}}\) \(\newcommand{\lt}{<}\) \(\newcommand{\gt}{>}\) \(\newcommand{\amp}{&}\) \(\definecolor{fillinmathshade}{gray}{0.9}\)The computations have been reviewed in detail in the prior section. In addition, this chapter is focused on using the \(F\)-formula to test a non-directional hypothesis. Therefore, this section will briefly review the most relevant steps for testing a non-directional hypothesis and will refer to the prior section of this chapter for the steps to using the \(F\)-formula.
Steps in Hypothesis Testing
In order to test a hypothesis, we must follow these steps:
1. State the hypothesis.
The hypothesis for Data Set 11.1 can be declared as follows: It is hypothesized that the mean confidence of a group of students will be different under three different conditions: before a class, immediately after the class, and 6 months after the class.
A one-tailed test is used when conducting the omnibus test in ANOVA, regardless of whether the hypothesis is stated directionally.
2. Choose the inferential test (formula) that best fits the hypothesis.
The means of one sample measured under three conditions are being compared so the appropriate test is a repeated-measures ANOVA.
3. Determine the critical value.
In order to determine the critical value for a repeated-measures ANOVA, three things must be identified:
- the alpha level,
- the Degrees of Freedom Between (\(d f_b\)), and,
- the Degrees of Freedom Error (\(d f_e\)).
The default alpha level of .05 is appropriate because only one hypothesis is being tested and there is no clear indication that a Type I Error would be especially problematic. Thus, alpha can be set to 5%, which can be summarized as \(\alpha \) = .05.
The \(d f_b\) and the \(d f_e\) are reported here (see the prior section for computations): \[\begin{aligned}
& d f_b=2 \\
& d f_e=14
\end{aligned} \nonumber \]
These three pieces of information are used to locate the critical value from the test. The full tables of the critical values for F-tests are located in Appendix E. Under the conditions of an alpha level of .05, \(d f_b\) = 2, and \(d f_e\) = 14, the critical value is 3.739.
\[CV = 3.739 \nonumber \]
The critical value represents the value which must be exceeded in order to declare a result significant. Thus, in order for the result to significantly support the hypothesis is needs to exceed the critical value of 3.739.
4. Calculate the test statistic.
A test statistic can also be referred to as an obtained value. The formula needed to find the test statistics (\(F\)) for this scenario is as follows:
\[F=\dfrac{M S S_b}{M S S_e} \nonumber \]
Computations are detailed in in the prior section of this chapter and only the result is shown here. The obtained value for this test was 54.60 when rounded to the hundredths place.
5. Apply a decision rule and determine whether the result is significant.
Assess whether the obtained value for \(F\) exceeds the critical value as follows:
The critical value is 4.737.
The obtained \(F\) value is 54.60.
The obtained \(F\)-value exceeds (i.e. is greater than) the critical value. Therefore, the criteria has been met to declare the result significant. Thus, the result significantly supports the hypothesis.
If the hypothesis had a direction, the directions of each group by group comparison would need to be checked in post-hoc analyses before concluding that all aspects of the hypothesis had been supported. However, the current hypothesis was simply that groups would differ. Therefore, we are able to conclude that the hypothesis was supported without needing post-hoc results. However, it is still good practice to report the means and standard deviations for each condition in our APA-formatted summary of the results.
6. Calculate the effect size and/or other relevant secondary analyses.
When it is determined that the result is significant, effect sizes and post-hoc analyses may be computed. We reviewed effect size and post-hocs for ANOVA in Chapter 10. Here we will briefly note changes that needs to be made to the formulas for effect size and post-hoc formulas when using repeated-measures ANOVA. However, these are not a focus of this chapter and the post-hocs will not be computed for the present hypothesis because it was non-directional. Instead, the adjustments that must be made to these computations when using repeated-measures ANOVA are included here for your reference.
The effect size is a calculation of the percent of variance observed that was systematic (and, thus, was uniquely between groups) which is reported in decimal form (see Chapter 10 for a review of this computation). The symbol for this effect size is \(\eta^2\)(which is named “eta squared”). An amended version of the formula is used for repeated measures ANOVA which removed participant differences. This is because participant differences (\(S S_p\)) are not captured in sum of squares between when using a repeated measures ANOVA, and, therefore, need to be removed from the denominator of the formula. Therefore, effect sizes for repeated-measures ANOVAs are computed with the amended formula as follows:
\[\eta^2=\dfrac{S S_b}{S S_T-S S_p} \nonumber \]
For the same reason, the denominator for the post-hoc formula is also changed when it accompanies a repeated-measures ANOVA. Specifically, the mean sum of squares error (\(M S_e\)) is used for the numerator of the HSD formula for repeated-measures ANOVA instead of the mean sum of squares within (\(M S_w\)) which was used for the simple, independent groups ANOVA (see Chapter 10 for review of post-hoc analyses). Therefore, the effect size HSD formula for repeated-measures ANOVA is as follows:
\[H S D=q \sqrt{\dfrac{M S_e}{n}} \nonumber \]
7. Report the results in American Psychological Associate (APA) format.
Results for inferential tests are often best summarized using a paragraph that states the following:
- the hypothesis and specific inferential test used,
- the main results of the test and whether they were significant,
- any additional results that clarify or add details about the results,
- whether the results support or refute the hypothesis.
Keep in mind that results are reported in past tense because they report on what has already been found. In addition, the research hypothesis must be stated but the null hypothesis is usually not needed for summary paragraphs because it can be deduced
from the research hypothesis. Finally, APA format requires a specific format be used for reporting the results of a test. This includes a specific format for reporting relevant symbols and details for the formula and data used. Following this, the results for our hypothesis with Data Set 11.1 can be written as shown in the summary example below.
A repeated-measures ANOVA was used to test the hypothesis that the mean confidence of a group of students would be different under three different conditions: before a class, immediately after the class, and 6 months after the class. The mean confidence scores were significantly different before (\(M\) = 4.75, \(SD\) = 1.39), immediately after (\(M\) = 7.75, \(SD\) = 1.75), and 6 months after a class (\(M\) = 7.00, \(SD\) = 1.60), \(F(2, 14) = 54.60, p < .05\). Thus, the hypothesis was supported.
Reading Review 11.4
- What information is needed to find the critical value for a repeated-measures ANOVA?
- How is significance determined for a repeated-measures ANOVA?