12.4: Commentary

The commentary for the ANOVA is mostly about how there are different terms for the ANOVA.

12.4.1: ANOVA Terminology

Be mindful of the terminology when discussing the ANOVA and the F-test research design. There are different terms that mean the same thing. Sometimes we call it a turntable or a record player, pancake or flapjack, coat or jacket. There is no discernible difference between the two terms; in certain situations, people prefer one term over another. Even the terms ANOVA and F-test are interchangeable. “Running an ANOVA” or “Conducting an F-test” are the same thing.

Different terminology is used when describing the variables and their groups. Recall that the design setup for the ANOVA is the independent variable, which is nominal or categorical, and consists of three or more groups, and one dependent variable. The groups are part of an overall variable. If we want to examine race, we have one variable called “race,” and within that variable of race, we have four groups: White vs. Black vs. Hispanic/Latino vs. Asian. Sometimes we say we are analyzing groups, and what we really refer to is one variable with multiple groups. For example, we can say we are analyzing medication, mindfulness, and a control group. These are different treatment groups, and the overall variable is treatment. Sometimes we say groups, other times we say categories. They mean the same thing. We can have a variable called diagnosis, and there are three categories for that diagnosis: depression, anxiety, and substance use. The issue to note here is that while we say we are examining three or more groups, we are really saying that one nominal or categorical variable is divided into three or more groups, or categories.

Sometimes we say the independent variable has three groups or levels. It’s the same thing. We say “race” as our independent variable, and it has four levels: White vs. Black vs. Hispanic/Latino vs. Asian. It is the same thing as groups.

12.4.2: More than One IV and Terminology

There are many ways to use the ANOVA. Recall that ANOVA is not a statistical analysis with one function, in this case, to examine the variance among three or more groups. ANOVA analyzes variance, and there are many sources of variance. By sources of variances, I mean ways to vary, and everything can vary with two or more variables, not just groups, over time, and with additional continuous variables.

Here we will introduce the idea that ANOVA can be used with two or more independent variables (IV), not just two or more groups. You can use two variables, race and gender. Or you can use treatment and gender. In these examples, the groups are nominal, categorical variables. Each variable can have two or more groups. So, for race we can have Black, White, Hispanic, Asian, for gender we can have male, female, non-binary, for treatment we can have treatment, control, and usual care.

The terms statisticians use when we have two or more IVs are factor and way. What is the difference between these terms? I do not know, and it does not make any difference as far as I am concerned.

The term factor is used in the phrases in Factorial Analysis of Variance, two-factor ANOVA, or an ANOVA with two factors. The factor refers to an independent variable that is nominal/categorical. When we say two factors, we are indicating that we have two independent variables, and the variables can have two or more groups or categories. For example, a two-factor ANOVA means we have gender and race, or treatment and gender. Sometimes we say one-way ANOVA or two-way ANOVA. The one-way usually refers to one variable with multiple groups, such as race: White vs. Black vs. Hispanic/Latino Vs. Asian. Two-way usually refers to two variables, each with multiple groups, such as gender: male vs. female, and Race: White vs. Black vs. Hispanic/Latino Vs. Asian.

Researchers use different terms because that is just how they were taught. It is just different terminology. There is no standard way of describing something. If someone corrects you about using factor or way, then tell them “Bite me,” because you are not making any violations or errors if you casually use one term of the other.

So why learn these terms? For the darn EPPP exam. These exams will test you with story problems, and the problem might disguise the independent variables and the groups. You have to read closely to determine what the IV is and how many categories/groups/levels there are.

12.4.3: Factorial Analysis of Variance

Now that we have introduced the idea that ANOVAs can have more than one categorical/nominal IV, we can introduce the idea of how we usually analyze two or more IVs. When using this design, typically we are interested in interaction effects. Interactions are like combining the effects of two variables. Suppose you are interested in examining the effects of gender and race on an outcome variable. You have gender: male, female, and race: White, Black, Hispanic, and Asian. If you combine all of them, you have the following: male White, female White, male Black, female Black, male Hispanic, female Hispanic, male Asian, female Asian. For now, that is all we will say about interactions. And ANOVAs can have two or more independent variables.

12.4.4: Analysis of Covariates (ANCOVA)

ANOVA is useful for analyzing covariates. What is a covariate? Up to now, you have learned about independent variables and dependent variables. Covariates are additional independent variables but are not your independent variable of interest. Huh?

When we use covariates as part of an ANOVA, we call it ANCOVA, or analysis of covariance. Covariates are variables that are conceptually related to the DV, or useful for predicting the DV. Think of covariates as something you have to include when predicting the DV, because leaving out those covariates means missing important information. However, covariates are not your independent variable, which means the updated definition of an independent variable is not just something that predicts the outcome but something that is of interest when predicting the outcome.

The difference between covariates and independent variables is that you want to include a covariate because you are interested in how this additional variable explains the DV and then see if your IV still predicts any variation in the DV after accounting for the covariate.

For example, the IV is treatment – mindfulness treatment vs. control. The DV is depression severity. The covariate is gender. The reason why we include gender as a covariate is that it is known that males and females differ in the level of depression. Males and females also differ in their level of acceptance of mindfulness as a treatment option. We should include gender as a covariate because it likely impacts how the participants respond to the treatment, and how it affects the outcome of depression, but the variable of interest is treatment, and the question is if mindfulness treatment lowers depression, after we account for how the covariate, gender, affects both the treatment and the outcome of depression.

Covariates are usually demographic variables. We want to account for known demographics and their impact on the outcome. The usual demographics are gender, age, race, income, and level of education. All these demographics usually have an impact on psychological outcomes.

Covariates can also be known as psychological processes that undoubtedly need to be acknowledged as impacting the independent and dependent variables. Continuing with our example, let us use the covariate level of stress. The idea is that people get depressed based on their stress level. Mindfulness can help with depression, but only after we account for how much depression is associated with stress. People who are more stressed may not be able to fully engage in a mindfulness treatment, which is what the mindfulness treatment is supposed to address in the first place. We should account for their stress level first, then see if the mindfulness treatment works. We need to account for the stress level so we can isolate the effect of just mindfulness treatment on depression.

Accounting for covariates helps. Including covariates is a good idea because the covariate usually represents what we already know as impacting the outcome. We know that gender and stress are associated with depression. However, the independent variable is the one of interest. In this example, the research question is, how much more can mindfulness reduce depression after we account for the effects of gender and stress? Conceptually, it helps to add covariates because we cannot talk about the outcome, in this case depression, without acknowledging the known effects of covariates, such as gender and stress, on the outcome.

From a statistical viewpoint, adding covariates reduces the unexplained variance in the DV. Recall that the goal of statistics is to explain variation. We want to know why something varies. We try to explain 100% of the variation in the DV by using the IV. But the IV does not explain everything. If we use just the IV, we will explain some of the variance, and whatever variance is left over is the error variance. In this case, the error variance is unexplained variance, not necessarily random error. We simply do not know what else could explain the variation in the DV. We include a covariate and say, “This percentage of the variability in the DV can be explained by the covariate.” We can say that this percentage of variability in the DV is explained by the covariate, and this percentage of variability is explained by the IV. We can compare how much variance is explained by the covariate and the IV. Hopefully, the percentage of variance explained by the IV is greater than the percentage of variance explained by the covariate. That is what we really want to know.

Of note, do not use covariates haphazardly; choose covariates based on a theoretical premise. The above example makes it seem like we should include as many covariates as possible. However, including too many covariates can reduce your degrees of freedom, depending on your sample size. always in statistics, do not just throw in all the variables into your analysis. That is sloppy and poor planning.

But this section's main message is to include covariates whenever possible, which means, try to use an ANCOVA whenever possible.

Exactly how do you conduct an ANCOVA? Well, in SPSS, the option to include covariates is already incorporated into an ANOVA. There really is no such thing as a separate selection of ANOVA vs. ANCOVA. When you operate a statistics program, you do not see an option to select ANOVA or select ANCOVA. The option to include covariates is built into the ANOVA stats analysis. You don’t need to select an ANCOVA versus an ANOVA. The ANOVA already includes the covariate option.

From a research design perspective, you will find that including covariates helps establish the equivalency of the group research design or to rule out potential confounds. For an equivalent group research design, we want to establish the equivalency of two groups. This is a research method issue, so for now, the best research design is when everything is the same, except for the unique factor between the two groups. Continuing with our example, if we want to examine the effect of mindfulness versus no mindfulness on depression, we want to establish that the mindfulness group is equivalent to the no mindfulness group in every way, except for the mindfulness. We include gender, age, race, income, and level of education to ensure that we have equal numbers of these demographics. We never have equivalent amounts, so we include these demographics as covariates so we can account for the variance in the demographics, leaving the variance accounted for by the mindfulness group. So, we “filter out” the covariates to establish a semblance of equivalency between the two groups. The same idea applies to ruling out confounds. In brief, confounds are factors that are alternative explanations for the outcome. Demographics could be confounded. So, we include the demographics as covariates to rule out the possibility that the demographics could be responsible for the outcome. Continuing with our example, by including gender, age, race, income, and level of education, we rule out the possibility that younger, female, higher income, and higher education have something to do with reducing depression, rather than the mindfulness treatment. The lesson here is that we use covariates to establish the equivalency between treatment and control groups, and we use covariates as a method to rule out confounds that could negate treatment results. , so be on the lookout for them.

12.4.5: Trend Analysis

ANOVA and the post-hoc tests are used for what is called trend analysis. Trend analysis is used to examine trends among the groups. There are two types of trends, linear and curvilinear. We are looking for an increase in the mean scores for each group. When does this situation occur? For example, we can examine education budgets, which show an increase when we sort the states from lowest to highest education budgets. We can examine crime rates as we sort the neighborhood crime rate from lowest to highest. The states and the neighborhoods are the groups, the education budgets and crime rates are the outcome variables, and we want to see if they increase in a linear or a curvilinear fashion across the groups.

These linear or curvilinear increases or decreases across the groups are best illustrated when we turn to longitudinal analyses. For me, I hate referring to these analyses as trend analyses because (a) the group order is arbitrary, and (b) trend makes me think of a trend over time, not necessarily by groups. For now, if there is a licensing exam question that asks you what the analysis of choice is to analyze changes in mean scores across groups, the answer is likely a trend analysis, and the analysis is an ANOVA or an F-test, with post-hoc tests.

12.4.6: T-test and ANOVA

The t-test examines two groups only; the ANOVA analyzes one independent variable that is nominal/categorical with several groups, or more than one independent variable with two or more groups.

What happens if you run an ANOVA with two groups? Will your statistical software explode? Nope. You can run an ANOVA with one independent variable and two groups. You will get an F-test, and if you take the square root of the F-test, you will have a value equivalent to the t-test if you conduct a t-test with the two groups. Or, if you run a t-test and square the t-test value, you will get a value equivalent to the F-test value. Basically, a t-test squared is equivalent to an F-test with one independent variable and two groups.

Does this matter? Nope. Do you get more information using ANOVA to examine mean differences between two groups? Nope. Is it a huge error to use an ANOVA when you should have used a t-test? Nope. All you need to know is that the t-test value squared is equivalent to the F-test, which is useless information, but it might be information for a potential licensing exam question. And that is all.

12.4.7: BTW…. I hate MANOVA

The ANOVA can only examine one dependent or outcome variable. If you have more than one dependent or outcome variable, the analysis is called a MANOVA, which stands for Multiple Analysis of Variance. The term multiple in the MANOVA refers to multiple outcomes or dependent variables. The word “multiple” refers to multiple independent or predictor variables. Go figure.

if you had multiple dependent variables, you could conduct several ANOVAs, one for each dependent variable, and call it a day. So why don't we? Here is your EPPP alert – Q: Why do you use MANOVA? A: You do not conduct multiple ANOVAs because you want to avoid the experimenter error-wise rate. Recall the problem with conducting too many t-tests, or too many statistical tests in general. Running multiple ANOVAs is “fishing; when you continue to test you will eventually find an effect, but it’s likely due to random chance, which is a Type I error. The experimenter error-wise rate is the same as an inflated Type I error. You increase the chances of finding something, but it’s a random effect because you run several ANOVA tests. If we have multiple dependent variables, we use a MANOVA to avoid the experimenter-wise error rate, which is the answer to that licensing exam question.

The second reason I am addressing this issue now is this – I hate MANOVA. Why do I hate MANOVA? Because everyone automatically thinks that you have multiple DVs you HAVE to use a MANOVA. But every study has multiple outcome variables. If you think about it, every issue and phenomenon can have multiple outcomes because everything is multi-dimensional. Academic achievement is multi-dimensional; it can be divided by subject matter, GPA, and standardized test scores. Alcohol consumption is multi-dimensional; it can be divided by the number of drinks, frequency of drinking, time frame during which the drinking occurred, and attitudes about drinking. There are always multiple outcomes, which does not mean we conduct MANOVAs for everything, and it would mean there are no situations where you would ever conduct an ANOVA for a single outcome variable. In the end, what will happen is that SPSS MANOVA will provide you with an overall multivariate effect, but then, similar to a post-hoc t-test, the analysis will break down the multivariate effect with several ANOVA. In essence, you conduct multiple ANOVAs anyway to determine which groups had what effect on which dependent variable.

Why would you use a MANOVA? You pick multiple outcome variables because they are conceptually related. Conceptually related, they need to have something in common, and that commonality is based on your explanation about why the combination, or the collection of the outcome variables, is necessary for you to understand the issue at hand. There needs to be a reason you are interested in the multivariate effect, or why are all those outcome variables related? There needs to be a theoretical reason. If you don’t know how those are related theoretically, do not conduct a MANOVA.

An example of related dependent variables in psychology could be self-harm. There are multiple reasons why individuals engage in self-harm, such as seeking help from others, inflicting punishment on themselves when others wrong them, feeling something other than negative feelings, and inflicting punishment on themselves for feeling negative about themselves. These reasons can be related in terms of the functions of self-harm. A MANOVA might be appropriate to conduct here to see if group effects, such as treatment vs. control groups, have multiple effects across all these reasons for using self-harm.

Instead of using MANOVA, my preference would be to use structural equation modeling. Other statisticians likely have other preferences. The point is this – if someone tells you that you should be using a MANOVA when you have multiple outcome variables, tell them “bite me” and call Dr. Ji, because automatically using a MANOVA just because you have multiple outcome variables indicates to me that they do not fully understand the concept of a MANOVA.