Skip to main content
Statistics LibreTexts

4.9: Nested Anova

Skills to Develop

  • Use nested anova when you have one measurement variable and more than one nominal variable, and the nominal variables are nested (form subgroups within groups). It tests whether there is significant variation in means among groups, among subgroups within groups, etc.

When to use it

Use a nested anova (also known as a hierarchical anova) when you have one measurement variable and two or more nominal variables. The nominal variables are nested, meaning that each value of one nominal variable (the subgroups) is found in combination with only one value of the higher-level nominal variable (the groups). All of the lower level subgroupings must be random effects (model II) variables, meaning they are random samples of a larger set of possible subgroups.


Fig. 4.9.1 Ben.

Nested analysis of variance is an extension of one-way anova in which each group is divided into subgroups. In theory, you choose these subgroups randomly from a larger set of possible subgroups. For example, a friend of mine was studying uptake of fluorescently labeled protein in rat kidneys. He wanted to know whether his two technicians, who I'll call Brad and Janet, were performing the procedure consistently. So Brad randomly chose three rats, and Janet randomly chose three rats of her own, and each technician measured protein uptake in each rat.

If Brad and Janet had measured protein uptake only once on each rat, you would have one measurement variable (protein uptake) and one nominal variable (technician) and you would analyze it with one-way anova. However, rats are expensive and measurements are cheap, so Brad and Janet measured protein uptake at several random locations in the kidney of each rat:

Technician: Brad Janet
Rat: Arnold Ben Charlie Dave Eddy Frank
  1.119 1.045 0.9873 1.3883 1.3952 1.2574  
  1.2996 1.1418 0.9873 1.104 0.9714 1.0295  
  1.5407 1.2569 0.8714 1.1581 1.3972 1.1941  
  1.5084 0.6191 0.9452 1.319 1.5369 1.0759  
  1.6181 1.4823 1.1186 1.1803 1.3727 1.3249  
  1.5962 0.8991 1.2909 0.8738 1.2909 0.9494  
  1.2617 0.8365 1.1502 1.387 1.1874 1.1041  
  1.2288 1.2898 1.1635 1.301 1.1374 1.1575  
  1.3471 1.1821 1.151 1.3925 1.0647 1.294  
  1.0206 0.9177 0.9367 1.0832 0.9486 1.4543

Because there are several observations per rat, the identity of each rat is now a nominal variable. The values of this variable (the identities of the rats) are nested under the technicians; rat A is only found with Brad, and rat D is only found with Janet. You would analyze these data with a nested anova. In this case, it's a two-level nested anova; the technicians are groups, and the rats are subgroups within the groups. If the technicians had looked at several random locations in each kidney and measured protein uptake several times at each location, you'd have a three-level nested anova, with kidney location as subsubgroups within the rats. You can have more than three levels of nesting, and it doesn't really make the analysis that much more complicated.

Note that if the subgroups, subsubgroups, etc. are distinctions with some interest (fixed effects, or model I, variables), rather than random, you should not use a nested anova. For example, Brad and Janet could have looked at protein uptake in two male rats and two female rats apiece. In this case you would use a two-way anova to analyze the data, rather than a nested anova.

When you do a nested anova, you are often only interested in testing the null hypothesis about the group means; you may not care whether the subgroups are significantly different. For this reason, you may be tempted to ignore the subgrouping and just use all of the observations in a one-way anova, ignoring the subgrouping. This would be a mistake. For the rats, this would be treating the \(30\) observations for each technician (\(10\) observations from each of three rats) as if they were \(30\) independent observations. By using all of the observations in a one-way anova, you compare the difference in group means to the amount of variation within each group, pretending that you have \(30\) independent measurements of protein uptake. This large number of measurements would make it seem like you had a very accurate estimate of mean protein uptake for each technician, so the difference between Brad and Janet wouldn't have to be very big to seem "significant." You would have violated the assumption of independence that one-way anova makes, and instead you have what's known as pseudoreplication.

What you could do with a nested design, if you're only interested in the difference among group means, is take the average for each subgroup and analyze them using a one-way anova. For the example data, you would take the average protein uptake for each of the three rats that Brad used, and each of the three rats that Janet used, and you would analyze these six values using one-way anova. If you have a balanced design (equal sample sizes in each subgroup), comparing group means with a one-way anova of subgroup means is mathematically identical to comparing group means using a nested anova (and this is true for a nested anova with more levels, such as subsubgroups). If you don't have a balanced design, the results won't be identical, but they'll be pretty similar unless your design is very unbalanced. The advantage of using one-way anova is that it will be more familiar to more people than nested anova; the disadvantage is that you won't be able to compare the variation among subgroups to the variation within subgroups. Testing the variation among subgroups often isn't biologically interesting, but it can be useful in the optimal allocation of resources, deciding whether future experiments should use more rats with fewer observations per rat.

Null hypotheses

A nested anova has one null hypothesis for each level. In a two-level nested anova, one null hypothesis is that the groups have the same mean. For our rats, this null would be that Brad's rats had the same mean protein uptake as the Janet's rats. The second null hypothesis is that the subgroups within each group have the same means. For the example, this null would be that all of Brad's rats have the same mean, and all of Janet's rats have the same mean (which could be different from the mean for Brad's rats). A three-level nested anova would have a third null hypothesis, that all of the locations within each kidney have the same mean (which could be a different mean for each kidney), and so on.

How the test works

Remember that in a one-way anova, the test statistic, \(F_s\), is the ratio of two mean squares: the mean square among groups divided by the mean square within groups. If the variation among groups (the group mean square) is high relative to the variation within groups, the test statistic is large and therefore unlikely to occur by chance. In a two-level nested anova, there are two \(F\) statistics, one for subgroups (\(F_{subgroup}\)) and one for groups (\(F_{group}\)). You find the subgroup \(F\)-statistic by dividing the among-subgroup mean square, \(MS_{subgroup}\) (the average variance of subgroup means within each group) by the within-subgroup mean square, \(MS_{within}\) (the average variation among individual measurements within each subgroup). You find the group \(F\)-statistic by dividing the among-group mean square, \(MS_{group}\) (the variation among group means) by \(MS_{subgroup}\). You then calculate the \(P\) value for the \(F\)-statistic at each level.

For the rat example, the within-subgroup mean square is \(0.0360\) and the subgroup mean square is \(0.1435\), making the \(F_{subgroup}\) \(0.1435/0.0360=3.9818\). There are \(4\) degrees of freedom in the numerator (the total number of subgroups minus the number of groups) and \(54\) degrees of freedom in the denominator (the number of observations minus the number of subgroups), so the \(P\) value is \(0.0067\). This means that there is significant variation in protein uptake among rats within each technician. The \(F_{group}\) is the mean square for groups, \(0.0384\), divided by the mean square for subgroups, \(0.1435\), which equals \(0.2677\). There is one degree of freedom in the numerator (the number of groups minus \(1\)) and \(4\) degrees of freedom in the denominator (the total number of subgroups minus the number of groups), yielding a \(P\) value of \(0.632\). So there is no significant difference in protein abundance between the rats Brad measured and the rats Janet measured.

For a nested anova with three or more levels, you calculate the \(F\)-statistic at each level by dividing the \(MS\) at that level by the \(MS\) at the level immediately below it.

If the subgroup \(F\)-statistic is not significant, it is possible to calculate the group \(F\)-statistic by dividing \(MS_{group}\) by \(MS_{pooled}\), a combination of \(MS_{subgroup}\) and \(MS_{within}\). The conditions under which this is acceptable are complicated, and some statisticians think you should never do it; for simplicity, I suggest always using \(MS_{group}/MS_{subgroup}\) to calculate \(F_{group}\).

Partitioning variance and optimal allocation of resources

In addition to testing the equality of the means at each level, a nested anova also partitions the variance into different levels. This can be a great help in designing future experiments. For our rat example, if most of the variation is among rats, with relatively little variation among measurements within each rat, you would want to do fewer measurements per rat and use a lot more rats in your next experiment. This would give you greater statistical power than taking repeated measurements on a smaller number of rats. But if the nested anova tells you there is a lot of variation among measurements but relatively little variation among rats, you would either want to use more observations per rat or try to control whatever variable is causing the measurements to differ so much.

If you have an estimate of the relative cost of different parts of the experiment (in time or money), you can use this formula to estimate the best number of observations per subgroup, a process known as optimal allocation of resources:

\[n=\sqrt{\frac{(C_{subgroup}\times V_{within})}{(C_{within}\times V_{subgroup})}}\]

where \(n\) is the number of observations per subgroup, \(C_{within}\) is the cost per observation, \(C_{subgroup}\) is the cost per subgroup (not including the cost of the individual observations), \(V_{subgroup}\) is the percentage of the variation partitioned to the subgroup, and \(V_{within}\) is the percentage of the variation partitioned to within groups. For the rat example, \(V_{subgroup}\) is \(23.0\%\) and \(V_{within}\) is \(77\%\) (there's usually some variation partitioned to the groups, but for these data, groups had \(0\%\) of the variation). If we estimate that each rat costs \(\$ 200\) to raise, and each measurement of protein uptake costs \(\$ 10\), then the optimal number of observations per rat is \(\sqrt{\frac{(200\times 77)}{(10\times 23)}}\), which equals \(8\) rats per subgroup. The total cost per subgroup will then be \(\$ 200\) to raise the rat and \(8\times \$ 10=\$ 80\) for the observations, for a total of \(\$ 280\); based on your total budget for your next experiment, you can use this to decide how many rats to use for each group.

For a three-level nested anova, you would use the same equation to allocate resources; for example, if you had multiple rats, with multiple tissue samples per rat kidney, and multiple protein uptake measurements per tissue sample. You would start by determining the number of observations per subsubgroup; once you knew that, you could calculate the total cost per subsubgroup (the cost of taking the tissue sample plus the cost of making the optimal number of observations). You would then use the same equation, with the variance partitions for subgroups and subsubgroups, and the cost for subgroups and the total cost for subsubgroups, and determine the optimal number of subsubgroups to use for each subgroup. You could use the same procedure for as higher levels of nested anova.

It's possible for a variance component to be zero; the groups (Brad vs. Janet) in our rat example had 0% of the variance, for example. This just means that the variation among group means is smaller than you would expect, based on the amount of variation among subgroups. Because there's variation among rats in mean protein uptake, you would expect that two random samples of three rats each would have different means, and you could predict the average size of that difference. As it happens, the means of the three rats Brad studied and the three rats Janet studied happened to be closer than expected by chance, so they contribute \(0\%\) to the overall variance. Using zero, or a very small number, in the equation for allocation of resources may give you ridiculous numbers. If that happens, just use your common sense. So if \(V_{subgroup}\) in our rat example (the variation among rats within technicians) had turned out to be close to \(0\%\), the equation would have told you that you would need hundreds or thousands of observations per rat; in that case, you would design your experiment to include one rat per group, and as many measurements per rat as you could afford.

Often, the reason you use a nested anova is because the higher level groups are expensive and lower levels are cheaper. Raising a rat is expensive, but looking at a tissue sample with a microscope is relatively cheap, so you want to reach an optimal balance of expensive rats and cheap observations. If the higher level groups are very inexpensive relative to the lower levels, you don't need a nested design; the most powerful design will be to take just one observation per higher level group. For example, let's say you're studying protein uptake in fruit flies (Drosophila melanogaster). You could take multiple tissue samples per fly and make multiple observations per tissue sample, but because raising \(100\) flies doesn't cost any more than raising \(10\) flies, it will be better to take one tissue sample per fly and one observation per tissue sample, and use as many flies as you can afford; you'll then be able to analyze the data with one-way anova. The variation among flies in this design will include the variation among tissue samples and among observations, so this will be the most statistically powerful design. The only reason for doing a nested anova in this case would be to see whether you're getting a lot of variation among tissue samples or among observations within tissue samples, which could tell you that you need to make your laboratory technique more consistent.

Unequal sample sizes

When the sample sizes in a nested anova are unequal, the \(P\) values corresponding to the \(F\)-statistics may not be very good estimates of the actual probability. For this reason, you should try to design your experiments with a "balanced" design, meaning equal sample sizes in each subgroup. (This just means equal numbers at each level; the rat example, with three subgroups per group and \(10\) observations per subgroup, is balanced). Often this is impractical; if you do have unequal sample sizes, you may be able to get a better estimate of the correct \(P\) value by using modified mean squares at each level, found using a correction formula called the Satterthwaite approximation. Under some situations, however, the Satterthwaite approximation will make the \(P\) values less accurate. If you cannot use the Satterthwaite approximation, the \(P\) values will be conservative (less likely to be significant than they ought to be), so if you never use the Satterthwaite approximation, you're not fooling yourself with too many false positives. Note that the Satterthwaite approximation results in fractional degrees of freedom, such as \(2.87\); don't be alarmed by that (and be prepared to explain it to people if you use it). If you do a nested anova with an unbalanced design, be sure to specify whether you use the Satterthwaite approximation when you report your results.


Nested anova, like all anovas, assumes that the observations within each subgroup are normally distributed and have equal standard deviations.


Fig. 4.9.2 Old man's beard lichen, Usnea longissima.

Keon and Muir (2002) wanted to know whether habitat type affected the growth rate of the lichen Usnea longissima. They weighed and transplanted \(30\) individuals into each of \(12\) sites in Oregon. The \(12\) sites were grouped into \(4\) habitat types, with \(3\) sites in each habitat. One year later, they collected the lichens, weighed them again, and calculated the change in weight. There are two nominal variables (site and habitat type), with sites nested within habitat type. You could analyze the data using two measurement variables, beginning weight and ending weight, but because the lichen individuals were chosen to have similar beginning weights, it makes more sense to use the change in weight as a single measurement variable. The results of a nested anova are that there is significant variation among sites within habitats (\(F_{8,\: 200}=8.11,\; \; P=1.8\times 10^{-9}\)) and significant variation among habitats (\(F_{3,\: 8}=8.29,\; \; P=0.008\)). When the Satterthwaite approximation is used, the test of the effect of habitat is only slightly different (\(F_{3,\: 8.13}=8.76,\; \; P=0.006\))

Graphing the results

The way you graph the results of a nested anova depends on the outcome and your biological question. If the variation among subgroups is not significant and the variation among groups is significant—you're really just interested in the groups, and you used a nested anova to see if it was okay to combine subgroups—you might just plot the group means on a bar graph, as shown for one-way anova. If the variation among subgroups is interesting, you can plot the means for each subgroup, with different patterns or colors indicating the different groups.

Similar tests

Both nested anova and two-way anova (and higher level anovas) have one measurement variable and more than one nominal variable. The difference is that in a two-way anova, the values of each nominal variable are found in all combinations with the other nominal variable; in a nested anova, each value of one nominal variable (the subgroups) is found in combination with only one value of the other nominal variable (the groups).

If you have a balanced design (equal number of subgroups in each group, equal number of observations in each subgroup), you can perform a one-way anova on the subgroup means. For the rat example, you would take the average protein uptake for each rat. The result is mathematically identical to the test of variation among groups in a nested anova. It may be easier to explain a one-way anova to people, but you'll lose the information about how variation among subgroups compares to variation among individual observations.

How to do the test


I have made spreadsheets to do two-level nested anova  nested2.xls, with equal or unequal sample sizes, on up to \(50\) subgroups with up to \(1000\) observations per subgroup. It does significance tests and partitions the variance. The spreadsheet tells you whether the Satterthwaite approximation is appropriate, using the rules on p. 298 of Sokal and Rohlf (1983), and gives you the option to use it. \(F_{group}\) is calculated as \(MS_{group}/MS_{subgroup}\). The spreadsheet gives the variance components as percentages of the total. If the estimate of the group component would be negative (which can happen), it is set to zero.

I have also written spreadsheets to do three-level nested anova  nested3.xls and four-level nested anova nested4.xls.

Web page

I don't know of a web page that will let you do nested anova.


Salvatore Mangiafico's \(R\) Companion has a sample R program for nested anova.


You can do a nested anova with either PROC GLM or PROC NESTED. PROC GLM will handle both balanced and unbalanced designs, but does not partition the variance; PROC NESTED partitions the variance but does not calculate P values if you have an unbalanced design, so you may need to use both procedures.

You may need to sort your dataset with PROC SORT, and it doesn't hurt to include it.

In PROC GLM, list all the nominal variables in the CLASS statement. In the MODEL statement, give the name of the measurement variable, then after the equals sign give the name of the group variable, then the name of the subgroup variable followed by the group variable in parentheses. SS1 (with the numeral one, not the letter el) tells it to use type I sums of squares. The TEST statement tells it to calculate the \(F\)-statistic for groups by dividing the group mean square by the subgroup mean square, instead of the within-group mean square (\(H\) stands for "hypothesis" and \(E\) stands for "error"). "HTYPE=1 ETYPE=1" also tells SAS to use "type I sums of squares"; I couldn't tell you the difference between them and types II, III and IV, but I'm pretty sure that type I is appropriate for a nested anova.

Here is an example of a two-level nested anova using the rat data.

DATA bradvsjanet;
   INPUT tech $ rat $ protein @@;

Janet  1  1.119    Janet  1  1.2996  Janet  1  1.5407  Janet  1  1.5084  Janet  1  1.6181

Janet  1  1.5962  Janet  1  1.2617  Janet  1  1.2288  Janet  1  1.3471  Janet  1  1.0206 

Janet  2  1.045    Janet  2  1.1418  Janet  2  1.2569  Janet  2  0.6191  Janet  2  1.4823 

Janet  2  0.8991  Janet  2  0.8365  Janet  2  1.2898  Janet  2  1.1821  Janet  2  0.9177 

Janet  3  0.9873  Janet  3  0.9873  Janet  3  0.8714  Janet  3  0.9452  Janet  3  1.1186 

Janet  3  1.2909  Janet  3  1.1502  Janet  3  1.1635  Janet  3  1.151    Janet  3  0.9367 

Brad   5  1.3883  Brad   5  1.104    Brad   5  1.1581  Brad   5  1.319    Brad   5  1.1803 

Brad   5  0.8738  Brad   5  1.387    Brad   5  1.301    Brad   5  1.3925  Brad   5  1.0832 

Brad   6  1.3952  Brad   6  0.9714  Brad   6  1.3972  Brad   6  1.5369   Brad   6  1.3727 

Brad   6  1.2909  Brad   6  1.1874  Brad   6  1.1374  Brad   6  1.0647   Brad   6  0.9486 

Brad   7  1.2574  Brad   7  1.0295  Brad   7  1.1941  Brad   7  1.0759   Brad   7  1.3249 

Brad   7  0.9494  Brad   7  1.1041  Brad   7  1.1575  Brad   7  1.294     Brad   7  1.4543

PROC SORT DATA=bradvsjanet;
   BY tech rat;
PROC GLM DATA=bradvsjanet;
  CLASS tech rat;
  MODEL protein=tech rat(tech) / SS1;
  TEST H=tech E=rat(tech) / HTYPE=1 ETYPE=1;

The output includes \(F_{group}\) calculated two ways, as \(MS_{group}/MS_{within}\) and as \(MS_{group}/MS_{subgroup}\).

Source       DF   Type I SS     Mean Sq.      F Value    Pr > F

tech           1   0.03841046   0.03841046    1.07      0.3065 MSgroup/MSwithin; don't use this
rat(tech)    4   0.57397543    0.14349386    3.98      0.0067 use this for testing subgroups

Tests of Hypotheses Using the Type I MS for rat(tech) as an Error Term

Source      DF   Type I SS        Mean Sq.   F Value   Pr > F

tech          1    0.03841046   0.03841046   0.27      0.6322 MSgroup/MSsubgroup; use this for testing groups

You can do the Tukey-Kramer test to compare pairs of group means, if you have more than two groups. You do this with a MEANS statement. This shows how (even though you wouldn't do Tukey-Kramer with just two groups):

PROC GLM DATA=bradvsjanet;
  CLASS tech rat;
  MODEL protein=tech rat(tech) / SS1;
  TEST H=tech E=rat(tech) / HTYPE=1 ETYPE=1;

PROC GLM does not partition the variance. PROC NESTED will partition the variance, but it only does the hypothesis testing for a balanced nested anova, so if you have an unbalanced design you'll want to run both PROC GLM and PROC NESTED. In PROC NESTED, the group is given first in the CLASS statement, then the subgroup.

PROC SORT DATA=bradvsjanet;
   BY tech rat;
PROC NESTED DATA=bradvsjanet;
   CLASS tech rat;
   VAR protein;

Here's the output; if the data set was unbalanced, the "\(F\) Value" and "Pr>F" columns would be blank.

Variance            Sum of        F                      Error    Mean        Variance       Percent
Source      DF    Squares      Value    Pr>F      Term   Square      Component    of Total

Total       59     2.558414                                        0.043363    0.046783    100.0000
tech         1      0.038410    0.27     0.6322    rat       0.038410   -0.003503      0.0000
rat          4       0.573975    3.98     0.0067    Error    0.143494    0.010746     22.9690
Error       54     1.946028                                        0.036038    0.036038     77.0310

You set up a nested anova with three or more levels the same way, except the MODEL statement has more terms, and you specify a TEST statement for each level. Here's how you would set it up if there were multiple rats per technician, with multiple tissue samples per rat, and multiple protein measurements per sample:

PROC GLM DATA=bradvsjanet;
  CLASS tech rat sample;
  MODEL protein=tech rat(tech) sample(rat tech)/ SS1;
  TEST H=tech E=rat(tech) / HTYPE=1 ETYPE=1;
  TEST H=rat E=sample(rat tech) / HTYPE=1 ETYPE=1;
PROC NESTED DATA=bradvsjanet;
   CLASS sample tech rat;
   VAR protein;


Picture of a rat from Posterwire.

Picture of lichen from Lichenological Society of Japan Lichen Photo Gallery

Keon, D.B., and P.S. Muir. 2002. Growth of Usnea longissima across a variety of habitats in the Oregon coast range. Bryologist 105: 233-242.


  • John H. McDonald (University of Delaware)