You have satisfaction scores from customers who used three different service channels, and the averages look different. Is that a real difference, or the kind of gap you would see by chance if you drew three random samples from the same customers? ANOVA is the standard way to answer that question for three or more groups, and it is one of the tests covered in our guide to statistical significance in survey research.
Key takeaways
- Analysis of variance (ANOVA) tests whether the means of three or more groups differ by more than chance would explain, by comparing between-group and within-group variance.
- The F ratio is between-group variance divided by within-group variance; values near 1 fit equal means, and a large F points to differences beyond noise.
- A significant F says at least one group mean differs, not which ones, so follow it with a post-hoc test such as Tukey’s HSD.
- Running several t-tests instead of ANOVA inflates the false-positive rate, which is the main reason ANOVA exists for three or more groups.
- ANOVA assumes independent observations, roughly normal residuals and similar variance across groups; for counts in categories, use a chi-square test instead.
What does ANOVA test?
ANOVA tests one claim, called the null hypothesis: all the group means are equal in the population. The alternative hypothesis is that at least one mean is different. A significant result does not mean every group differs from every other, only that the means are not all the same.
The name is a little misleading, because ANOVA compares means by looking at variances. If the groups really have the same mean, the group averages in your sample should differ only by the random noise you would expect given how much scores vary from person to person. If the averages are spread out much more than that noise explains, the means probably differ.
What are between-group and within-group variance?
Two quantities do all the work.
- Between-group variance measures how far each group’s mean sits from the grand mean, the mean of every observation pooled together. Groups with very different averages produce a large value.
- Within-group variance measures how far individual scores sit from their own group’s mean. It is the background noise: the ordinary differences between people who answered the same question under the same conditions.
Note that ANOVA does not expect the variance inside each group to differ when the means differ. It compares the spread of the means against the spread inside groups, and it assumes the within-group spread is similar across groups.
What is the F ratio?
The F ratio divides the between-group variance estimate by the within-group variance estimate. Both estimates are called mean squares (MS), each calculated as a sum of squares divided by its degrees of freedom.
- SSB (sum of squares between) is the sum, over groups, of the group size times the squared distance of the group mean from the grand mean.
- SSW (sum of squares within) is the sum of the squared distances of every score from its own group mean.
- df between is the number of groups minus 1. df within is the total number of observations minus the number of groups.
- MSB = SSB / df between, MSW = SSW / df within, and F = MSB / MSW.
If the null hypothesis is true, F tends to land near 1. A large F means the group means are more spread out than the noise explains.
The p-value is the probability of seeing an F at least as large as yours if the null hypothesis were true. It is not the probability that the null hypothesis is true, which is a common misreading. Our guide to p-values and statistical significance explains that distinction in more detail.
How do you run a one-way ANOVA? A worked example
A one-way ANOVA compares groups defined by a single factor. Here the factor is service channel, with three levels. The numbers below are illustrative, not real survey results: five customers per channel rated satisfaction from 0 to 10.
| Chat | Phone | |
|---|---|---|
| 5 | 7 | 8 |
| 6 | 8 | 9 |
| 6 | 7 | 9 |
| 7 | 8 | 8 |
| 6 | 7 | 9 |
| Mean 6.0 | Mean 7.4 | Mean 8.6 |
We test H0 (the three population means are equal) against H1 (at least one differs) at the 5% level.
- Grand mean. The 15 scores sum to 110, so the grand mean is 110 / 15 = 7.33.
- SSB. Each group has 5 people: 5 × (6.0 − 7.33)² + 5 × (7.4 − 7.33)² + 5 × (8.6 − 7.33)² = 8.89 + 0.02 + 8.03 = 16.93.
- SSW. Squared distances from each group’s own mean: email 1 + 0 + 0 + 1 + 0 = 2.0; chat 0.16 + 0.36 + 0.16 + 0.36 + 0.16 = 1.2; phone 0.36 + 0.16 + 0.16 + 0.36 + 0.16 = 1.2. The total is 4.4.
- Degrees of freedom. Between: 3 − 1 = 2. Within: 15 − 3 = 12.
- Mean squares. MSB = 16.93 / 2 = 8.47. MSW = 4.4 / 12 = 0.367.
- F ratio. F = 8.47 / 0.367 = 23.09.
- P-value. With 2 and 12 degrees of freedom, the 5% critical value of F is 3.89. Our F of 23.09 is far above it, and the p-value is about 0.00008.
Software reports all of this as one ANOVA table:
| Source | Sum of squares | df | Mean square | F | p |
|---|---|---|---|---|---|
| Between groups | 16.93 | 2 | 8.47 | 23.09 | under 0.001 |
| Within groups | 4.40 | 12 | 0.37 | ||
| Total | 21.33 | 14 |
The p-value is below 0.05, so we reject the null hypothesis: the three channels do not all have the same mean satisfaction. The between-group sum of squares is 16.93 of a total 21.33, so channel accounts for about 79% of the variation in these scores (this share is called eta squared). Real survey data is rarely this clean.
What is the difference between one-way and two-way ANOVA?
A one-way ANOVA has one grouping factor, as above. A two-way ANOVA has two factors at once, such as service channel and customer segment (new versus returning). It tests three things: the effect of channel, the effect of segment, and the interaction between them. An interaction means the effect of one factor depends on the level of the other, for example phone beating email for new customers but not for returning ones.
Two separate one-way tests would miss the interaction.
What are the assumptions of ANOVA?
ANOVA gives trustworthy p-values when these hold reasonably well:
- Independent observations. One respondent’s score does not influence another’s, and each person appears in one group only. Repeated measures on the same people need a different design.
- Roughly normal residuals. What matters is that the differences between scores and their own group mean are roughly bell-shaped, not that the raw data is. With moderate-to-large groups, ANOVA tolerates mild departures.
- Similar variance across groups. Check that the spread in each group is comparable. It is more of a worry when group sizes are very unequal. Welch’s ANOVA is a common alternative when spreads differ.
Which groups differ? Post-hoc tests
A significant F says the means are not all equal. It does not say which pairs differ. A post-hoc test compares the pairs while controlling the overall false-positive rate, which running separate t-tests does not.
Tukey’s honestly significant difference (HSD) is the usual choice when comparing every pair. In our example, the critical difference at the 5% level is about 1.02 points. Chat is 1.4 above email, phone is 2.6 above email, and phone is 1.2 above chat. All three gaps exceed 1.02, so each pair differs.
What are common mistakes with ANOVA?
- Reading a significant F as “every group differs.” It only says that at least one mean is different. Follow up with a post-hoc test.
- Running many t-tests instead. Three groups mean three pairwise tests, and each extra test inflates the chance of a false positive.
- Confusing significant with important. With thousands of respondents, a half-point difference can be significant. Report the size of the difference, not only the p-value.
- Using ANOVA on the wrong data. For counts in categories, such as a yes/no answer by region, use a chi-square test of homogeneity instead.
How do you do this in practice?
Most analysts run ANOVA in a statistics package and read a table like the one above. Halo Reports provides crosstabs and banner tables with weighting and significance testing, plus charts and interactive report pages, so you can see how groups compare in one place. Pulse then surfaces meaningful movement across your trackers and studies.
Next time you compare three or more averages, write down the null hypothesis first, run the ANOVA, and check the group sizes and spreads before reading the p-value. If F is significant, add a post-hoc test and report the size of each difference, not just whether it cleared 0.05. To follow brand health, consideration and competitors over time, market and brand intelligence shows what changed and why.