Your banner table is back, and one age group is 10 points more satisfied than another. Before that gap goes into a headline, someone will ask whether it is real or just noise. Significance testing answers that question, and on a crosstab it comes with its own conventions: letters in cells, bases above columns and footnotes about confidence levels. If you are new to the table itself, start with what a crosstab is and how banners work.
Key takeaways
- Significance testing on a crosstab asks whether a gap between columns is bigger than sampling error would likely produce.
- The chi-square test checks the whole table at once; column proportions z-tests and t-tests compare individual cells.
- A letter in a cell means that column is significantly higher than the column with that letter.
- Many comparisons inflate false positives; plan key comparisons or tighten the threshold.
- Weighting and overlapping columns both change the test.
What does significance testing on a crosstab tell you?
Every percentage in a crosstab is an estimate from a sample; survey a different 300 people and the number would shift. Significance testing asks how often a gap this size would appear if the two groups were identical in the population. If the answer is rarely, usually less than 5% of the time, the gap is statistically significant.
For the core idea, read what statistical significance means and what p-values do and do not say. This guide applies those ideas inside a table.
What is the difference between the chi-square test and column tests?
Crosstab statistics work at two levels that answer different questions.
| Chi-square test of independence | Column comparisons | |
|---|---|---|
| Question | Is there any relationship between the row and column variables? | Is this cell different from that cell? |
| Scope | The whole table | One row, two columns at a time |
| Data | Counts | Percentages (z-test) or means (t-test) |
| Output | One statistic and p-value per table | Letters in individual cells |
| Limitation | Does not say where the difference is | Many tests, so more false positives |
The chi-square test is a useful first check: if the table as a whole is not significant, be wary of individual cell differences. Banner tables mostly show column comparisons, because readers want to know which group is higher.
How do you run significance testing on a crosstab?
- Choose a confidence level before you look. 95% is the usual standard; some teams also flag 90% as directional.
- Decide which columns to compare. Compare columns within a banner group, such as age bands against each other, not “18–34” against “North”, which share respondents.
- Run the tests. Use a column proportions z-test for percentages and a t-test for mean scores, such as a 0–10 rating.
- Mark the letters. Give each column a letter and place it in any cell that is significantly higher than that column.
- Check the bases. Suppress or flag tests where a base falls below a minimum you set in advance.
- Adjust for many comparisons. With many columns and rows, tighten the threshold or focus on planned comparisons.
How does the column proportions z-test work?
Illustrative numbers: in a customer survey, 62% of 300 respondents aged 18–34 (column A) say they are satisfied, against 52% of 250 respondents aged 35–54 (column B). Is the 10-point gap significant at 95%?
| Column | Base | Satisfied | Percentage |
|---|---|---|---|
| A: 18–34 | 300 | 186 | 62% |
| B: 35–54 | 250 | 130 | 52% |
Step 1: Pool the proportions
If there is no difference, both columns share one true rate. Pooled p = (186 + 130) / (300 + 250) = 316 / 550 = 0.5745.
Step 2: Calculate the standard error
SE = square root of [p × (1 − p) × (1/300 + 1/250)]
- p × (1 − p) = 0.5745 × 0.4255 = 0.2444
- 1/300 + 1/250 = 0.00333 + 0.00400 = 0.00733
- 0.2444 × 0.00733 = 0.001793
- SE = square root of 0.001793 = 0.0423, or 4.23 points
Step 3: Calculate z
z = (0.62 − 0.52) / 0.0423 = 0.10 / 0.0423 = 2.36
Step 4: Compare with the critical value
At 95% confidence, the two-sided critical value is 1.96. Since 2.36 is larger, the difference is significant, with a p-value of about 0.02. It would not pass at 99%, where the critical value is 2.58.
How do significance letters work in banner tables?
Each column in a banner group gets a letter. When a cell is significantly higher than another column in the same row, that column’s letter appears next to it. The table below adds a third column (illustrative, tested at 95%).
| Total | 18–34 (A) | 35–54 (B) | 55+ (C) | |
|---|---|---|---|---|
| Base | 750 | 300 | 250 | 200 |
| Satisfied | 57% | 62% B | 52% | 55% |
| Would recommend | 47% | 46% | 40% | 56% AB |
In the “Satisfied” row, A carries a B: 18–34s are significantly more satisfied than 35–54s, the test we just ran. A versus C (62% against 55%) gives z = 1.56, so no letter. In “Would recommend”, C is significantly higher than both A and B. The Total column is not tested because it overlaps every other column. For a row-by-row walkthrough, see how to read a crosstab.
Why do many comparisons cause false positives?
Each test at 95% carries a 5% chance of flagging a difference that is not there. A banner group with six columns produces 15 pairwise comparisons per row. If no difference were real and the tests were independent, the chance of at least one false letter would be 1 − 0.95^15, about 54%. Across 40 rows, some letters will be noise. You have three options:
- Use a stricter confidence level for exploratory scans, such as 99%.
- Apply a Bonferroni-style adjustment: divide 0.05 by the number of comparisons. For 15 comparisons that gives 0.0033, a critical z of about 2.94. Our example’s 2.36 would not pass.
- Focus on planned comparisons: decide before fieldwork which few differences matter, and treat the rest as leads.
Why do overlapping samples need a different test?
Standard column tests assume independent groups: no respondent counts in both. That fails for a subgroup against the Total, which contains it, and for multi-response banners such as “brands used in the past month”, where one person can sit in several columns.
Treating overlapping columns as independent gives the wrong standard error. Compare a subgroup with everyone outside it rather than with the Total, and use a test built for overlapping samples on multi-response banners.
How do weighting and effective base size change the result?
Weighting makes a sample match the population, but it costs precision. The design effect measures that cost; the effective base is the actual base divided by it. A column of 300 with a design effect of 1.5 carries the information of about 200 unweighted respondents.
Rerun the example with a design effect of 1.5 in both columns. The bases become 200 and 167, the standard error rises to 0.0518, and z falls to 0.10 / 0.0518 = 1.93. That is below 1.96, so the same 10-point gap is no longer significant at 95%. Testing on the raw bases would have shown a letter the data cannot support.
Is a statistically significant difference always important?
No. Statistical significance says a gap is probably not chance; practical significance asks whether it is big enough to change a decision. With large bases, a 2-point gap can earn a letter and still not justify a new campaign. With small bases, a 12-point gap can miss the threshold and still deserve a follow-up. Read the letter, the gap and the base together.
What are the common mistakes?
- Reporting letters without bases. Readers need to see that a column has only 45 respondents.
- Testing subgroups against the Total. The groups overlap, so the standard test does not apply.
- Running tests on weighted data with raw bases. The tests become too generous.
- Treating every letter in a 40-row banner as a finding. Some are false positives.
- Choosing the confidence level after seeing the results. Dropping to 90% to rescue a favorite gap is not analysis.
How do you do this in practice?
Few teams run these tests by hand. Halo Reports provides crosstabs and banner tables with weighting and significance testing. Whatever tool you use, set the confidence level and minimum base before you run the tables, and write findings in plain words: “satisfaction is significantly higher among 18–34s than 35–54s.”
For wide, complex studies and global brand trackers, mTab offers one governed platform for insights and research teams. Then put the letters to work: our guide to cross-tabulation analysis shows how to turn significant differences into a story worth acting on.