You ran a survey and two groups gave different answers. Younger customers rate a product higher than older ones, or this quarter’s awareness score is a few points above last quarter’s. Before anyone acts, you need to know whether the gap reflects something real in the market or just the luck of who happened to answer. That is the job of statistical significance, and this guide covers the idea, the arithmetic and the tests behind it.
Key takeaways
- Statistical significance means a gap in your sample is unlikely to be chance alone, which usually means a p-value below 0.05 set in advance.
- The p-value is the probability of a result at least this extreme if there were no difference; it is not the probability that your finding is true.
- Choose the significance level before testing; at 0.05, about 1 in 20 tests of a truly null difference comes out significant by chance.
- Significant does not mean important: with 5,000 respondents per group, 51% versus 49% is significant, yet a 2-point gap rarely justifies a change.
- Match the test to the data: z-test for a percentage in two groups, chi-square for answer patterns, t-test for two means, ANOVA for three or more.
What does statistically significant mean?
A result is statistically significant when the difference you see in a sample is larger than random sampling variation would plausibly produce. If you survey 400 people twice, you will get slightly different percentages each time even though nothing changed. Significance testing asks how often a gap this big would show up from that noise alone.
If the answer is “rarely”, usually less than 5% of the time, the result is called statistically significant. If the answer is “fairly often”, the result is not significant, which means the data do not give you enough evidence to say the groups differ. It does not prove they are the same.
This applies to differences between groups (men and women, two brands, two ad concepts), to changes over time (wave 1 versus wave 2 of a tracker), and to relationships between variables (does satisfaction rise with tenure?).
What does a p-value actually say?
The p-value is the number that does the judging. It is the probability of seeing a result at least as extreme as the one you observed, assuming there is no real difference in the population. That assumption of no difference is called the null hypothesis.
A p-value of 0.03 means this: if the two groups were truly identical, samples that far apart would turn up about 3% of the time. That is why a small p-value counts as evidence against the null hypothesis. For a deeper treatment, including what the number does not mean, read p-values and statistical significance: what you should know.
Two misreadings are common, and both appear in plenty of published reports:
- A p-value is not the probability that your finding is true. A p-value of 0.03 does not mean there is a 97% chance the difference is real.
- A non-significant result is not a probability that chance caused it. It means the evidence is insufficient to rule chance out, often because the sample was small.
How do you choose a significance level?
Before looking at results, you set a threshold called the significance level, written alpha. The convention is 0.05, which corresponds to a 95% confidence level. Some fields use 0.01 for stricter evidence, and some survey teams treat 0.10 as a “directional” flag.
The choice is a judgment call rather than a law of nature, and 0.05 is a habit more than a proof. What matters is that you pick the level before you test, state it in the report, and apply it to every comparison. Alpha is also the false-positive rate you are willing to accept: at 0.05, you accept that about 1 in 20 tests of a truly null difference will come out significant by chance.
How do you test statistical significance?
Here is a worked example. The numbers are illustrative, not from a real study. Suppose you show 400 customers package design A and a different 400 customers design B, and ask whether they would buy.
| Design | Respondents | Would buy | Percentage |
|---|---|---|---|
| A | 400 | 208 | 52% |
| B | 400 | 176 | 44% |
The gap is 8 percentage points. To test it, use a two-proportion z-test.
- State the hypotheses. The null hypothesis is that purchase intent is the same for both designs in the wider population. The alternative is that it differs.
- Set alpha. We choose 0.05, decided before looking.
- Pool the proportions. Under the null hypothesis both groups share one true rate. Pooled rate = (208 + 176) / (400 + 400) = 384 / 800 = 0.48.
- Calculate the standard error. SE = square root of 0.48 x 0.52 x (1/400 + 1/400) = square root of 0.001248 = 0.0353, or 3.53 points.
- Calculate the test statistic. z = 0.08 / 0.0353 = 2.26.
- Find the p-value. A z of 2.26 gives a two-sided p-value of about 0.024.
- Decide. 0.024 is below 0.05, so the difference is statistically significant at the 95% level. It is not significant at 0.01.
The 95% cutoff for z is 1.96, the same number marked on the curve at the top of this page. Any z beyond plus or minus 1.96 falls in the outer 5% of the distribution and counts as significant. Ours, at 2.26, falls outside.
A confidence interval tells you more than the verdict. The 8-point gap has a standard error of about 3.5 points (calculated without pooling), so the 95% interval runs from about 1 to 15 points. The difference is probably real, but it could be small, and that matters for the decision.
What happens when the sample is smaller?
Keep the same 52% versus 44% split but survey only 100 people per design. The standard error rises to 7.07 points, z falls to 1.13, and the p-value is about 0.26. The gap is identical, but now it is entirely consistent with chance.
This is why base size belongs next to every percentage in a table. A 10-point gap on a base of 40 may not be significant, while a 2-point gap on a base of thousands can be.
Which statistical significance test should you use?
The right test depends on the type of data and the number of groups you are comparing. The z-test above handles one percentage in two groups. Other situations need other tools.
| Question | Data | Test |
|---|---|---|
| Is one percentage different between two groups? | Categorical | Two-proportion z-test |
| Does the whole pattern of answers differ across groups? | Categorical, several answer options | Chi-square test |
| Is the mean different between two groups? | Numeric | T-test |
| Is the mean different across three or more groups? | Numeric | ANOVA |
| Is the base too small for chi-square to be reliable? | Categorical, small counts | Fisher’s exact test |
The chi-square test of homogeneity compares the full distribution of an answer, such as a five-point agreement scale, across groups in one test. It is the standard test behind the significance markers in crosstabs.
Analysis of variance (ANOVA) compares average scores across three or more groups at once. Running many pairwise t-tests instead inflates your false-positive rate, which is the main reason ANOVA exists. If ANOVA says the groups are not all equal, follow it with pairwise comparisons to find which ones differ.
Does statistically significant mean important?
No, and this is the most expensive confusion in applied research. Significance answers “is there probably a difference?” It does not answer “is the difference big enough to act on?”
Sample size drives much of this. With 5,000 respondents per group, 51% versus 49% gives a standard error of 1 point and a z of 2.0, a p-value of about 0.046. That is significant at 0.05, yet a 2-point gap rarely justifies changing a pack, a price or a campaign. In the other direction, a large gap in a small sample can fail the test even when it deserves a follow-up study.
So read three things together: whether the result is significant, how large the difference is, and how much uncertainty surrounds it (the confidence interval). A related source of uncertainty is sampling error, which sets how far any survey percentage can drift from the true value.
Where is statistical significance used?
The same logic applies wherever data come from a sample rather than from everyone:
- Survey and market research. Comparing segments, brands, concepts and tracking waves.
- Clinical and medical research. Comparing outcomes for a treatment group and a control group.
- Education and social science. Comparing the effect of different teaching methods on test scores, or outcomes across demographic groups.
- Engineering and economics. Judging whether a change in a process or a measure reflects a real shift.
In each case the test supports one decision: is the evidence strong enough to treat the difference as real? It cannot tell you why a difference exists. A significant link between two variables still needs a study design that rules out other explanations before anyone talks about cause.
What are the common mistakes?
These are the errors we see most often when teams read significance results.
- Testing everything. Each test at 0.05 carries a 5% chance of a false positive. Run 20 independent comparisons and the chance that at least one comes out significant by luck is 1 - 0.95^20, about 64%. Run 100 and you expect around five. Decide which comparisons matter before you look at the data, or adjust for the number of tests.
- Treating 0.05 as a cliff. A p-value of 0.049 and one of 0.051 are nearly the same evidence. Report the p-value or the interval, not just a pass or fail.
- Reading non-significance as “no difference”. A small base often cannot detect a real gap. The honest summary is “not enough evidence”, and sometimes the fix is a larger sample.
- Testing weighted data as if it were unweighted. Weighting reduces the effective sample size, so tests run on the raw count are too generous. Use software that tests on the effective base.
- Treating wave-to-wave movement as real. A tracker moves every wave. Test the change, and look for movement sustained over several waves before acting. Pulse monitors trackers and studies and surfaces meaningful movement, with the evidence one question away.
- Ignoring how the sample was drawn. Significance tests assume a probability-style sample. A biased sample can produce a very significant, very wrong result.
How do you do this in practice?
Most survey teams run these tests inside their tabulation tool rather than by hand, as covered in significance testing on a crosstab. Halo Reports provides crosstabs and banner tables with weighting and significance testing, so differences between columns are flagged on the table itself. Whatever tool you use, state the confidence level in the footnote, show the bases, and write the finding in plain words: “consideration among first-time buyers is significantly lower than last wave” says more than a p-value.
Next time you see a gap in your data, work through three questions before you share it. Is it significant at the level you set in advance? How large is it, with what interval? And is it big enough to change a decision?
If you track brand health, consideration and competitors continuously, market and brand intelligence shows what changed and why. If you want to go deeper, start with the chi-square test of homogeneity for categorical answers, or ANOVA for comparing averages across several groups.