You have two groups of survey respondents and one has a higher number than the other. The question is whether that gap is a real difference in the market or just the luck of who happened to answer. The p-value is the tool most reports use to answer it, and it is also one of the most misquoted numbers in research.
This article covers the precise meaning, so you can read one correctly. It sits inside our wider guide to statistical significance in survey research.
Key takeaways
- A significant p value is one at or below a threshold chosen before seeing the data, most often 0.05, which means chance alone is an unlikely explanation.
- The p-value is the probability of seeing a gap at least as large as yours if there were truly no difference in the population.
- A p-value is not the probability that the null hypothesis is true, so p = 0.036 does not mean a 3.6% chance the groups are the same.
- Significant does not mean important, and a non-significant result does not prove there is no difference, since small samples have little power to detect real gaps.
- Running 20 independent tests at 0.05 gives about a 64% chance of at least one false positive, so decide key comparisons before analysis.
What does a p-value actually measure?
A p-value answers one narrow question: if there were no real difference in the population, how often would random sampling alone produce a gap at least as big as the one you observed?
That “no real difference” assumption is called the null hypothesis. You start by assuming it is true, calculate how surprising your data would be under it, and report that surprise as a probability between 0 and 1.
A small p-value says your data would be unusual in a world with no difference. It does not say how likely the null is to be true. That distinction matters, and we come back to it below.
What is a significant p value?
A result is called statistically significant when its p-value is at or below a cutoff chosen in advance. That cutoff is the significance level, written alpha. The most common choice is 0.05, which corresponds to the 95% confidence level used in most survey reports.
The 0.05 line is a convention, not a law of nature. It became popular because it is a workable balance: strict enough to filter out a lot of noise, loose enough to let real effects through in samples of ordinary size.
Some fields use 0.01 or stricter. Exploratory work sometimes uses 0.10. What matters is that you pick the threshold before you see the data, because choosing it afterward lets you pick whichever one makes your result look good.
The threshold also has a plain meaning. If the null is true and you test at 0.05, you will wrongly call a difference significant about 1 time in 20. That is the error rate you are accepting, not the chance that any particular finding is wrong.
How do you calculate a p-value? A worked example
Say you survey two independent samples of 1,000 people each about whether they would buy a product. These numbers are illustrative.
| Concept A | Concept B | |
|---|---|---|
| Respondents | 1,000 | 1,000 |
| Said “would buy” | 220 | 260 |
| Share | 22.0% | 26.0% |
Concept B leads by 4 percentage points. Is that real? Run a two-proportion z-test.
- State the hypotheses. Null: the true purchase rates are equal. Alternative: they differ (a two-sided test).
- Pool the samples. Under the null both groups share one rate: (220 + 260) / 2,000 = 0.24.
- Calculate the standard error of the difference.
SE = sqrt( 0.24 × 0.76 × (1/1,000 + 1/1,000) )= sqrt(0.0003648) = 0.0191. - Calculate z.
z = (0.26 − 0.22) / 0.0191= 2.09. The observed gap is about two standard errors from zero. - Convert z to a p-value. For a two-sided test, p is the probability of landing beyond ±2.09 on a standard normal curve: p = 0.036.
At the 0.05 level this is significant, because 0.036 is below 0.05. If there were truly no difference between the concepts, a gap this large would show up in about 3.6% of samples like this one.
Notice how thin the margin is. A confidence interval for the gap, using the unpooled standard error of 0.0191, runs from about 0.3 to 7.7 percentage points. The data are consistent with a tiny edge and with a large one, and the p-value alone hides that. For a 2×2 table like this one, a chi-square test gives the same answer: z squared is 4.39, with the same p of 0.036.
What are the most common misreadings of p-values?
Is p = 0.05 a 5% chance the result is due to chance?
No, and this is the most common error, including in many popular explanations. The p-value is calculated assuming the null is true, so it cannot tell you the probability that the null is true. A p-value of 0.036 means “if there were no difference, data like this would be rare.” It does not mean “there is a 3.6% chance Concept A and B are the same.” Getting that second number requires information the p-value does not contain, such as how plausible a difference was before you collected data.
Does significant mean important?
No. Statistical significance tells you a difference is unlikely to be noise. It says nothing about whether the difference is big enough to matter. With enough respondents, trivial gaps become significant.
In an illustrative case, 22.0% against 23.0% with 50,000 respondents per group gives z = 3.79 and p = 0.00015, yet a 1-point gap may not change any decision. Reverse it and the same 1-point gap in samples of 1,000 each is nowhere near significant.
So always read the size of the effect next to the p-value. Ask whether a difference of that size would change what you do.
What happens when you run many tests?
Every test at 0.05 carries a 5% chance of a false alarm if nothing is going on. Run 20 independent tests on data with no real differences and the chance of at least one “significant” result is 1 − 0.95^20, about 64%. Banner tables with dozens of columns and rows are exactly this situation.
This is the sound version of a claim in the older version of this article: if you analyze data enough ways, some results will be significant by chance. The 5% figure applies when there is truly nothing there, so the exact count of false positives depends on how many real effects exist.
To guard against it, decide your key comparisons before the analysis, treat unplanned findings as leads to confirm rather than conclusions, and consider a stricter threshold. The simplest correction, Bonferroni, divides alpha by the number of tests, so 20 tests means testing each at 0.0025. When you are comparing more than two groups on one measure, a single analysis of variance avoids running many pairwise tests in the first place.
Does a non-significant result mean there is no difference?
No. It means you did not find enough evidence of one. Small samples have little power, so a real 4-point gap in samples of 200 each could easily return a p-value above 0.05.
How do you use p-values well in practice?
Report the exact p-value rather than “p less than 0.05”, and put the effect size and a confidence interval beside it. Treat 0.049 and 0.051 as nearly the same evidence, since nothing special happens at the line.
If you work with survey crosstabs, Halo Reports applies weighting and runs significance testing across banner tables for you, which saves the arithmetic. It does not decide which comparisons deserve attention. That judgment stays with you. Pulse can help with the watching, since it monitors trackers and studies and surfaces meaningful movement.
Next time a report highlights a significant difference, ask three questions before you act: how large is the gap, how many comparisons were run to find it, and would a gap that size change a decision? For continuous tracking of brand health, sentiment and competitors, see market and brand intelligence.