You have a new wave of results in front of you and last year’s numbers beside them. Awareness is up, satisfaction is down, and someone wants to know why by Friday. Before anyone explains the change, you need to know whether there is a change at all. This guide covers the checks that come first, as part of the wider market research process.
Key takeaways
- To compare past and present survey results, first confirm the waves are comparable: same question wording, answer scale, question order, sample frame, mode and weighting.
- If any of those changed, start a new trend line or run a bridging test instead of splicing the series together.
- A two-proportion z-test shows whether a gap is real, and a 4-point rise between two waves of 1,000 respondents is not significant at 95%.
- Check seasonality and outside events, and report both waves with sample sizes, significance and change in points rather than percent.
What makes two survey waves comparable?
Two waves are comparable when anything that could move the numbers, other than the thing you care about, stayed the same. Check these five points before you put the waves in one chart.
| Check | What to confirm | How it goes wrong |
|---|---|---|
| Question wording | Identical text, including the answer labels | A leading question in one wave and a neutral one in the next |
| Answer scale | Same scale length and anchors | A 1 to 5 rating in one wave, agree or disagree in the next |
| Question order | Same position, same preceding questions | An earlier question primes the answer to a later one |
| Sample frame | Same population and sampling method | A panel one year, a customer email list the next |
| Mode and weighting | Same way of asking, same weighting targets | Phone one wave, online the next; new age or region weights |
If one of these changed, you have two options. You can stop the trend line there and start a new one, or you can run a bridging test, asking both versions to split halves of one sample so you can measure the gap between them. Do not quietly splice the series together.
Write the answers down once, in a short methods log kept with the data. The person running wave six will not remember what happened in wave two. Connecting findings across studies and waves helps teams reuse what they know instead of starting over. If you are designing a tracker from scratch, defining the problem and the measures up front is what keeps later waves comparable.
How do you test whether a change is significant?
Two different samples of people will never give identical results, even if nothing in the market moved. Statistical significance tells you whether the gap between waves is bigger than that sampling noise would typically produce. For the background, see our explainer on what statistical significance is.
For a percentage that moves between two independent waves, the standard tool is a two-proportion z-test. Here is a worked example. The numbers are illustrative, not real data.
Worked example: awareness across two waves
Brand awareness was 52% in wave 1 and 56% in wave 2, with 1,000 respondents in each wave. Is the 4-point rise real?
- State the two results: p1 = 0.52 and p2 = 0.56, with n1 = n2 = 1,000.
- Pool them: p = (0.52 + 0.56) / 2 = 0.54. This is valid here because the samples are the same size.
- Compute the standard error: sqrt(0.54 x 0.46 x (1/1,000 + 1/1,000)) = sqrt(0.0004968) = 0.0223.
- Compute z: (0.56 - 0.52) / 0.0223 = 1.79.
- Compare with the cutoff. At the usual 95% level (two-sided), z must exceed 1.96. At 1.79, the two-sided p-value is about 0.07.
The 4-point rise is not significant at the 95% level. A 95% confidence interval for the difference runs from about minus 0.4 to plus 8.4 points, so a flat result and a good-sized gain are both plausible.
Now suppose wave 2 had come in at 58%. The pooled value is 0.55, the standard error is about 0.0222, z is about 2.70, and the p-value is about 0.007. That 6-point rise is significant.
Two lessons follow. A 4-point move can be noise at 1,000 respondents per wave, and the same move would clear the bar with larger samples. And “not significant” does not mean “no change”. It means you cannot tell, so say that.
Two cautions on the method. This test assumes simple random samples. Weighted data and complex designs need a larger standard error, so use software that accounts for the weights.
And if you test many questions at once, a few will look significant by chance. Decide in advance which measures matter.
How do seasonality and outside events affect the comparison?
Some differences come from the calendar or the news, not from your customers changing their minds. Retail sentiment moves around holidays, and travel and energy attitudes move with the seasons. A wave fielded in January and one in July can differ for reasons unrelated to anything you did.
Where you can, compare the same period year over year, or field waves at the same point each cycle. Then list events near each fielding window:
- Major news or weather events
- A product launch or discontinuation
- A change in your marketing or pricing
- A rebrand
- Press coverage of your business or a competitor
Lining up events with dates is a starting point, not proof. Two things happening in the same quarter does not show one caused the other. Treat each event as a hypothesis, and look for supporting evidence, such as whether the change appears in the groups that were exposed and not in those that were not. Our piece on deduction and induction in market research covers how to structure that reasoning.
How do you present wave-on-wave change honestly?
Show the numbers readers need to judge the result for themselves.
- Show both waves and the sample sizes. “52% (n=1,000) to 56% (n=1,000)” beats “up 4 points”.
- Flag significance, and say what the test was. Mark significant changes in the table, and state the confidence level used.
- Report change in points, not percent, for percentages. Going from 20% to 25% is 5 points. Calling it “a 25% increase” invites confusion.
- Use one baseline. Do not pick the starting wave that makes the story look best.
- Say when a change is within noise. “No reliable change” is a finding.
- Look at subgroups, but carefully. Movement among a segment can be real even when the total is flat, though smaller samples mean wider margins. See types of segmentation for ways to cut the data.
- Keep the axis honest. Truncated axes make small moves look dramatic.
Tracking tools can run these tests for you. Halo Reports, for example, supports crosstabs and banner tables with weighting and significance testing, so each wave is compared against the last. See Halo Reports. If you follow brand health, consideration, sentiment and competitors continuously, you can see what changed and why between waves.
What are the most common mistakes?
- Changing the question and keeping the trend line. Even small wording changes can move results. Bridge them or break the series.
- Treating every difference as a finding. Without a test, you are reading noise.
- Confusing “significant” with “important”. With large samples, trivial moves can pass the test.
- Explaining a move with the nearest event. Correlation is not causation.
- Ignoring a changed sample. A different recruitment source can look like a market shift.
What should you do next?
Open your last two waves and fill in the comparability table above for each tracked question. Mark the ones where something changed. Then run the test on the three measures your team acts on, and write the result down with its sample sizes and confidence level. That one page will save the next argument about whether anything really moved.