The survey has closed, the responses are sitting in a data file, and the real work starts. Survey analysis is where scores, percentages and comments become something a team can decide on, and it goes wrong in predictable ways: dirty data, unweighted samples, differences that are noise, and findings no one can act on. This guide gives you the survey data analysis method we use, in order, with a worked example you can check by hand.
Key takeaways
- Survey analysis turns raw responses into findings: clean, weight, run toplines, crosstab by key groups, test significance, code open-ends, then chart and report.
- Decide the business questions and the groups you will compare before you open the data file.
- A difference between groups is only a finding once a significance test says it is larger than sampling noise.
- Report every number with its base, and lead with what changed and what to do about it.
What should you settle before you analyze survey data?
Start with the questions the business needs answered, and write them down. A survey on customer satisfaction might need to answer “Which part of the experience is dragging satisfaction down, and for which customers?” That sentence tells you the variables you need: an overall satisfaction score, ratings for product quality, service and pricing, and the groups you will compare, such as customer tenure or region.
A variable is simply a characteristic you measure. List each one, note its question type, and write the tables you expect to build. This short analysis plan keeps you from wandering through the data looking for something interesting, and it gives you a standard to check the results against.
What are the three core types of survey analysis?
Nearly every survey analysis technique does one of three jobs: it describes the answers, compares groups, or relates one measure to another. Most projects need all three, in that order.
| Approach | The question it answers | Typical methods | Example |
|---|---|---|---|
| Describe | What did people say? | Frequencies, percentages, means, medians, standard deviations, top-2 box scores | 62% rated satisfaction 4 or 5 on a 5-point scale |
| Compare | Do groups, or waves, differ? | Crosstabs, significance tests, t-tests, analysis of variance | Satisfaction among customers of one year or less versus longer-tenured customers |
| Relate | What moves together? | Correlation analysis, regression, driver analysis, factor analysis | Which attribute ratings track overall satisfaction most closely |
Describing is the foundation, and it is where a lone average tends to mislead. Comparing is where most business value sits, because almost every decision is about a particular group. Relating is the most powerful and the easiest to over-read, since a pattern between two measures shows association, not cause.
How do your question types limit the analysis?
The kind of question you asked decides which statistics are valid. Four levels of measurement cover most surveys.
- Nominal: categories with no order, such as “Which brand do you use most often?” Report counts and percentages.
- Ordinal: ordered categories, such as “Rank these reasons for choosing your phone” or an agreement scale. Report percentages and medians; be cautious with averages.
- Interval: ordered with equal spacing but no true zero. A 0 to 10 rating is commonly treated this way, which allows means and standard deviations.
- Ratio: equal spacing and a true zero, such as “How many trips did you take last year?” All the usual statistics apply.
Pick the method after you know the level. A mean of a nominal code, or a correlation on unordered categories, produces a number but not a meaning. The same logic helps when you read charts later, because the question type also decides which chart is honest.
How do you do survey data analysis, step by step?
The steps below run in a fixed order, because each one protects the next. The worked example uses illustrative numbers: a satisfaction survey with 1,040 completes, fielded to a population that is 50% women and 50% men.
Step 1: How do you clean and check the data?
Cleaning comes first because every later number inherits its mistakes. Check for the following:
- Duplicates. The same respondent counted twice.
- Incomplete responses. Decide in advance how much missing data disqualifies a record, and apply the rule to everyone.
- Speeders. Respondents who finished far faster than a reasonable reader could.
- Straight-liners. Respondents who gave the same answer to every item in a grid.
- Failed attention checks. Questions that ask the respondent to choose a specific answer.
- Impossible values. An age of 214, or a “never purchased” respondent who rates the checkout.
Then recode where needed: group open numeric answers into bands, collapse categories that are too sparse to report, and build derived variables such as a segment label or a combined scale score. In our example, removing 40 speeders and duplicates leaves 1,000 usable respondents, 600 women and 400 men. Keep a log of every record you drop and why, so the base is defensible.
Step 2: How do you weight survey data?
Weighting adjusts the sample so it matches the population on characteristics you know, such as gender, age or region. Here the sample is 60% women, but the population is 50%, so women are over-represented.
The weight for each group is its population share divided by its sample share.
| Group | Sample size | Sample share | Population share | Weight | Weighted base |
|---|---|---|---|---|---|
| Women | 600 | 60% | 50% | 0.50 / 0.60 = 0.833 | 600 x 0.833 = 500 |
| Men | 400 | 40% | 50% | 0.50 / 0.40 = 1.25 | 400 x 1.25 = 500 |
Suppose 70% of women and 50% of men gave a satisfaction score of 4 or 5. Unweighted, the total is (600 x 0.70 + 400 x 0.50) / 1,000 = 62%. Weighted, it is (500 x 0.70 + 500 x 0.50) / 1,000 = 60%. The two-point gap exists because the unweighted sample leaned toward the more satisfied group.
Weighting is not free. Large weights make a few respondents count for a lot and reduce the effective sample size, which widens the margin of error. Record both the weighted and unweighted bases, and show the unweighted base on tables so readers can judge how much data sits behind each figure.
Step 3: How do you run toplines?
A topline is the overall result for each question: the frequency table, the percentage for each answer, and the mean or median for scales. Run toplines for every question before you cut anything. They show you the shape of the data, reveal coding errors that cleaning missed, and give you the headline numbers.
For rating scales, report the top-2 box (the share choosing 4 or 5 on a 5-point scale) alongside the mean, since the share who are clearly satisfied is easier to explain than an average of 3.8. In the example, the weighted topline is 60% top-2 box satisfaction. A topline is a starting point, because one overall number can hide very different results underneath.
Step 4: How do you crosstab by key groups?
Almost every business question is “for whom,” so cut the key measures by the groups the decision depends on. A crosstab puts the answers to one question in rows and a grouping variable, called the banner, in columns. Use the same banner on every question so readers learn where to look. Our guide to survey crosstabs walks through bases, nets and weights question by question.
| Top-2 box satisfaction | Total (weighted) | Women | Men |
|---|---|---|---|
| Base (unweighted) | 1,000 | 600 | 400 |
| Satisfied (4 or 5) | 60% | 70% | 50% |
Now the story changes. Satisfaction is 20 points higher among women than men, and the total of 60% describes neither group. Look for differences between segments and for movement concentrated in one group that the total hides. Choose banner groups that map to decisions, such as customer versus prospect, tenure, product line, region or age, rather than every available variable. When a follow-up cut comes up that your plan did not list, you can ask your data in plain language and get an answer from your own research, with the source shown.
Multiple-response questions need special handling, because respondents can pick more than one option and percentages sum to more than 100. The article on filtering multiple-response questions shows how to filter and cut on any selected answer without distorting the base.
Step 5: How do you test for significance?
A difference between two groups in a sample may be real or may be sampling noise. A significance test tells you how likely a gap this size would be if the groups did not differ in the population. See what statistical significance means for the full logic, and significance testing on a crosstab for how the letters in a table are produced. For a comparison of two percentages, a standard approach is a z-test:
z = (p1 - p2) / sqrt( p x (1 - p) x (1/n1 + 1/n2) )Here p1 and p2 are the two percentages, n1 and n2 their bases, and p is the pooled percentage across both groups. At the 95% confidence level, a z above about 1.96 (or below -1.96) counts as significant.
First, the women versus men gap. With 70% of 600 and 50% of 400, the pooled p is (420 + 200) / 1,000 = 0.62. The denominator is the square root of 0.62 x 0.38 x (1/600 + 1/400) = 0.2356 x 0.004167 = 0.000982, which is 0.0313. So z = 0.20 / 0.0313 = 6.4, far past 1.96. The gap is significant.
Now a tracker example. Satisfaction moves from 58% in wave 1 to 61% in wave 2, with 1,000 respondents in each wave. Pooled p is 0.595, so the denominator is the square root of 0.595 x 0.405 x (2/1,000) = 0.2410 x 0.002 = 0.000482, which is 0.0220. So z = 0.03 / 0.0220 = 1.37, below 1.96. The three-point rise is not significant, so report it as directional, not as improvement.
Two rules keep this honest. State the confidence level and show base sizes on every table. And when you test dozens of cells, expect a few false positives by chance, so give more weight to differences that are large, that match your hypotheses, or that persist across waves. Weighted data needs tests that account for the weights, which survey software typically does.
Step 6: How do you analyze open-ended responses?
The numbers say what moved; the open-ended answers say why. Read a sample of the actual responses first, before any counting, so you know what people are talking about. Then code the responses into themes, such as “slow delivery” or “helpful staff,” and count how many respondents mention each.
The coded themes become a variable you can crosstab like any other. Cross the themes with score groups and segments: what do the dissatisfied say, and how does that differ from the satisfied? In the example, you might find that men’s lower satisfaction clusters around one theme. Treat that as a hypothesis worth testing, not a finding yet. There are trade-offs in asking these questions at all, covered in the benefits and challenges of open-ended survey questions.
Step 7: How do you chart and report the findings?
Charts and a clear write-up are how the analysis reaches the people who decide. Choose the chart for the comparison you want to show: bars for comparing groups, columns or lines for change over time, stacked bars for scale distributions. Charting survey data covers how to match chart type to question type and avoid misleading displays.
Write each finding in three parts: what changed or differs, why it matters, and what to do next, with the table or chart behind it. For the example: women rate satisfaction 20 points higher than men (what), men are half of the customer base (why it matters), so investigate the service theme men mention most (what next). Route each finding to the team that can act, and measure in the next wave whether the action moved the number. Pulling in a regression or segmentation at this stage is worthwhile only if it serves a decision; otherwise it adds complexity without a recommendation.
How do you do survey analysis in practice?
Doing this in a spreadsheet works for a small survey, but weighting, multiple-response questions, nested banners and significance testing quickly outgrow it. Halo Reports builds crosstabs and banner tables with weighting and significance testing, turns them into charts, and publishes interactive report pages, so the steps above run on one set of tables. See Halo Reports for how it works. Teams that run wide, complex studies and global trackers can keep all of it in one governed platform for insights and research teams.
What mistakes make survey analysis go wrong?
- Skipping the plan. Without written objectives, you test everything and report whatever looks interesting.
- Reporting only the total. The overall figure hides segment differences, as the 60% satisfaction figure did above.
- Explaining non-significant differences. A three-point wiggle on a base of 1,000 is often noise. Test first, then explain.
- Ignoring the base. A percentage from 30 respondents cannot carry the weight of one from 600. Show the base every time.
- Treating association as cause. Two measures that move together may both depend on something you did not measure.
What should you do next?
Take your last survey and write the survey analysis plan you wish you had before fielding: the decisions it should inform, the banner groups, and the tables. Then run the seven steps in order, and put every finding into the what, why and next-step format before you build a single slide.