A stakeholder messages late in the afternoon: did the spring campaign move consideration with younger buyers? The tracker holds the answer, but getting it usually means waiting for an analyst to build a table. Conversational analytics for survey data lets you ask that question in plain English and get a weighted, tested answer back in seconds. It is one of the categories covered in our guide to AI market research tools. This article explains how natural language survey analysis works, what a good answer contains and how to check one before you pass it on.
Key takeaways
- Natural language survey analysis turns a plain-English question into a crosstab, computes it and explains the result in words.
- Survey questions carry rules that business data does not: weights, bases, nets, multi-response questions, significance tests and waves.
- The safest design translates the question into an analysis specification that a statistical engine computes, rather than letting a model write one-off code.
- Check every AI answer for four things: base, weight, test and source.
- Use it for stakeholder questions, first cuts and report building; let a researcher drive custom nets and weights.
What is natural language survey analysis?
Natural language survey analysis is asking questions of survey data in everyday language and getting tables, charts and written answers back. You type “How does satisfaction differ by region?” instead of picking a row variable, a column variable and a filter from menus. The answer is the same crosstab an analyst would build, plus a short written reading of it.
The broader term is conversational analytics, or “talk to your data.” Most conversational analytics products were built for business data such as sales by month. Survey data is different, and the difference decides whether an answer can be trusted.
How is it different from conversational analytics on business data?
Conversational analytics on survey data has to respect the study’s design, while business data usually needs only a sum or a count. A survey estimates what a population thinks from a sample, so a natural language query has to respect these rules:
- Weights. Most trackers weight respondents to match the population; an unweighted percentage is a different number.
- Bases. “Satisfaction among customers” might mean all respondents or only those routed to the question.
- Nets. “Top-two box” or “any streaming service” are defined groupings of answer codes, and the definition must match the study’s.
- Multi-response questions. When people can tick several brands, percentages add to more than 100%, and the base is respondents, not answers.
- Significance. A three-point gap on 150 people may be noise, and the answer should say so.
- Waves. Trackers repeat over time, so “since last wave” has to resolve to a specific wave and comparable questions.
How does natural language survey analysis work under the hood?
Natural language survey analysis works by translating your question into an analysis specification, computing it with a statistical engine and then having the AI explain the result.
- Translate. The model reads the question and turns it into an analysis specification: which question goes in the rows, which in the columns, which filter, which weight, which waves, which test.
- Compute. A statistical engine runs that specification against the study data, applying the study’s own weights, net definitions and bases.
- Explain. The model writes a short answer from the computed table and shows the specification, so you can see exactly what was run.
The alternative is a general-purpose assistant such as ChatGPT, Claude, Gemini or Copilot writing ad hoc code against an exported file. The code is new every time, it rarely knows the study’s weights or nets, and two people asking the same question can get different numbers. Our article on using ChatGPT for survey analysis covers where that approach fits.
What should a good answer contain?
A good answer names the table it computed and the rules it applied, not just the headline number. The table shows what common questions imply.
| Question | Crosstab it implies | What must be applied |
|---|---|---|
| ”Which age group is most likely to recommend us?” | Recommend (0–10) by age band | Weight; NPS net definition; test between age bands |
| ”Did awareness of Brand B rise this quarter?” | Aided awareness by wave | Weight; same question wording in both waves; test between waves |
| ”What do lapsed buyers dislike most?” | Coded dislikes by buyer status | Base of lapsed buyers only; multi-response base is respondents |
| ”Is the premium segment more satisfied in Germany?” | Satisfaction by segment, filtered to Germany | Weight; small-base warning; test between segments |
How would it answer “Has consideration moved among 25–34s since last wave?”
The numbers below are illustrative. The question translates to: brand consideration, filtered to ages 25–34, by wave, last two waves, weighted and tested.
| Wave | Base (ages 25–34) | Considering the brand | Consideration |
|---|---|---|---|
| Wave 7 | 400 | 148 | 37.0% |
| Wave 8 | 400 | 176 | 44.0% |
| Change | +7.0 points |
For clarity, the table shows unweighted counts. A two-proportion z-test checks whether the change is larger than sampling error.
- Pooled proportion: (148 + 176) ÷ (400 + 400) = 324 ÷ 800 = 0.405.
- Standard error: √[0.405 × 0.595 × (1/400 + 1/400)] = √(0.240975 × 0.005) = √0.001205 = 0.0347.
- z-score: (0.440 − 0.370) ÷ 0.0347 = 2.02.
- Compare: 2.02 is above 1.96, the cutoff for 95% confidence in a two-sided test (p ≈ 0.04).
The written answer should read something like: “Consideration among 25–34s rose from 37% in Wave 7 to 44% in Wave 8, a 7-point increase that is significant at 95% confidence. Base: 400 respondents aged 25–34 in each wave.”
Now apply the weight. Weighting usually lowers the effective base, the sample size the weighted data is worth for testing. If the weights carry a design effect of 1.3, each wave’s effective base is 400 ÷ 1.3 ≈ 308. The standard error becomes √(0.240975 × 2/308) = 0.0396, and z = 0.070 ÷ 0.0396 = 1.77, below 1.96 (p ≈ 0.08). The same 7-point move is no longer significant at 95%. This is why an answer has to say which weight and which base it used.
How do you check an AI answer about survey data?
You check an AI answer by confirming four things: the base, the weight, the test and the source. If any one of them is missing from the answer, ask for it before you share the number.
- Base. Who is in the denominator, and how many? A filter applied in the wrong place changes everything.
- Weight. Is the result weighted, with which weight, and is the test run on the effective base?
- Test. Which test, at what confidence level, comparing what to what?
- Source. Which study, which wave, which question wording? You should be able to rebuild the table yourself.
Our guide to AI survey analysis covers checks for coded open-ended data. In mTab, plain-language questions return crosstabs, charts and written answers that are weighted, significance tested and traced to the study, wave and base.
Where does natural language help most, and where should a researcher drive?
Natural language helps most when the question is clear and the study is already set up; a researcher should drive when the analysis needs new definitions. Three uses work well:
- Stakeholder questions. Product and brand leads can ask questions in plain language and get answers from your own research, with the source shown, instead of queueing for a table.
- First cuts. Analysts can scan many cuts quickly to find the ones worth a proper look.
- Report building. Answers can be added to reports, pages or slides, which our guide to AI report generators explores.
A researcher should drive when the question needs a complex net that does not exist yet, a custom weight, or a judgment about whether a change is real or a shift in sample. The AI can compute these once defined; defining them is research work.
How does conversational analytics compare with a dashboard?
A dashboard answers the questions someone predicted; conversational analytics answers the one you have now. Dashboards suit a fixed set of measures everyone reads the same way. Conversational analytics suits follow-up questions and new cuts, as long as each answer shows its working.
What are the common mistakes?
- Accepting an unweighted number. As the example shows, weighting can turn a significant change into a non-significant one.
- Ignoring small bases. A filter to “premium buyers in Germany aged 25–34” can leave 40 people.
- Comparing waves with different question wording. If the question changed, the trend is not a trend.
- Summing multi-response percentages into a net. They overlap, so the sum overstates the net.
- Sharing a number without its source. If no one can trace it to a study, wave and base, it will be challenged.
What should you do next?
Pick a recent stakeholder question that took a day to answer, and write down the crosstab, weight, base and test it really needed. Use it as your test case for any natural language survey analysis tool. For recurring readouts, you can recurring analysis and reporting as governed AI workflows with a person approving. To reach governed answers from other AI assistants, see MCP for market research.