You have a few thousand open-ended answers, a deadline, and an AI tool that says it can find the themes in minutes. AI thematic analysis can do a lot of that work, but it can also hand you a tidy set of themes that is quietly wrong. This guide covers what AI does well, how to check it, and how to keep the method intact. If you are new to the method itself, start with our guide to thematic analysis.
Key takeaways
- AI thematic analysis is fast at drafting a code frame, applying a codebook consistently and handling thousands of responses in several languages.
- It fails in predictable ways: plausible but wrong themes, missed minority themes, sarcasm and negation, invented quotes and drift between runs.
- Keep the researcher in charge: humans set the objectives and approve the codebook, and the AI codes against a locked version.
- Check a random sample by hand, report the agreement rate, and fix the codebook based on the disagreements.
- Trace every count to the responses behind it and verify every quote word for word.
How does AI fit into thematic analysis?
Thematic analysis has the same steps whether a person or a model reads: get familiar with the data, create codes, group them into themes, review and report. AI can help with three of them.
- Proposing codes and themes. The model reads a batch of responses and suggests a draft code frame with labels and short definitions.
- Coding responses. Given a codebook, the model assigns each response to one or more codes.
- Summarizing themes. The model drafts a description of each theme with example responses.
AI should not decide what the study is for or which themes matter. Those are research decisions. AI qualitative analysis is the broader term for the same pattern applied to interviews, reviews and support tickets. Our guide to qualitative coding explains how codes, definitions and codebooks work if you need the basics first.
What does AI thematic analysis do well?
- Speed at volume. Coding 2,000 responses by hand can take days. A model can code them in minutes, which frees time for review.
- First-draft code frames. Editing a draft is faster than starting from a blank page.
- Consistent application. A person coding at 5 p.m. drifts from the person coding at 9 a.m. A model with a locked codebook applies the same definitions to every response.
- Multilingual responses. In multi-market trackers, the model can code answers in several languages against one codebook.
These strengths are also why its mistakes are easy to miss: the output looks finished.
Where does AI get themes wrong?
- Plausible but wrong themes. The model may produce a theme that sounds right, such as “customer service frustration,” when most of those responses are about delivery times. The label fits the category, not the data.
- Missed minority themes. Models favor what is common. A theme mentioned by 3% of respondents can be the most important finding in the study and still get folded into “Other.”
- Sarcasm and negation. “Great, another late delivery” is a complaint. “It wasn’t the price, it was the recipes” is about recipes. Models still misread both.
- Invented quotes. Asked for examples, a model can blend responses into a sentence no respondent wrote.
- Drift between runs. Run the same prompt twice and you can get two different code frames. Without a locked codebook, your wave 2 themes will not match wave 1.
For these risks in survey work, see our guide to AI survey analysis.
What does a governed AI thematic analysis workflow look like?
A governed workflow keeps the method and lets AI handle the volume, while the researcher owns every decision.
- Set objectives first. Write down the decision the study supports. This tells you which themes matter.
- Have the AI draft a code frame, then review it. Read the draft against 100 or so raw responses. Merge, split, rename and define each code, with an example and a “do not code here” rule.
- Lock the codebook. Save a versioned copy. The AI codes every response against that version, with the same instructions each time.
- Check accuracy on a random sample. Code a random sample yourself, without looking at the AI’s codes, then compare.
- Trace counts to responses. Every number in the report should open to the responses behind it. If you want to ask questions of your own research in plain language, look for answers with the source shown.
- Verify quotes verbatim. Search the raw data for the exact text of every quote you plan to use. If it is not there word for word, do not use it.
How do you check AI coding on 2,000 responses?
Here is a worked example. The numbers are illustrative, not from a real study. A meal-kit company asks 2,000 former subscribers, “What is the main reason you cancelled?” The AI codes every response to one main theme using the approved codebook.
| Theme | Responses | Share of 2,000 |
|---|---|---|
| Price and value | 620 | 31.0% |
| Delivery problems | 380 | 19.0% |
| Recipe variety | 300 | 15.0% |
| Food quality | 280 | 14.0% |
| Too much packaging | 140 | 7.0% |
| Other | 160 | 8.0% |
| No usable answer | 120 | 6.0% |
| Total | 2,000 | 100% |
First, the arithmetic: 620 + 380 + 300 + 280 + 140 + 160 + 120 = 2,000. Every response has exactly one code.
Next, the researcher draws 200 responses at random and codes them blind. The two agree on 176 of them, so agreement is 176 / 200 = 88%. With a sample of 200, the 95% margin of error is about plus or minus 4.5 points, so true agreement likely sits between roughly 83.5% and 92.5%.
The 24 disagreements matter more than the 88%. Sort them by cause:
| Cause of disagreement | Responses |
|---|---|
| Negation or sarcasm read the wrong way | 8 |
| A missing theme: life changes such as moving or a new baby | 7 |
| Unclear line between Food quality and Recipe variety | 6 |
| Genuinely ambiguous, either code defensible | 3 |
| Total | 24 |
Each cause has a fix. Add a negation rule to the codebook (“code the reason the respondent gives, not the one they rule out”). Add a “Life change” theme: 7 of 200 is 3.5%, which suggests roughly 70 of the 2,000 responses belong there. Sharpen the Food quality and Recipe variety definitions with examples. Leave the 3 ambiguous cases alone; no codebook removes all judgment.
Then lock the new codebook version, re-run the coding and check a fresh random sample of 200. Agree the acceptable agreement level before you check, not after.
How should you handle privacy and data?
Open-ended answers often contain names, contact details and health information that respondents volunteer. Treat AI processing as a data-handling decision, not just an analysis step.
- Remove personal details first. Strip names, contact details and IDs before the text goes to any model.
- Know where the data goes. Confirm where responses are processed and stored, and whether the provider can use them to train its models.
- Check your consent terms. Make sure the original survey or interview consent covers this kind of processing.
- Limit access and retention. Your team should decide who sees coded data and for how long. A governed research repository helps here.
What are the common mistakes?
- Letting the AI set the objectives. You get the themes the model finds easiest to see, not the ones your decision needs.
- Accepting the first code frame. We see teams ship the draft unedited, then struggle to explain what a theme includes.
- Skipping the sample check. Without it, you cannot say whether the coding is 70% or 95% right.
- Changing the codebook mid-run. Edit, lock, then code. Mixed versions make counts meaningless.
- Quoting without checking. One invented quote in a readout can undermine trust in every finding next to it.
What should you do next?
Take one finished survey with open-ended answers, run AI thematic analysis on it, and do the full check: review the frame, lock it, code a random sample of 200 yourself, and sort the disagreements by cause. For a step-by-step guide to the wider process, see how to analyze open-ended survey responses, and for where AI fits across research, see AI for market research.
Once the method holds up, repeat it each wave. Recurring wave readouts reviewed by your team before they go out save time, and connecting findings across studies and waves lets teams reuse what they know instead of starting over.