You have five product ideas, budget to develop one or two, and a launch date that does not move. Choosing on instinct is fast, but it is how weak ideas reach the shelf. Concept testing puts each idea in front of the people you want to buy it, in a consistent format, and measures how they react before you spend on development. This guide covers the main test designs, the metrics to report, sample size, action standards, and a worked example with the arithmetic shown.
Key takeaways
- Concept testing is a survey that measures reaction to an idea, such as a product, positioning, package, ad or name, before you build or launch it.
- Monadic designs, where each person sees one concept, give the cleanest read. Sequential monadic designs cost less but carry order effects.
- Report purchase intent as top-2-box, alongside appeal, uniqueness, relevance, believability and value for money.
- Plan 150 to 300 respondents per cell, and test the gap between concepts for significance before you name a winner. At 200 per cell, gaps under about 10 points are usually within noise.
- Set the action standard, the score a concept needs to move forward, before fieldwork begins, and base it on your own past tests.
What is concept testing, and when should you run it?
A concept is a short, plain statement of an idea: what it is, who it is for, what it does for them and, often, what it costs. A concept test shows that statement to a sample of your target audience and asks them to rate it. Because every concept is shown and rated the same way, you can compare ideas fairly and see which ones are worth more investment.
You can test almost any idea that has to win over a buyer:
- Product ideas. A new flavor, feature, service tier or line extension.
- Positioning. Which benefit or reason to believe should lead. This overlaps with message testing, which compares claims and wording in more depth.
- Packaging. Pack designs, formats and sizes, shown as images.
- Advertising ideas. Early ad concepts as boards or scripts, before production. Finished ads are judged later through creative testing.
- Names. Candidate brand or product names, tested for fit, appeal and memorability.
Run a concept test when you have more ideas than you can develop, when the cost of a wrong choice is high, and when the idea can be described clearly enough for people to react to it. It is less useful for ideas that only make sense once people try them, such as a new texture or a piece of software. There, a product test or prototype test tells you more.
Concept testing usually happens twice. An early screen narrows many rough ideas to a few. A later test refines the finalists, with sharper wording and a price, and checks them against an action standard.
What types of concept test are there?
The main design choice is how many concepts each respondent sees. That decides how clean the read is and how much sample you need.
| Design | What each respondent sees | Strengths | Weaknesses | Best for |
|---|---|---|---|---|
| Monadic | One concept | No order effects; closest to real life | A separate cell per concept | Final decisions |
| Sequential monadic | Several concepts, one at a time, rotated | Less sample | Order effects and fatigue | Screening many ideas |
| Comparative (paired) | Two or more side by side | Sensitive to small differences | No absolute score | Close variants |
| Proto-monadic | One concept rated fully, then a paired choice | Clean first read plus a preference check | Longer survey | Two finalists |
Monadic is the default for concept testing decisions that matter, because each score reflects the idea alone. Sequential monadic is a fair trade when budget is tight and you are screening, as long as you rotate the order and check the first-position scores. Our guide to monadic vs sequential monadic designs covers that trade-off in detail.
Which metrics does a concept testing survey measure?
Most concept tests measure a short, stable set of questions, asked the same way for every concept and every study. Keeping them stable is what lets you build a history of past results to compare against.
| Metric | What it tells you | Typical question |
|---|---|---|
| Purchase intent | Would people buy it? | ”How likely would you be to buy this?” (5 points) |
| Appeal | Do they like the idea? | ”Overall, how much do you like this idea?” |
| Uniqueness | Does it feel new? | ”How new and different is this?” |
| Relevance | Does it meet a need? | ”How well does this meet a need you have?” |
| Believability | Do they trust the claims? | ”How believable is this description?” |
| Value for money | Is it worth the price? | ”At this price, how good is the value?” |
Purchase intent is the headline. It is the closest concept testing gets to the purchase decision, the stage where the purchase funnel turns consideration into buying.
The other metrics explain it. A concept with high appeal but low believability, for example, has a claim problem, not an idea problem. For wording and scale points, see our guide to concept testing questions.
Why report top-2-box?
Top-2-box is the share of respondents who pick the two most positive points on the scale, such as “definitely would buy” plus “probably would buy”. It is easy to explain, and it focuses on the people most likely to act. A mean score mixes enthusiasts with people who are mildly negative, so two concepts can share a mean and still differ a lot in how many people really want them.
Read top-2-box as a relative measure, not a sales forecast. Stated intent tends to run ahead of actual buying, so 45% saying they would buy does not mean 45% will. It does mean the idea beat one that scored 30% under the same conditions.
How do you design the concept stimulus?
The stimulus is the concept as respondents see it. Most concept testing goes wrong here, not in the analysis.
- Use one format for every concept. Same length, same layout, same style of image, same price presentation. If one concept has a polished photo and another has a sketch, you are testing the art.
- Put one idea in each concept. If a concept has a new benefit, a new pack and a new price, you will not know which one drove the score.
- Follow a fixed structure. A common one is a consumer insight (the need), the benefit, the reason to believe, and the product details with price.
- Write in the buyer’s words. Avoid internal names and technical claims people cannot judge.
- Make it realistic. Show a price if the real product will have one. A concept without a price tends to score higher than it should.
How many respondents do you need per cell?
In concept testing, a cell is the group of respondents who see a given concept. In a monadic test of three concepts, you have three cells. Each cell should be matched on the same quotas, such as age, gender, region and category usage, so that differences come from the concept and not from who saw it.
Cell size decides how small a gap you can detect. The table shows the smallest difference in top-2-box that would test as significant at the 95% level, for two cells with scores near 45%.
| Respondents per cell | Standard error of the difference | Smallest significant gap |
|---|---|---|
| 150 | 5.7 points | about 11 points |
| 200 | 5.0 points | about 10 points |
| 300 | 4.1 points | about 8 points |
| 500 | 3.1 points | about 6 points |
These come from the standard error of a difference, √(2 × 0.45 × 0.55 / n), multiplied by 1.96. A range of 150 to 300 per cell suits most tests. Go higher when ideas are close, or when you need to read segments, because each segment within a cell has its own, smaller sample.
How do you set an action standard?
An action standard is the rule, agreed before fieldwork, that decides what happens next. For example: “Advance a concept if top-2-box purchase intent is at least X%, and if two pass, choose the one that is significantly ahead.” Writing it down first stops the team from moving the bar once results arrive.
The hard part is X. Concept testing has no universal good score, because purchase intent varies a lot by category, price point and country. The most useful benchmark is your own history of tests: past concepts in the same category, asked the same questions, with the same sample design. If you have no history yet, test against a control, such as your current product described in the same format, and require the new idea to beat it.
What does a concept test look like in a worked example?
Here is an illustrative example, not a real study. A snack brand has three concepts for a new protein bar:
- Concept A: a high-protein bar for everyday energy.
- Concept B: a bar with live cultures for gut health.
- Concept C: a low-sugar bar for children’s lunchboxes.
Each concept was shown monadically to its own matched cell of 200 category buyers. The action standard, based on the brand’s past tests, was top-2-box purchase intent of at least 35%.
First, top-2-box for Concept A. Of 200 respondents, 30 said “definitely would buy” and 64 said “probably would buy”. That is 94 out of 200, or 47%. The same calculation gives the other scores.
| Concept | Top-2-box purchase intent | Top-2-box uniqueness | Meets 35% standard? |
|---|---|---|---|
| A: Everyday protein | 94 / 200 = 47% | 62 / 200 = 31% | Yes |
| B: Gut health | 82 / 200 = 41% | 96 / 200 = 48% | Yes |
| C: Kids’ lunchbox | 58 / 200 = 29% | 54 / 200 = 27% | No |
Concept A leads on purchase intent by 6 points. Is that gap real? Use a two-proportion z test.
- Pooled proportion = (94 + 82) / (200 + 200) = 176 / 400 = 0.44
- Standard error = √(0.44 × 0.56 × (1/200 + 1/200)) = √(0.2464 × 0.01) = √0.002464 = 0.0496
- Difference = 0.47 − 0.41 = 0.06
- z = 0.06 / 0.0496 = 1.21
A z value needs to exceed 1.96 to be significant at the 95% level. At 1.21, the 6-point lead is within noise. With 200 per cell, as the earlier table showed, a gap of about 10 points would be needed.
Now uniqueness, where B leads A by 17 points. The pooled proportion is (96 + 62) / 400 = 0.395, and the standard error is √(0.395 × 0.605 × 0.01) = √0.00239 = 0.0489. So z = 0.17 / 0.0489 = 3.48, well above 1.96.
B is significantly more distinctive. C trails A on purchase intent by 18 points, with z = 3.71, and misses the standard, so it drops out.
The reading: A and B both pass the action standard, and neither is clearly ahead on intent. A is the safer, more familiar idea. B is the more distinctive one.
The data does not pick a winner for you. The next step is to find out why each concept scored the way it did.
How do diagnostics explain why a concept won?
The core metrics tell you which concept did best. Diagnostics tell you why, and what to change.
What do likes and dislikes reveal?
Ask two open-ended questions after the ratings: “What, if anything, do you like about this idea?” and “What, if anything, do you dislike?” Code the answers into themes and compare them across cells. In the example, suppose B’s dislikes cluster on doubt that a snack bar can deliver gut-health benefits. That is a believability problem, and you would expect it to show up in the believability rating too.
What do attribute ratings add?
Attribute ratings ask how well the concept fits statements such as “is good for me”, “tastes like a treat” or “is worth the price”. They show which parts of the idea people accept. If B scores well on “is good for me” but poorly on “I believe the claims”, the idea is sound and the proof is weak. A stronger reason to believe, tested again in a second round, may lift its purchase intent past A.
How do you read results by segment?
A total score can hide two very different reactions. Run crosstabs of each metric by segment, such as age, current brand or region. In practice, we see the most useful findings here: a concept that is average overall but strong with a group the brand wants to win.
Watch the base sizes, though. A cell of 200 split into four age groups leaves about 50 per group, so only large gaps will be significant. For crosstabs by segment with weighting and significance testing, Halo Reports builds banner tables across cells in one run.
What are the common mistakes in concept testing?
- Testing the art, not the idea. Uneven images, lengths or prices between concepts change the scores. Hold the format constant.
- Crowning a winner without a significance test. A 6-point lead with 200 per cell looks decisive on a chart and is not. Test every gap you plan to act on.
- Setting the action standard after the results. If the bar moves to fit the favorite, the test was a formality. Write the rule down before fieldwork.
- Treating top-2-box as a sales forecast. Stated intent overstates real buying. Use it to compare ideas, not to size the market.
- Stopping at the scores. A ranking without diagnostics tells you which idea won but not how to make it better.
How do you run concept testing in practice?
Start small and stay consistent. Good concept testing uses the same core questions, the same scale and the same stimulus format every time, because consistency turns single tests into a history you can benchmark against. Store each test’s results so the next team can find them. Pulling past concept tests, trackers and qualitative work together through research synthesis helps teams reuse what they already know instead of starting over.
For your next step, take the ideas on your current shortlist and write each one as a concept in the same structure: need, benefit, reason to believe, details and price. Then write your action standard before you brief any fieldwork. If you cannot agree on what score would make you proceed, settle that first, because it is the decision the whole test exists to inform.