You have a draft questionnaire, a deadline, and a sense that it is probably fine. Most of the problems that ruin survey data are invisible from the author’s chair: a question that reads clearly to you and means something else to a respondent, a scale that two items push in opposite directions, an answer list with no option that fits. Survey validation is how you find them while they are still cheap to fix, and it is the first line of defense in survey data quality.
Key takeaways
- Survey validation is checking, before launch, that a questionnaire measures the concept you care about and does so consistently.
- Validity asks whether a survey measures the right thing and reliability asks whether it does so consistently; reliability is necessary for validity but never enough.
- Validate in eight steps: map questions to objectives, get expert review, run cognitive interviews, pilot, clean the pilot data, check factor structure and alpha, then revise.
- Cronbach’s alpha of 0.70 or higher is a common benchmark for a multi-item scale, and a very high value can mean near-duplicate items.
- Skip factor analysis and alpha for standalone questions; rely on expert review, cognitive testing and a pilot instead.
What does survey validation mean?
Validating a survey means checking, before launch, that the questions measure the concept you care about (validity) and that they do so consistently (reliability). It is a set of tests, not a single step, and it covers wording, structure, scales and the data the survey produces.
It is worth separating this from a different use of the word. Survey tools have a setting called “validation” that forces a number to be entered as a number, or an email field to look like an email. That is data-entry validation. It keeps junk out of a field, but it says nothing about whether the question itself is any good.
A published method is a useful anchor. Dave Collingridge, a research statistician, set out a six-step validation approach in an article on Sage’s MethodSpace: expert review, a pilot test, data cleaning, principal components analysis, an internal consistency check, and revision. The procedure below follows that spine and adds the respondent-facing and analysis-facing checks that practitioners usually build around it.
What is the difference between validity and reliability?
Validity asks whether the survey measures the right thing. Reliability asks whether it measures that thing consistently. They are related but not interchangeable.
A bathroom scale that always reads five pounds heavy is reliable and not valid. A survey works the same way. A “satisfaction” scale made of questions that really measure how long someone has been a customer may give stable results every time, and still answer the wrong question. Reliability is necessary for validity, but it is never enough.
Older write-ups, including the earlier version of this page, sometimes use “dependability” as a catch-all for both. Keep the two terms apart. They fail in different ways and you test them with different tools.
What are the types of validity?
Four types come up in practice. Each has a different check.
| Type | The question it answers | How you check it |
|---|---|---|
| Content validity | Do the questions cover the whole concept? | Subject experts review the questionnaire against the study objectives |
| Face validity | Do the questions look reasonable and relevant to respondents? | Cognitive interviews and pilot feedback |
| Construct validity | Do items meant to measure one concept behave as a group? | Factor analysis and reliability statistics on pilot data |
| Criterion validity | Does the measure relate to an outcome it should predict? | Compare to an external result, such as stated intent and later purchase |
One correction to common usage: having topic experts review your questions establishes content validity. Face validity is about how the questions strike the people answering them. Both matter, and you check them with different groups.
How do you validate a survey, step by step?
Here is the procedure we would follow for a new questionnaire. Skip the statistical steps if your survey has no multi-item scales, and see the section on when not to use them below.
Step 1: Write the objectives and map every question to one
List the decisions the survey will inform. Then tie each question to a decision. A question that informs nothing costs respondent time and adds noise, so cut it. This is also the moment to prepare the analysis side, which is covered in four pre-survey steps for getting the most from analysis software.
Step 2: Get two expert reviews
Ask people who know the subject to check whether the questions capture the topic (content validity). Ask a second reviewer who knows questionnaire design to look for leading wording, confusing phrasing and double-barreled questions, which ask about two things at once. They are different skills, so use different people where you can.
Step 3: Run cognitive interviews
Recruit five to ten people from your target group and ask them to think aloud while they complete the survey. You are listening for hesitation, rephrasing and “what do you mean by this?” moments. This is where you find missing answer options, terms respondents read differently from you, and questions that feel loaded. It is the strongest check on face validity.
Step 4: Pilot the survey
Field the final-form survey to a small sample drawn from the same population as the main study. A pilot of roughly 30 to 100 completes is typical. It is large enough to show completion time, drop-off and response patterns, and to run reliability statistics on short scales. Do not use a different audience for convenience, because their answers will tell you little about how the real sample behaves.
A pilot is not a mini-launch for headline results. Treat its data as test data and keep it out of the final dataset unless the survey did not change.
Step 5: Clean the pilot data and check the basics
Load the responses and look for problems before any statistics.
- Out-of-range values. On a five-point scale, a 6 is an entry or programming error.
- Reverse coding. If you worded some items negatively, flip them so a high score means the same thing on every item. On a 1 to 5 scale the recoded value is 6 minus the original. A respondent who gives 5 to “The checkout was easy” and 4 to “The checkout was hard” is contradicting themselves. After recoding, the second answer becomes 2, and the gap makes the inconsistency visible.
- Straight-lining. Count respondents who give the same answer to every row of a grid. In an illustrative pilot of 60, if 9 do this on a 10-item grid, that is 15% (9 divided by 60). A rate that high says the grid is too long or the items too alike.
- Item non-response and drop-off. Questions people skip, or the page where they quit, point to wording or length problems.
- Time to complete. Compare the median to what you promised respondents.
Then build the key tables the objectives call for. If a question cannot be analyzed the way the decision needs, fix it now.
Step 6: Check the factor structure of your scales
If a set of questions is meant to measure one concept, test whether they act that way. Factor analysis, or principal components analysis, which many practitioners use for the same purpose, shows which questions group together. Each question has a loading between negative 1 and 1 on each factor. Conventions vary; many analysts treat about 0.4 as a floor and look for 0.6 or higher on the items that define a factor.
Read the result for sense, not just for numbers. A question that loads on no factor either measures something different or is poorly worded. If two groups of items emerge from what you planned as one scale, you may be measuring two concepts.
Stable factor results need more respondents than many pilots have. Rules of thumb often ask for five to ten respondents per item, so a 12-item scale calls for 60 to 120 completes.
Step 7: Check internal consistency
For each scale, calculate Cronbach’s alpha, which summarizes how closely the items move together. It ranges from 0 to 1. A common benchmark is 0.70 or higher, and some exploratory work tolerates 0.60. Higher is not always better: a very high value can mean the items are near-duplicates.
The next section works through a calculation.
Step 8: Revise, re-test and document
Drop or rewrite questions that failed. If a question matters but does not fit a scale, keep it and analyze it on its own. After minor edits you can usually proceed. After major ones, such as removing several items or rewording a scale, run another pilot and repeat steps 5 to 7.
Then write down what you did. Reporting that experts reviewed the questionnaire, that it was piloted, and that scales were checked for reliability tells readers how much to trust the numbers.
How do you calculate Cronbach’s alpha for a scale?
The formula, in plain text:
alpha = k / (k - 1) x (1 - sum of item variances / variance of total score)Here k is the number of items, and the total score is each respondent’s sum across the items. In this illustrative example, a pilot gave a four-item “ease of use” scale. Each item uses a 1 to 5 scale. The numbers below are made up to show the arithmetic.
| Quantity | Value |
|---|---|
| Items (k) | 4 |
| Variance of item 1 | 1.0 |
| Variance of item 2 | 1.2 |
| Variance of item 3 | 0.9 |
| Variance of item 4 | 1.1 |
| Sum of item variances | 1.0 + 1.2 + 0.9 + 1.1 = 4.2 |
| Variance of total score | 10.0 |
Then:
- k / (k - 1) = 4 / 3 = 1.333
- Sum of item variances divided by total variance = 4.2 / 10.0 = 0.42
- 1 - 0.42 = 0.58
- alpha = 1.333 x 0.58 = 0.77
An alpha of 0.77 clears the common 0.70 benchmark, so the four items hang together acceptably. Now suppose one of the items was negatively worded and nobody reverse coded it. It would move against the other three, shrink the variance of the total score, and pull alpha down. That is why reverse coding comes before the reliability check.
If alpha comes out low, many statistics packages report “alpha if item deleted”. An item whose removal raises alpha is a candidate for rewriting or cutting. Test the change, rather than assuming it.
What problems does survey validation catch most often?
Validation finds the same handful of faults again and again:
- Double-barreled questions, such as “How satisfied are you with the price and quality?”
- Leading wording that signals the preferred answer.
- Answer lists that are not exhaustive or overlap, such as age bands of 18 to 25 and 25 to 34.
- Scales without a neutral point where respondents need one, or with unlabeled midpoints.
- Order effects, where an earlier question colors the next.
- Terms that respondents interpret differently from the researcher.
Each is cheap to fix before fielding and expensive afterward.
What are common mistakes in validating a survey?
- Treating the pilot as a formality. Piloting on colleagues, or on ten people who will not be in the real sample, finds typos but not comprehension problems.
- Running factor analysis on too few respondents. A 30-person pilot can show you drop-off and alpha, but its factor structure will shift when you collect more data.
- Chasing a high alpha. Adding near-identical questions raises alpha and lengthens the survey without improving measurement.
- Skipping reverse coding. Negatively worded items that are not flipped before analysis distort both alpha and any summed score.
- Validating once and never again. A tracker changes when wording, sample source or mode changes. Recheck scales at each wave and test any wording change against the prior wave.
When should you not use factor analysis and alpha?
Not every survey needs scale statistics. If your questions are standalone, such as “Which brands have you bought in the past three months?”, there is no scale to test. Alpha also tells you nothing about a single-item measure, and factor analysis on a few unrelated items produces noise.
In those cases, rely on expert review, cognitive testing and the pilot data check. Sampling quality is a separate issue that no questionnaire check can fix, and sampling error deserves its own planning.
How do you do this in practice?
The pilot only helps if you can analyze it the way you will analyze the real data. Halo Reports builds crosstabs and banner tables with weighting and significance testing, plus charts and interactive report pages, so you can set up the key tables from pilot data once and reuse them when the full study arrives. See Halo Reports for what it covers. If you validate questionnaires for wide studies and global trackers, one governed platform for your studies and trackers keeps that work in one place.
What should you do next?
Take the draft you have now and run step 3 this week: five people from your target group, thinking aloud, while you take notes and say nothing. It takes a morning and nearly always changes at least two questions. Then pilot, clean the data, and only reach for factor analysis and alpha if you have scales to test.