You have a brand trust score in your tracker, and it moved four points this quarter. Before anyone acts on it, two questions need an answer. Would the score have come out the same if nothing had changed, and does it actually measure trust? Those are the questions of reliability and validity, and they sit at the center of survey data quality. This guide explains both terms, how to check each on a real questionnaire, and works an example with the arithmetic shown.
Key takeaways
- Reliability is consistency: the same measure gives the same result when nothing real has changed. Validity is accuracy: the measure captures the concept you intend.
- A measure can be reliable without being valid, but it cannot be valid without being reliable.
- Check reliability with test-retest, internal consistency, inter-rater or parallel-forms methods.
- Build validity evidence of four kinds: face, content, construct and criterion.
- In trackers, keep wording, scales and translations stable across waves and markets, or you cannot tell real change from measurement change.
What do reliability and validity mean?
Reliability asks whether a measure is consistent: would the same people, asked the same questions under the same conditions, give the same scores? Validity asks whether the measure is accurate. Does a “brand trust” scale measure trust, or something nearby like familiarity?
The classic picture is a target. Shots tightly grouped in the bullseye are reliable and valid. Shots tightly grouped in one corner are reliable but not valid: consistent, and consistently wrong. Scattered shots are neither, because you cannot hit the bullseye on purpose without first being consistent.
What is the difference between reliability and validity?
| Reliability | Validity | |
|---|---|---|
| Question it answers | Does the measure give consistent results? | Does the measure capture the intended concept? |
| What it protects against | Random error | Systematic error and measuring the wrong thing |
| Typical survey check | Re-ask, compare items, compare coders | Expert review, pilot, compare with related measures and outcomes |
| Main statistic | Correlation, Cronbach’s alpha, kappa | Correlations with other measures and outcomes |
What are the types of reliability, and how do you check each in a survey?
Test-retest reliability
Ask the same respondents the same questions twice and correlate the two sets of scores. Too short a gap and people remember their answers; too long and their views may really change. Use it only for attitudes you expect to stay stable over the gap.
Internal consistency
When several items measure one concept, check that they move together in a single administration. Cronbach’s alpha is the usual statistic, and our guide to how Cronbach’s alpha can strengthen your survey walks through the calculation and how to read it.
Inter-rater reliability
When people code open-ended answers into a code frame, two coders should put the same verbatim into the same code. Have both code the same set of responses and compare. Percent agreement is the simple measure; Cohen’s kappa corrects that agreement for the matches you would expect by chance.
Parallel-forms reliability
Write two versions of a question set that measure the same thing, give both to the same respondents, and correlate the scores. It matters when you refresh tracker wording without wanting to break the trend.
What are the types of validity in survey research?
In research, validity and reliability are judged differently: reliability usually comes down to one coefficient, but validity is never proven with one number. You build a case from several kinds of evidence, and validating a survey before launch sets out the full procedure.
| Type | What it asks | Survey example |
|---|---|---|
| Face | Do respondents see the questions as relevant and clear? | Cognitive interviews show people read “trust” as intended |
| Content | Do the items cover the whole concept? | A trust scale covers honesty, dependability and keeping promises |
| Construct: convergent | Does the score agree with measures of the same or related concepts? | Trust correlates with an existing satisfaction measure |
| Construct: discriminant | Does it stay apart from concepts it should not match? | Trust correlates only weakly with brand awareness |
| Criterion: concurrent | Does it match an outcome measured at the same time? | Current customers score higher than lapsed ones |
| Criterion: predictive | Does it forecast a later outcome? | High scorers repurchase more in the next three months |
Systematic error is the main threat to validity. A scale whose items are all worded positively can produce consistent scores that partly reflect a habit of agreeing, one form of response bias, rather than the attitude itself.
How do you check reliability and validity on a real scale?
The numbers below are illustrative, and eight respondents is far too few for a real study. They show the arithmetic. A four-item customer trust scale, each item rated 1 to 5, gives a total from 4 to 20. The same eight customers answered it twice, two weeks apart.
Step 1: Calculate the test-retest correlation
Both sets of totals have a mean of 14 (112 ÷ 8). Subtract the mean from each score, multiply the two deviations for each respondent, and add them up.
| Respondent | Time 1 | Time 2 | Time 1 − 14 | Time 2 − 14 | Product |
|---|---|---|---|---|---|
| R1 | 18 | 17 | 4 | 3 | 12 |
| R2 | 12 | 13 | −2 | −1 | 2 |
| R3 | 15 | 15 | 1 | 1 | 1 |
| R4 | 9 | 10 | −5 | −4 | 20 |
| R5 | 16 | 14 | 2 | 0 | 0 |
| R6 | 11 | 11 | −3 | −3 | 9 |
| R7 | 14 | 15 | 0 | 1 | 0 |
| R8 | 17 | 17 | 3 | 3 | 9 |
| Sum | 112 | 112 | 53 |
The squared Time 1 deviations add to 16 + 4 + 1 + 25 + 4 + 9 + 0 + 9 = 68. The squared Time 2 deviations add to 9 + 1 + 1 + 16 + 0 + 9 + 1 + 9 = 46. The Pearson correlation is:
r = 53 / √(68 × 46) = 53 / √3,128 = 53 / 55.93 = 0.95A test-retest correlation of 0.95 says respondents kept nearly the same order across the two weeks. Check the means too, because correlation ignores a shift where everyone moves up two points. Both means are 14, so there is no drift.
Step 2: Check criterion validity against repeat purchase
A reliable score still has to mean something. Match each respondent’s Time 1 score to whether they bought again within three months, taken from the company’s own customer records.
| Group | Trust scores | Repurchased |
|---|---|---|
| Top four scorers | 15, 16, 17, 18 | 3 of 4 (75%) |
| Bottom four scorers | 9, 11, 12, 14 | 1 of 4 (25%) |
The four repurchasers average 16.25 ((18 + 16 + 14 + 17) ÷ 4) and the four non-repurchasers 11.75 ((12 + 15 + 9 + 11) ÷ 4). Coding repurchase as 1 or 0 and running the same Pearson formula gives a correlation of 0.77. Higher trust goes with more repurchase, which is evidence of predictive validity. On a real sample you would test whether that gap is statistically significant before claiming it.
How do you keep reliability and validity in a tracker across waves and markets?
In a tracker, every change to the instrument competes with real change as the explanation for a moved score. Keep wording, answer options, scale points and labels identical across waves. A switch from five points to seven can shift scores on its own, and the scale type also limits which statistics are fair to run, as covered in mind your measurement scales in market research.
Across markets, translation is the usual weak point. Use forward and back translation, have a local researcher confirm that terms carry the same meaning, and recalculate alpha in every market and wave. Keeping the analysis in one governed platform for wide studies and global brand trackers holds definitions consistent across waves, so a moved score is less likely to be a changed question.
What are the common mistakes?
Most errors with reliability and validity come from treating one check as the whole answer.
- Treating a high alpha as proof of validity. Items can agree with each other and all measure the wrong thing.
- Running test-retest on a concept that should change. Low correlation for weekly ad recall reflects real change, not a bad question.
- Rewording a tracker question mid-stream without a split test. You lose the ability to tell real movement from a wording effect.
- Coding open ends with one coder and no agreement check. Code counts then reflect one person’s reading, not the data.
- Assuming a validated scale stays valid in a new market. Check meaning and reliability again after translation.
What should you do next?
Pick the multi-item score your stakeholders watch most closely. Recalculate its alpha on the latest wave, then match it to an outcome you already hold, such as repurchase or renewal. Run that comparison as crosstabs with weighting and significance testing by score band, and check that the gap holds in each segment before you call the score a driver of anything.