Your brand tracker runs in eight markets through three agencies. Each delivers a clean file, and none of them line up. One codes brands 1 to 12, another 101 to 130. One asks satisfaction on a 5-point scale, another on a 10-point scale. Data harmonization is the work that turns those files into one view you can trust. This guide covers what to harmonize, a step-by-step process, and where data harmonization reaches its limits.
Key takeaways
- Data harmonization makes data from different sources comparable by aligning definitions, codes, scales, time periods and units, so you can analyze it together.
- It is not the same as data integration. Integration moves data into one place; harmonization makes it mean the same thing once it is there.
- The core tool is a master data dictionary, with a documented mapping rule from every source to every harmonized variable.
- Rescaling to a common 0 to 100 score works well for means. Top-2-box comparisons across different scales need clear caveats.
- Harmonization cannot fix differences in sampling, question order or interview mode. Flag those data as not comparable instead of forcing them together.
What is data harmonization?
Data harmonization takes data that measures similar things in different ways and makes it comparable. In research, that means aligning five things: what each variable means, how its answers are coded, which scale it uses, which time period it covers, and which unit or base it is reported on.
When you harmonize data, you do not change what respondents said. You build a shared structure and map each source into it with a written rule. A rating of 4 on a 5-point scale stays a 4 in the source file, and becomes 75 on a 0 to 100 scale in the harmonized file. The goal: a harmonized number means the same thing whichever source it came from, and where it cannot, the data says so.
How is data harmonization different from data integration and standardization?
These terms get used interchangeably. They are related steps, and most teams need all three.
| Practice | What it does | Example | What it does not do |
|---|---|---|---|
| Data integration | Moves data from many sources into one place | Loading tracker files, CRM extracts and CX survey results into one store | Make the variables comparable |
| Data standardization | Applies one format or convention to values | Writing every date as YYYY-MM-DD; using one list of country codes | Resolve differences in meaning, such as two satisfaction scales |
| Data harmonization | Makes data from different sources measure the same thing in the same way | Mapping a 5-point and a 10-point scale to one 0 to 100 score | Fix differences in how the data was collected |
Integration answers “where is the data?” Harmonization answers “can I compare it?” For a fuller look at the first question, see our guide to customer data integration.
Standardization is usually one part of data harmonization. Using the same country codes in every file is standardization. Deciding how one agency’s “Benelux” maps to the Belgium and Netherlands figures other sources report is harmonization, because it changes the definition, not only the format. Together, the three make a single source of truth for research more than a shared folder.
Why do insights teams need to harmonize data?
Insights teams rarely design every data source. They inherit them, each with its own conventions.
- Multi-market trackers. A global brand tracker is often run by different agencies by region, or set up market by market over years. Brand codes, age bands and even question wording drift apart.
- Syndicated data. Retail scan data, consumer panels and industry studies come with the provider’s own category definitions, product hierarchies and reporting periods. Our guide to syndicated data explains how these sources are built.
- CRM and CX surveys. These use their own scales, segments and customer IDs, usually set by another team.
- Web analytics. These report on days and sessions, not survey waves and sampled respondents.
Without harmonization, every cross-source question becomes a one-off project, and two analysts making slightly different mapping choices give the same question two answers. In multi-market trackers, the first problem we usually see is not a missing dataset. It is three versions of the same metric that disagree.
What gets harmonized?
These elements come up in nearly every research program.
- Variable names. Q12, S4_SAT and overall_csat may all be overall satisfaction. Give each concept one harmonized name.
- Question wording and scales. A 5-point and a 10-point satisfaction scale need a common score, and the wording needs checking to confirm both ask the same thing.
- Brand and product lists. Each source has its own codes, spellings, local brand names and sub-brand groupings.
- Market and region codes. Country names, codes and regional groupings, such as which countries count as “Europe”, differ by source.
- Demographic bands. One source uses 18 to 24, 25 to 34 and so on; another uses 18 to 34 and 35 to 54.
- Time periods. Survey waves with fieldwork dates, calendar months, quarters, and the four-week periods some retail and panel data use.
- Weights and bases. Each source is weighted to its own population, and “base” can mean all respondents, category buyers or people aware of the brand.
- Open-end code frames. Coded verbatims use different theme lists, at different levels of detail.
How do you harmonize data, step by step?
Treat data harmonization as a repeatable process, not a cleanup job. The steps are the same for two sources or twenty.
- Inventory your sources. List every dataset with its owner, questionnaire, time coverage, sample and weighting. You cannot map what you have not described.
- Build a master data dictionary. For each harmonized variable, write the name, a one-sentence definition, the allowed values, the scale and the base. Start with the metrics people actually compare.
- Map each source to the dictionary. Record which source field feeds each variable and the rule that transforms it, as in the mapping table below.
- Rescale and recode. Apply the rules: convert scales, collapse bands, translate brand and market codes, assign waves to periods.
- Validate. Check that harmonized figures reproduce each source’s own reported results, that every code has a home, and that no values fall outside the allowed range.
- Document. Write down every decision and exception, including data you judged not comparable.
- Maintain it each wave. New waves bring new brands, questions and markets. Update the mapping before the data is used.
What does a mapping table look like?
Here is an illustrative mapping for three sources: a tracker in Market 1, the same tracker run by a different agency in Market 2, and a CX survey.
| Harmonized variable | Definition | Tracker, Market 1 | Tracker, Market 2 | CX survey | Mapping rule |
|---|---|---|---|---|---|
| SAT_100 | Overall satisfaction, 0 to 100 | Q12, 1 to 5 | S4, 1 to 10 | csat, 1 to 5 | (x − min) / (max − min) × 100 |
| BRAND_ID | Master brand ID | Codes 1 to 12 | Codes 101 to 130 | Brand name as text | Lookup in master brand list |
| AGE_BAND | 18 to 34, 35 to 54, 55+ | 18 to 24, 25 to 34, 35 to 44, 45 to 54, 55+ | 18 to 34, 35 to 54, 55+ | Exact age | Collapse bands; band exact age |
| PERIOD | Calendar quarter | Wave 3, fieldwork in March | Wave 3, fieldwork in May | Response date | Assign by fieldwork midpoint or date |
| BASE | All respondents aged 18+ | All respondents | Category buyers only | Customers only | Flag Market 2 and CX as different bases |
Notice the last row. Not every difference can be mapped away, and the mapping table is where you say so.
How do you rescale a 5-point and a 10-point scale to compare them?
This is the most common data harmonization question in survey work. The figures below are illustrative. Market 1 asks overall satisfaction on a 1 to 5 scale, and Market 2 on a 1 to 10 scale:
| Market 1 (1 to 5) | Respondents | Market 2 (1 to 10) | Respondents |
|---|---|---|---|
| 1 | 20 | 1 to 4 | 55 |
| 2 | 40 | 5 | 40 |
| 3 | 100 | 6 | 55 |
| 4 | 160 | 7 | 100 |
| 5 | 80 | 8 | 120 |
| 9 | 80 | ||
| 10 | 50 | ||
| Total | 400 | Total | 500 |
In Market 2, points 1 to 4 had 10, 10, 15 and 20 respondents, grouped here to save space.
Step 1: calculate each mean on its own scale
- Market 1: (1 × 20 + 2 × 40 + 3 × 100 + 4 × 160 + 5 × 80) / 400 = (20 + 80 + 300 + 640 + 400) / 400 = 1,440 / 400 = 3.60
- Market 2: (1 × 10 + 2 × 10 + 3 × 15 + 4 × 20 + 5 × 40 + 6 × 55 + 7 × 100 + 8 × 120 + 9 × 80 + 10 × 50) / 500 = 3,565 / 500 = 7.13
You cannot compare 3.60 with 7.13 directly. They sit on different scales.
Step 2: rescale to 0 to 100
Use a linear rescale: score = (x − min) / (max − min) × 100. The lowest point becomes 0 and the highest becomes 100.
- Market 1: (3.60 − 1) / (5 − 1) × 100 = 2.60 / 4 × 100 = 65.0
- Market 2: (7.13 − 1) / (10 − 1) × 100 = 6.13 / 9 × 100 = 68.1
On a common 0 to 100 scale, the two markets are about 3 points apart. Because the rescale is linear, you get the same result whether you rescale each answer and then average, or average and then rescale.
Step 3: compare top-2-box, and see the problem
Top-2-box is the share of respondents choosing the two highest points.
- Market 1, points 4 and 5: (160 + 80) / 400 = 240 / 400 = 60%
- Market 2, points 9 and 10: (80 + 50) / 500 = 130 / 500 = 26%
The means say the markets are close. Top-2-box says Market 1 leads by 34 points. That gap is an artifact: the top two points are 40% of a 5-point scale but only 20% of a 10-point scale.
Match the share of the scale instead, using the top four points of the 10-point scale (7 to 10), and you get (100 + 120 + 80 + 50) / 500 = 350 / 500 = 70%. Now Market 2 leads by 10 points. Each cut point tells a different story, and none is the true one.
The practical rule: across scales, report the rescaled mean as the headline and treat any top-box figure as indicative, with the cut point stated. Track top-box within each source over time, where the scale is constant. And remember that linear rescaling assumes people use both scales the same way, which is a convenient assumption, not a fact.
How do you harmonize brand lists and markets?
In our experience, brand lists cause more harmonization work than any other variable, because they change every wave and differ by market.
Start with a master brand list. Give each brand one permanent ID, a global name, a parent company, and the local names, spellings and codes that map to it. Decide how sub-brands roll up: if one market asks about a brand’s light variant separately and another does not, the harmonized brand-level figure must use the same definition in both. When a brand is added to one market’s list mid-year, record the wave it entered, so a trend line does not imply the brand was measured earlier.
Markets need the same discipline. Use one list of country codes, define every region once (which countries make up “Europe”, and whether that changes over time), and decide how to treat sources that report combined markets. If one source only reports “Nordics”, you either harmonize every source to that level or exclude it from country comparisons.
For trackers specifically, including how to handle a change of agency or questionnaire midstream, see our guide to harmonizing brand trackers.
What can data harmonization not fix?
Data harmonization aligns how data is described. It cannot change how it was collected, and three differences survive any amount of recoding.
- Sampling. A nationally representative sample, category buyers and a customer list are different populations. Rescaling does not make them the same people.
- Question order and context. A satisfaction question asked after a list of product problems tends to get different answers from the same question asked first. Matching the wording does not match the context.
- Mode. Answers given to an interviewer often differ from answers typed into an online survey, particularly on sensitive or socially desirable questions. A change of mode between waves can look like a change in the market.
Translation is a fourth risk in multi-country work. A scale label that reads as neutral in one language can read as mildly positive in another.
The answer is to flag, not force. Give each variable and source a comparability flag in the dictionary: comparable, comparable with caveat, or not comparable. Carry the flag into reports, break trend lines where a method changed, and footnote charts that combine sources. A visible break beats a smooth line that hides a change of method.
Who owns harmonized data, and how do you govern it?
Harmonized data needs an owner, usually someone in insights or research operations who understands both the methods and the data. Without one, the dictionary drifts and each analyst ends up with a private version. Three practices help:
- A versioned dictionary. Number each release, such as v1.0 and v1.1, and record which version every report used, so you can explain why a figure changed.
- A change log. Note every new brand, recoded band, added market and changed rule, with the date, the wave it applies from and who approved it.
- A review each wave. Before new data is released, check the mapping against the new questionnaire and files. This is the step most teams skip, and it is where most errors come in.
What are the common mistakes in data harmonization?
- Harmonizing to the most detailed version. If one source has five age bands and another has three, the harmonized variable can only have three. Forcing detail you do not have means inventing it.
- Comparing top-2-box across different scales. As the worked example showed, the cut point can reverse the conclusion. Use rescaled means and state the caveat.
- Stacking weighted files without checking the weights. Each source’s weights target its own population. Combining them without a stated rule for the combined total gives a figure nobody designed.
- Skipping validation against the source. If the harmonized figure for a market does not reproduce that source’s own result, the mapping is wrong somewhere. Check before anyone uses it.
- Leaving decisions undocumented. A mapping only one person understands stops working when that person moves on.
How do you put data harmonization into practice?
Start with the five or six metrics your stakeholders compare most. Build the dictionary for those, map your sources, validate, and add more as you go. The process matters more than the tooling, but tooling decides how easily you repeat it each wave. Halo Models lets you import survey and tracker data and harmonize it in one governed layer, so Pulse and Reports answer from all of it. From there, banner tables with weighting and significance testing across global trackers can run on the harmonized variables instead of on each market’s file.
Harmonized data also makes past work reusable. When every study uses the same brand IDs, markets and periods, it becomes easier to connect findings across studies and waves, so a new question starts from what the team already knows.
For your next step, pick one metric that two of your sources both report, such as overall satisfaction, and write its dictionary entry: definition, scale, base and the mapping rule for each source. If the team cannot agree on the definition, you have found the first thing to harmonize.