The Duplicate Customer Problem Nobody Owns
A regional retailer merged its online store and three brick-and-mortar POS systems two years ago. Customer records now live in four systems: the e-commerce platform (1.2M records), a loyalty app (900K), POS (600K), and a legacy CRM (400K) that is still used by the call center. There is no shared customer identifier. The analytics team estimates 30-40% duplication but has never measured it. Marketing wants to run a win-back campaign next month and needs a single customer view. The data engineering lead says a full master data management program will take nine months. The CMO has approved budget for the campaign but not for MDM. A junior analyst proposes using email address as the match key and deduplicating in a spreadsheet export.
1. The CMO asks you whether the junior analyst's approach is acceptable for the upcoming campaign. What do you advise, and how do you justify it?
Read the worked answer
The spreadsheet approach is not acceptable as-is, but the reasoning matters more than the verdict.
Why it fails:
- Email is a weak match key. Shared household emails, opt-out blanks, and POS records that never captured email mean the match will systematically miss in-store customers - exactly the ones a win-back campaign targets.
- No measurement of the 30-40% estimate means no way to know if the result is 15% or 55% duplicated. A campaign sent to duplicates produces multiple contacts to the same person, which is both a cost problem and a consent/compliance problem under contact-preference rules.
- A spreadsheet export is a one-time artifact with no lineage, no audit trail, and no reproducibility. If the campaign is challenged or repeated, there is nothing to defend.
What I would advise instead:
- Run a bounded measurement first - a 50K-record stratified sample matched on email, phone, and name+postcode to produce an actual duplication rate and per-key match quality. This takes days, not months, and turns an estimate into evidence.
- Use a tiered match: deterministic on normalized email and phone, probabilistic on name+postcode+last-activity where deterministic keys are absent.
- Suppress duplicates at send time rather than trying to perfect the golden record. The campaign needs a contact list, not a customer master.
- Keep the output in a governed table with match scores and lineage, not a spreadsheet.
Justification to the CMO: the nine-month MDM program solves a different problem (ongoing single view). The campaign needs a defensible, time-boxed match for one use case. Doing the measurement also de-risks the MDM business case by quantifying the actual problem.
2. The data engineering lead argues that doing anything short of full MDM creates technical debt. How do you respond, and what conditions would make you agree with them?
Read the worked answer
Both positions contain a real risk; the question is which risk is being traded for which.
Where the engineering lead is right:
- A one-off match built for a campaign, if left running, becomes a shadow customer master that nobody governs. That is genuine debt.
- If the organization will need a single customer view for regulatory reporting, personalization, or service within 12 months, building a throwaway now means doing the work twice.
Where the argument overreaches:
- "Short of full MDM" is a false binary. A measurement exercise and a governed match table are not an MDM program, and they produce reusable assets: normalized keys, match rules, a duplication baseline. These feed the MDM program rather than compete with it.
- Nine months with no interim capability means the business either waits or goes around data management entirely - the worst outcome for governance.
Conditions under which I would agree with the lead:
- If the duplication rate measured in the sample is low (say under 10%), the campaign can proceed with simple suppression and the full program should own the real fix.
- If there is a hard regulatory or contractual requirement for a certified single customer view within the campaign window, a stopgap creates audit exposure.
- If no owner can be named to retire the interim solution once MDM lands, the debt will persist, and the lead's objection wins.
My recommendation: proceed with the bounded match, but write the retirement condition and owner into the plan before starting. Interim solutions are only debt when nobody has agreed to end them.