Sandbox Promotion Broke Production Audiences
An organization uses a sandbox to build and test changes, then promotes them to production. A practitioner updated a core schema in the sandbox - renaming an attribute from order_total to order_value and changing its type from string to number - then promoted it. After promotion, three production audiences that referenced order_total stopped updating, two destinations began rejecting records, and the data steward reports that the change was made without review. There is no documented change-management process. The schema is used by 11 datasets and 40+ audiences.
1. Explain the failure chain from the schema change to the broken audiences and rejected destination records.
Read the worked answer
The failure chain is a breaking schema change propagated without dependency analysis:
- Rename breaks references: audiences, derived attributes, and computed fields that reference
order_totalnow point at a field that no longer exists. They don't error loudly - they silently stop qualifying, which is why audiences 'stopped updating' rather than throwing visible failures. - Type change breaks ingestion/mapping: changing string → number means incoming data that isn't cleanly numeric fails validation, and existing mappings/transformations built for a string now misbehave. Destination field mappings expecting the old name/type reject records.
- Promotion without review: with no change-management gate, the breaking change reached production with 11 datasets and 40+ audiences depending on it. Sandbox success only proves it works in the sandbox, not that downstream production dependencies are safe.
Root cause is process, not the schema edit. The edit itself is reasonable; the absence of impact analysis and review is what caused the outage.
2. What change-management and hygiene practices would you institute to prevent recurrence? Be specific about schema evolution rules.
Read the worked answer
Schema evolution rules (technical):
- Additive by default: add new fields rather than renaming. If renaming, add the new field, dual-write, migrate consumers, then deprecate the old field - never rename in place on a shared schema.
- Type changes are breaking: treat any type change as a migration, not an edit. Validate source data quality first.
- Deprecation window: keep the old field populated until all audiences/destinations are migrated.
Change management (process):
- Impact analysis before promotion: enumerate dependent datasets, audiences, derived attributes, and destination mappings. With 11 datasets/40+ audiences, this must be scripted or checklist-driven.
- Review gate: schema changes require data steward/owner approval - the steward's complaint shows this was missing.
- Promotion checklist: verify dependencies in sandbox, stage a canary, monitor audience counts and destination acceptance rates post-promotion.
- Monitoring/alerting: alert when an audience's membership drops to zero or a destination's rejection rate spikes - silent breakage should page someone.
- Documentation: maintain a schema/attribute registry and a change log.
Judgement: the exam wants additive-first schema evolution + review gates + post-change monitoring, not just 'be careful in sandbox.'