Silent schema change splits the online and offline feature definitions
A fraud-detection team serves a gradient-boosted model behind a REST endpoint at 900 requests/second. Features are computed twice: an Apache Flink job writes them into Redis for online serving, and a nightly Spark job writes Parquet into the warehouse for training. On a Tuesday the payments team adds a nullable merchant_category_v2 column and starts emitting it at 14:00. The Flink job's Avro reader is pinned to the previous schema, so the new field is dropped silently; the online store keeps serving the old merchant_category, while offline training data written after 14:00 contains only the new field. Offline holdout AUC rises from 0.81 to 0.86. In production, approval rate for the affected merchant segment falls 6 points and chargebacks rise 0.4%. Nobody notices for nine days, because the drift monitor compares each feature's distribution against a 30-day rolling baseline, which absorbs the step change. The team has 40 minutes before a release freeze.
1. Explain the failure mechanism end to end, and explain specifically why the existing drift monitor could not have caught it.
Read the worked answer
Failure mechanism. There are two separate divergences stacked on top of each other.
- Semantic divergence. The model was trained on
merchant_category. After 14:00 the offline table no longer contains that column at all; it containsmerchant_category_v2. Anyone retraining from post-14:00 data produces a model whose expected input vector differs from what the serving path can supply. - Vector divergence at serve time. The online path still emits
merchant_category. So the deployed model is actually still receiving the encoding it was trained on, but the offline world used to judge it has moved. Offline holdout AUC of 0.86 is measuring the new field's signal, not the deployed model's behaviour.
The AUC rise is a trap, not a win: it says the new field carries more signal than the old one, which means offline and online are now maximally unalike. The real symptom is the 6-point approval drop and the 0.4% chargeback rise - the deployed model is scoring a population whose feature semantics no longer match the evaluation it is being credited with.
Why monitoring missed it. The monitor watched per-feature marginal distributions against a 90-day... against a 30-day rolling baseline. A rolling baseline is recomputed from recent data, so a nine-day-old step change is inside the baseline window and gets absorbed as the new normal. The monitor also never compared online and offline vectors for the same entity, never checked schema compatibility, and never watched a business KPI with a fixed reference. Every one of those gaps is independent; fixing only the baseline window would still have missed it.
2. Design a detection mechanism that would surface this class of problem within one hour of the schema change. Give concrete components, thresholds, and the failure mode each one covers.
Read the worked answer
Four layers, each covering a different blind spot:
1. Schema contract gate (covers producer-side breaks, ~0 latency). Put the Avro schemas in a registry with BACKWARD_TRANSITIVE compatibility enforced. The producer's CI fails if the new schema breaks compatibility; a consumer that declares a schema older than the writer either fails loudly or, if defaults are used, logs a SchemaDefaulted counter. Alert if that counter is non-zero for 5 minutes. This alone catches the Tuesday event at 14:00, not day nine.
2. Online/offline skew job (covers semantic divergence, ~1h). Sample 0.1% of online requests, persist the served feature vector plus entity key and timestamp. Hourly, join those samples back to the offline table for the same entity and event time and compute a per-feature mismatch rate. Alert at >0.5% mismatch on any feature, or a null-rate delta >2 sigma versus the prior seven days. This is the check that would have fired within the hour.
3. Score recomputation (covers encoder and preprocessing drift). Take the same logged online vector, run it through the offline scoring code, compare to the served score. They must match to 1e-6. Any divergence means the two paths disagree on preprocessing even when raw features agree.
4. Fixed-reference business guardrail (covers everything upstream). Approval rate per merchant segment compared against the same-weekday median of the previous eight weeks - a fixed reference, not rolling. Alert at 3 sigma. This gives a safety net that does not depend on anyone having anticipated the specific break.
3. The nine days of post-change data are already in the warehouse. What is your remediation plan for the data and the model, and what is the ordering constraint?
Read the worked answer
Do not retrain on the mixed table, and do not simply drop the nine days. Dropping them leaves a gap exactly where the population shifted, which is the region you most need to learn from. Retraining on them as-is produces a model whose feature semantics changed mid-window, which is worse than either extreme.
Pick one semantic and make it true for the entire history:
- Preferred: backfill
merchant_category_v2from the raw event log for the full training window. This only works if the raw events retained the field before the Flink sink dropped it - verify by sampling raw events at days -30 and -9. If the raw log has it, backfill, retrain on one consistent definition, and the nine days become usable. - Fallback: revert the offline pipeline to
merchant_categoryfor all history and keep serving the old field, then do a coordinated migration later: fix Flink, backfill, retrain, shadow, canary.
Ordering constraint: schema fix and backfill must land before retraining, and retraining must complete before the serving change. Reversing any pair recreates the skew.
Validation before promotion: replay the last 24 hours of logged online requests through the candidate offline pipeline and confirm the recomputed scores match the served scores within tolerance. Then shadow for 48 hours and compare approval rate by segment against the pre-incident baseline, not against the contaminated nine days.