Four nights of successful failures

The root cause is not the schema change; schema changes happen constantly. The root cause is a handler that converts failure into success.

Code

• Remove the catch-all. Catch specific, recoverable exceptions only, log with full context, and let everything else propagate so the process exits non-zero and the job is marked failed.
• Add a schema contract check as the first step: assert that every expected column is present with the expected type, and fail before doing any work. A nullable column added upstream should trip this immediately.
• Make writes idempotent - INSERT OVERWRITE the target dt partition or MERGE on the key - so reruns are safe and backfill does not duplicate rows.

Data-quality gate After the write, verify row count within an expected band, freshness (max dt equals today), null rate per critical feature, and a distribution check on a few stable columns. If any check fails, exit non-zero. This is the difference between "the job ran" and "the data arrived".

Alerting and SLA

• Enable job failure notifications and add an independent watchdog that checks the target table by 05:00 - the job should not be the only thing reporting on the job.
• Alert on the 06:00 deadline with enough lead time for a human to act.
• Add retry with backoff for transient failures, but never retry an assertion failure blindly.

Recovery Backfill the four missing partitions by running the fixed job for each date against the correct source snapshot. Because writes are idempotent, a partial or repeated backfill is harmless. Reconcile counts against the source and tell dashboard consumers when the gap was closed, so nobody silently consumes a four-day hole.

Interpret the signal first. PSI of 0.35 is conventionally read as a substantial shift (roughly: below 0.1 stable, 0.1–0.25 moderate, above 0.25 significant), but PSI on one feature tells you the inputs moved, not that the model is wrong. A pricing change is a known, explainable cause; the question is whether the model's learned relationship to txn_amount still holds under the new price regime.

Decide with evidence, not the PSI value alone

1. Check the output side: is the distribution of predicted scores also shifted, and by how much? An input shift that produces a stable score distribution is usually benign.
2. Check realised performance on the labels that have arrived since the change. If precision or error at the operating point is unchanged, retraining can wait.
3. If the score distribution has shifted or realised performance has degraded, retrain - but on data that includes post-change behaviour, otherwise the new model simply relearns the old regime.
4. If the relationship appears to be a genuine concept change (the same price now implies different demand), a monotonic adjustment or a reweighted retrain may be faster than a full rebuild.

Broader monitoring

• Feature-level: PSI or KS on all important features, plus null rate and range checks - not just txn_amount.
• Prediction-level: mean score, score distribution, and the positive rate at the operating threshold. These move first and need no labels.
• Outcome-level: realised error or precision once labels arrive, accepting label delay.
• Operational: job success, freshness, runtime, and row counts.

Policy Define thresholds and owners in advance. Auto-retrain on drift is a trap; drift should open a review, and retraining should require passing an evaluation gate on out-of-time data before promotion.

What the current setup can honestly answer: very little. Score rows carry no model identifier, so there is no link from a score to a build. Builds are kept 30 days, so a build from March is already gone by the time a later audit asks. Request logs are kept 7 days and, even if they survived, do not necessarily record the version. And there is no record of which data partition the run consumed. The honest answer is "we cannot demonstrate it", and saying so is better than constructing a plausible story from memory.

What must change

1. Stamp the output. Every score row carries model_version, run_id (the MLflow run), scored_at, and the source snapshot or partition range. This one change answers most lineage questions permanently.
2. Immutable, retained builds. Keep model builds and their runtime image digests for the period the regulation requires, not the platform default. A rebuild produces a new build ID; never overwrite.
3. Experiment metadata. MLflow run records with git SHA, image digest, hyperparameters, and data identity, retained on the same schedule.
4. Table and column lineage. Where Atlas lineage is available, register the pipeline so the transformation path from source to score is recorded automatically rather than reconstructed.
5. Access evidence. Keep the authorisation audit trail for the period required, so you can show who could read the data, not just who did.
6. Retention as policy. Write the retention rules down with the reason for each, and verify enforcement; a retention setting nobody has justified is a finding waiting to happen.

Interim action for this request: reconstruct what you can from job run history, git history, and any surviving build records, label the reconstruction as partial, and state what is not provable. Then close the gap before the next audit.