Fraud model where labels arrive two months late
A payments company runs a fraud model that scores every transaction in real time; roughly 0.8% of transactions are fraudulent. Chargebacks, the ground-truth label, arrive 45-60 days after the transaction, so the team has no accuracy signal in real time. Volume is about 900 transactions per second. In the last six weeks the decline rate rose from 1.9% to 3.4% while reported fraud stayed flat, and the support team says more legitimate customers are being blocked. The model was last retrained eight months ago. Feature values come from a streaming pipeline that was migrated to a new vendor two months ago.
1. What would you monitor given that labels are unavailable for up to 60 days?
Read the worked answer
You cannot monitor what you cannot observe, so monitor the closest proxies and be explicit that they are proxies.
- Input feature drift against the training baseline, per feature. This is the fastest available signal and, given a streaming pipeline migration two months ago, the most likely culprit - a vendor change that subtly alters a feature's distribution will degrade a model without any error appearing anywhere.
- Prediction distribution - the score histogram and decline rate. These are already anomalous: decline rate nearly doubled with flat reported fraud. That gap is the incident, regardless of drift metrics.
- Delayed-label backfill evaluation. When chargebacks land, join them to the scores recorded at decision time and compute the metrics you actually care about. Automate this on a schedule; a 60-day lag means a manual quarterly review catches problems two quarters late.
- Business counters (manual review queue volume, customer complaints, appeal overturn rate) as early human-sourced indicators.
Accuracy is the metric you want and cannot have today. Say that plainly to stakeholders instead of presenting a dashboard that implies you have it.
2. How would you configure Vertex AI Model Monitoring for this model, and what would make it noisy?
Read the worked answer
Two monitors with different jobs:
- Training-serving skew on the feature set, using the training dataset as baseline. This directly tests the hypothesis that the streaming migration changed feature semantics. It is the higher-value monitor right now.
- Prediction drift on the output distribution, which catches degradation when inputs look normal but the score mix shifts.
Configuration discipline matters more than the feature. At 900 transactions/s, sampling everything is wasteful; sample enough to get stable distribution estimates per interval (a few thousand events is usually plenty) and set the monitoring interval to match how fast you would act - hourly during the incident, daily in steady state. Skew for high-cardinality categorical features (merchant ID, device ID) will fire constantly; exclude or bucket them rather than training the team to ignore alerts.
Route alerts through Cloud Monitoring alert policies on the log-based metrics with severity tiers, so a single drifting feature pages nobody at 3 a.m. but a fleet-wide shift reaches the on-call engineer.
3. Drift fires on three features and the decline rate is high. How do you decide between a pipeline bug, a model problem, and a genuine fraud shift?
Read the worked answer
Order the hypotheses by how well they explain the evidence, and design a cheap test for each.
Pipeline bug first. The streaming migration is two months old, the symptom is six weeks old, and drift fired. Compare feature statistics from the new pipeline against the old one for a sample period where both are still queryable. If a feature's distribution moved at the migration boundary rather than gradually, it is a pipeline defect and no amount of retraining fixes it. Also check for silent null/default substitution - a vendor that emits 0 instead of null for a missing amount will shred a tree model.
Genuine fraud shift second. Check whether the drift is concentrated in a subset (new merchant category, new geography, new device type) and whether declined transactions cluster the same way. Fraud rings do produce sharp distribution shifts.
Model staleness third. Eight months without retraining is long for fraud, but staleness produces gradual degradation, not a step change correlated with a migration date.
Use Explainable AI feature attributions on recent high-scoring transactions to see which features are driving declines - that usually names the culprit feature directly and shortens the argument.