99.7% Accurate and Useless
A payments provider trains a fraud detector on 2.1 million transactions, 0.31% of which are labelled fraudulent. Test-set accuracy is 99.7%. The head of risk calls it 'a clear win over the rules engine, which catches only 40% of fraud'. Investigation shows: 12% of test rows are duplicated transactions from a batch replay; fraud labels come from chargebacks, which arrive on average 45 days after the transaction; the training extract was cut 30 days after each transaction, so recent frauds are labelled legitimate; and the merchant category 'crypto exchange' has 340 training rows with a 96% fraud rate. Go-live is in five days.
1. Explain why 99.7% accuracy is uninformative, and state the metrics you would report instead.
Read the worked answer
Why the figure is meaningless
The base rate is 0.31%. A classifier that outputs 'legitimate' for every transaction scores 99.69%. The model's 99.7% is at, or below, the trivial baseline - you cannot tell from it whether the model has learned anything at all. Accuracy is dominated by the majority class, and the only errors that matter here are in the 0.31%.
What to report instead
- Confusion matrix at the actual operating threshold - not at a default 0.5.
- Recall (detection rate) at a fixed false-positive budget. Fraud operations can review N alerts per day. That capacity is the deployment constraint. "Catches 40% of fraud" from the rules engine is a recall claim; it is only comparable if you state the false-positive rate at which it achieves 40%.
- Precision and false-positive rate, so the alert burden is visible.
- PR-AUC (precision-recall) rather than ROC-AUC. Under 0.3% prevalence, ROC curves flatter every model.
- Cost-weighted expected loss, using the chargeback amount distribution - a missed 2,000-unit fraud is not the same as a missed 5-unit one.
- Calibration, if scores are to be thresholded or compared with the rules engine.
- The same metrics computed on a clean, deduplicated, matured test set - otherwise all of the above inherit the biases described next.
Never present a single headline number to a risk committee; the base rate guarantees it will be misread.
2. Which input-data quality problems are present, and what input-data test addresses each?
Read the worked answer
A. Uniqueness / duplicates (12% of test rows). Test: exact-hash duplicate detection on transaction ID, plus near-duplicate detection on (card, amount, merchant, timestamp bucket). Report duplicate rate per split and per class. Deduplicate before splitting, never after - duplicates spanning a split boundary are a leakage path, and duplicated test rows silently re-weight the metric.
B. Label accuracy and timeliness. Chargeback lag averages 45 days, but the extract was cut at 30 days. Every fraud in the last 15+ days of the window is labelled legitimate - a systematic, non-random false-negative population concentrated at the end of the extract. Test: a label-maturation analysis - the distribution of days-to-chargeback; assert the labelling window covers at least the 95th percentile of the lag. Evaluate only on matured rows, and quantify the censoring bias explicitly rather than assuming it is small.
C. Representativeness and coverage. 'Crypto exchange' has 340 rows at a 96% fraud rate: a tiny, extreme segment. Two risks - the model may generalise badly to it, and it may learn 'crypto => fraud' and over-flag the legitimate 4%. Test: per-segment row counts and class balance compared with the production traffic mix (population stability index, coverage report); flag any segment below a minimum support level. Check whether the training population matches the live population segment by segment, not in aggregate.
D. Completeness, consistency, validity. Missing merchant category, amount currencies mixed, timestamps in local time versus UTC, negative amounts. Test with domain/range assertions, cross-source referential checks, and null-rate comparisons between training and live traffic.
E. Oracle quality. If the labels themselves derive from the rules engine's outcomes, the model is being trained to imitate the thing it is supposed to beat, and 'catches more fraud than the rules' may be unfalsifiable. A small stratified human audit estimates the label error rate.
3. You have five days. What is your recommendation, and what is the minimum evidence you would require before any production decision?
Read the worked answer
Recommendation: do not authorise a decision-making go-live on the current evidence.
This is not a disagreement about model quality. It is an evidence problem: the reported 0.94-equivalent metric is invalid because the test set contains duplicated rows and the labels are systematically immature. Any accept/reject decision made on it is unfounded, and the risk committee should be told that in those terms.
Minimum evidence before a decision
- Test set rebuilt: deduplicated before splitting, restricted to matured labels, split temporally.
- PR-AUC and recall reported at the actual review-capacity false-positive budget.
- Per-segment metrics for high-risk categories, with sample sizes and intervals; segments with insufficient support explicitly marked as unvalidated.
- A label-audit estimate (even a few hundred stratified rows) bounding label error.
- Reproducibility check: same input, same score, across the offline and production pipelines.
If the business cannot wait
Ship in shadow mode: score live transactions, enforce nothing, for four to six weeks. This generates exactly the missing evidence, on the real distribution, while bounding the risk to zero. Run the rules engine in parallel and compare decisions - disagreements are the highest-value test cases you will ever get.
In parallel: backfill chargebacks so the training labels can be corrected, extend the labelling window to at least the 95th percentile of the chargeback lag, and retrain. Set the production threshold from review capacity, not from 0.5.
What to say to the head of risk: 'the 40% figure is a recall number without a false-positive number attached, so the comparison is not yet a comparison. Shadow mode will give us both within a month.'