Synthetic Fraud Data with a Privacy Leak Signal
PayFleet, a payments processor, sees fraud in 0.04% of 2.1 million monthly transactions, about 840 fraud cases. To improve a rare-event fraud model, the data science team trains a gradient-boosted model only on synthetic data generated by CTGAN: 50,000 synthetic fraud and 200,000 synthetic non-fraud records. On a synthetic holdout, recall is 0.91 at a 0.5% false-positive rate. On a real, time-split holdout, recall is 0.64 at the same false-positive rate, and precision falls from 0.31 to 0.08. A privacy red team reports membership inference AUC of 0.68 for a small merchant segment represented by fewer than 20 real frauds. The team wants to deploy for real-time scoring next month. The model risk committee must decide whether to approve, conditionally approve, or reject.
1. Diagnose the performance gap and the privacy finding.
Read the worked answer
The synthetic holdout overstates performance because the generator has learned the synthetic distribution, not the real fraud process. A drop from 0.91 to 0.64 recall and precision from 0.31 to 0.08 at the same false-positive rate is material; the model is not ready.
The privacy finding is also serious: membership inference AUC 0.68 on a small merchant segment with fewer than 20 real frauds suggests memorization or overfitting to rare records. Small segments are exactly where synthetic generators can leak. The team's plan to deploy next month is not supportable. Root causes include training only on synthetic data, class imbalance, possible temporal leakage, and inadequate real-world validation. Treat synthetic results as a development signal, not evidence of production performance.
2. Redesign the validation and data strategy.
Read the worked answer
Use a real, time-split holdout as the primary test set and never tune on it. Compare synthetic-only, real-only, and mixed training. Stratify metrics by merchant size, fraud type, and region; report recall at fixed false-positive rate, precision-recall AUC, and calibration. Add temporal validation across at least three months.
For privacy, run membership inference and attribute inference on small segments, apply differential privacy or k-anonymity, cap synthetic duplication, and remove near-copies. Consider synthetic data only for augmentation, not as the sole fraud class, and use real fraud with class weighting or cost-sensitive learning. Set acceptance criteria: real-holdout recall at 0.5% false-positive rate within 0.05 of the champion, precision not worse than the champion by more than 10%, and privacy attack AUC below 0.55.
3. What decision should the committee make, and with what conditions?
Read the worked answer
Conditionally reject deployment. Approve a 60-90 day remediation: rebuild with real data plus limited synthetic augmentation, validate on a real holdout, and complete privacy testing.
Conditions should include:
- no synthetic-only training for production;
- segment-level performance review;
- privacy sign-off;
- a model card documenting synthetic data use;
- monitoring for drift and false positives;
- human review for high-risk merchant segments.
If remediation fails, keep the existing champion or rules-based model. The committee should also require a data provenance log and legal review of synthetic data rights.