A fraud ranking improvement with a worse review queue

A fraud-review team evaluates a model on a properly dated holdout sample. At one threshold the confusion matrix has thirty true positives, ten false positives, twenty false negatives, and forty true negatives. A challenger has a higher ROC AUC, yet its available review queue produces less expected recovery after investigation costs. The business can inspect only a limited number of files, and potential recoveries vary substantially. A separate calibration check shows that a group assigned event probability 0.20 has forty positives among one hundred cases. The manager wants to announce the challenger as universally superior from AUC alone. The analyst must separate ranking, threshold metrics, probability calibration, constrained decision value, and uncertainty before recommending a change.