AUC 0.85 and Ship It

What AUC >= 0.85 actually measures

AUC is a dataset-level ranking statistic. It is one narrow expression of functional correctness and nothing else. Specifically it is:

• threshold-free, so it says nothing about the operating point the credit union will actually use;
• pooled, so a strong majority slice can mask a weak slice;
• computed on a fixed historical sample, so it says nothing about the live distribution;
• insensitive to the relative cost of a false approval versus a false rejection.

Three characteristics to test first

1. Reliability (robustness / maturity). The score changes when the file has not changed. That is a reproducibility failure under nominally identical input, and it is observable today, cheaply. It outranks everything else because it undermines every other measurement you would make.
2. Functional suitability (functional correctness, evaluated per slice). The regulator question is really 'is correctness equal across sub-populations?'. Sparse records for self-employed and over-65 applicants mean per-slice correctness is likely worse there, and a pooled AUC cannot see it. Add a circularity caveat: the labels are past human decisions, not ground truth, so 'correctness' inherits whatever bias the old process had.
3. Maintainability (analysability / testability). If you cannot isolate why a score changed, you can neither debug the volatility nor answer the regulator. Testability is a precondition for the first two tests, not an afterthought.

Flagged but deprioritised on a two-week budget: safety (adverse outcomes for vulnerable applicants), security (applicant PII in features and logs), performance efficiency (triage latency).

Framing note for the exam answer: 'fairness' and 'transparency' are not first-class characteristics in the standards' quality models. Surface them as unequal functional correctness and weak analysability, otherwise you cannot write a pass/fail criterion for them.

Recommendation: replace the single number with an acceptance profile: per-slice correctness with confidence intervals, a reproducibility tolerance, and explanation availability at the decision point.

Diagnosis: it is primarily a requirements defect, and the test approach has been hiding it.

A recency feature is not inherently wrong - the business may genuinely want 'days since last enquiry' to matter. What is wrong is that no requirement states the expected behaviour for an unchanged file, so acceptance is undefined. The symptom is a reliability (robustness) failure; the root cause is a missing, unmeasurable requirement; the contributing cause is a test approach that evaluates a static snapshot once and therefore can never see time-varying output.

The test that exposes it: paired-execution test

1. Build a sample of N real and N synthetic applicant records (include the sparse slices: over-65, self-employed).
2. Score every record twice: once with the system clock pinned to reference date T, once pinned to T + 7 days.
3. Everything else frozen: model version, feature pipeline version, upstream data snapshot, random seed, and the clock itself injected as a parameter rather than read from the wall clock.
4. Assert the score delta is within a stated tolerance (e.g. <= 0.5 percentage points of predicted default probability).
5. Log the full feature vector for both executions and diff them, so you can attribute the delta to the recency feature rather than guess.

Making it reproducible

• Time must be an injectable parameter, not now(). If it is not, that is itself a testability finding.
• Pin and version: data snapshot, feature pipeline, model artefact, seed.
• Persist the feature vector with every scored decision; without it, no post-hoc investigation is possible.

Closing the loop: convert the observed behaviour into a requirement - either 'score variation for an unchanged file must be <= X' or 'remove the feature'. The same harness then becomes the regression test after every retrain, which you need anyway.

The honest answer is 'this data cannot support that judgement', and you should say so with numbers.

• With ~900 records, a 95% confidence interval on accuracy is roughly +/-3 percentage points even at 90% accuracy, and much wider for a low-prevalence outcome where the positive count is small. A point estimate is not evidence.
• If the slice has, say, 60 defaults, the interval on recall will span tens of points. You cannot distinguish 'as good as the pooled model' from 'materially worse'.

Decision rule to agree in advance

• Report the point estimate and the interval, per slice.
• Accept only if the lower bound of the interval exceeds (pooled performance minus a pre-agreed margin). If the interval is too wide to decide, the outcome is 'insufficient evidence', not 'pass'.
• Weight by error cost, not by count: an incorrectly rejected applicant is a different loss from an incorrectly approved one.

What to do about it

• Stratified sampling to enrich the slice for the next evaluation round; pool with adjacent slices only if you can justify that the risk profile is comparable.
• Remember the labels are past human decisions. If the old process under-approved this group, the model is being trained to reproduce that, and 'correctness' against those labels is circular. Consider a human-adjudicated audit sample of decisions rather than historical labels.
• If the evidence is insufficient at go-live, the defensible option is limited rollout with human review of every decision in the slice, and monitoring, not silent automation.