Reranker Rollout with Stratified Evaluation
A retrieval-augmented search platform is evaluating reranker v2 before rollout. The offline test set has 2,100 queries stratified by document type: contracts n=400, invoices n=350, support tickets n=600, emails n=450, and technical specs n=300. Overall answer accuracy rises from 74.2% to 78.9%. By stratum: contracts 81.2% to 76.4%, invoices 77.0% to 73.8%, tickets 69.1% to 75.3%, emails 72.5% to 78.0%, and specs 70.2% to 79.1%. Retrieval recall@10 moves from 88.1% to 89.0% overall, but contracts recall falls from 92% to 84%. The reranker was trained mostly on tickets, emails, and specs; contracts contribute only 4% of training pairs. P95 latency increases by 180 ms, which is within the 250 ms budget. Product wants a go/no-go decision this week.
1. The overall metric improved by 4.7 points. Why is that insufficient, and how would you analyze the stratified results?
Read the worked answer
Aggregate improvement can hide harm to high-value segments. Contracts and invoices are likely high-stakes, and the contracts recall drop from 92% to 84% is a leading indicator that the reranker is pushing relevant clauses out of the top 10. With n=400 and a proportion near 0.8, the standard error is about 2 percentage points; a 4.8-point drop is roughly 2.4 standard errors and likely significant. The invoice drop of 3.2 points with n=350 has a standard error near 2.2 points, so it is suggestive but not conclusive. Five subgroup comparisons also inflate the chance of a false positive, so predefine the primary subgroups.
Analyze paired bootstrap confidence intervals by query rather than independent intervals. Decompose retrieval from generation: falling contracts recall means ranking, not the reader, is the problem. Check whether the test-set stratum weights match production traffic; if contracts are 25% of real traffic, the aggregate is misleading. Inspect failure cases for long clauses, tables, and numeric matches. Recommend no global rollout until contracts and invoices are non-inferior, and evaluate doc-type routing as a guardrail.
2. Diagnose why contracts and invoices regress and propose a targeted fix. Include guardrails.
Read the worked answer
The training distribution is mismatched: contracts are only 4% of pairs, while the reranker is optimized on ticket, email, and spec phrasing. Contracts contain long legal clauses, cross-references, and defined terms; invoices contain numeric tables and identifiers. A semantic reranker can downrank exact clause or number matches that look lexically different but are decisive. The contracts recall drop confirms first-stage ranking failure, not just answer generation.
Fix the data before the model. Add stratified hard negatives from contracts and invoices, oversample them to at least 20-25% of training pairs, and label 1,500 new pairs. Add features for exact numeric match, section headers, and defined-term overlap. Calibrate scores per document type and evaluate recall@10 and nDCG by stratum. Guardrail: route contracts and invoices to v1 until v2 meets a non-inferiority margin, or ensemble v2 with a rule-based exact-match booster for those types. Add a regression suite and monitor online answer accuracy, recall, and escalation. Retraining on more tickets would worsen the imbalance.
3. Design the next online experiment and a decision rule that protects high-value strata.
Read the worked answer
Pre-register the hypothesis: v2 is non-inferior for contracts and invoices and superior overall. Randomize by query or session, stratify by document type, and use a fixed horizon or sequential testing with alpha spending. For a 3-point overall improvement from a 75% baseline, a two-proportion test at 5% significance and 80% power needs roughly 3,300 queries per arm. For contracts, oversample to at least 1,000 per arm or run a separate stratified analysis.
Primary metric: human- or judge-verified answer correctness. Secondary: recall@10, nDCG, latency, and escalation rate. Decision rule: ship only if the overall gain is at least 2 points with a 95% confidence interval lower bound above 0, and no stratum drops by more than 1 point; contracts and invoices must meet a 2-point non-inferiority margin. If contracts fail, keep routing. Monitor p95 latency against the 250 ms budget and segment by query length. This protects high-value traffic without blocking a genuine overall gain.