Prior-Authorization Assistant: Detecting Quality Regressions Before Trust Erodes
A hospital network deployed a Gemini-based prior-authorization assistant to 1,400 case managers. It summarizes charts, retrieves payer policies, and drafts authorization letters. Adoption is 63%, but weekly override rate rose from 9% to 14% in six weeks. The team sees no infrastructure incidents: latency and error rates are normal. Case managers report that answers look fluent but sometimes cite a withdrawn policy. The policy corpus is updated twice weekly, and 22% of queries are for two recently acquired payer contracts with little training data. There is no labeled production feedback beyond free-text comments and override reasons. Compliance requires a sample-based review of 200 interactions per month and evidence that model changes do not degrade health-equity outcomes. The team wants a monitoring plan that catches quality regressions before case managers lose trust.
1. Design a monitoring plan covering operational, retrieval, and answer-quality signals. Explain how to get labeled feedback without a large annotation team.
Read the worked answer
Separate operational monitoring from quality monitoring. Operational signals include latency, error rate, quota, and tool-call failures; they are currently normal and do not explain the override increase. Retrieval signals include index freshness, policy version, retrieval recall on a golden set, and citation validity against the current corpus. Answer-quality signals include override rate, edit distance on drafted letters, abstention rate, groundedness, and citation entailment.
Do not rely on free-text comments alone. Build a lightweight feedback loop: structured override reasons, a one-click 'citation wrong' flag, and a weekly stratified sample of 50 interactions for human review, plus the required 200 monthly compliance sample. Use an LLM judge calibrated against that human sample for daily scoring, but keep human review for policy citations and equity slices.
Alert on cohort-level changes, not just global averages: by payer, product line, tenure, and language. A rise from 9% to 14% overall may be concentrated in the two new payer contracts.
2. How would you triage the override-rate increase? List hypotheses and the evidence you would collect first.
Read the worked answer
Start by segmenting the override rate by payer contract, policy version, model version, prompt version, case-manager tenure, and query intent. Check whether the increase began with a policy-corpus update, a model or prompt change, or the onboarding of the two new payer contracts. Compare citation validity and retrieval recall for the affected cohorts against a pre-change baseline.
Hypotheses include: withdrawn policies still in the retrieval index; new payer policies poorly represented; a prompt or model change that increased fluent but unsupported answers; case managers overriding more because they distrust the assistant after seeing bad citations; and selection bias in who uses the tool. Review 30–50 overridden interactions, focusing on citation correctness and whether the draft letter omitted a required criterion.
Quarantine stale policy versions and re-run retrieval. If the new payer contracts are the main driver, add their policies and examples before changing the model. Do not treat the override rise as a model-quality problem until retrieval and corpus freshness are ruled out.
3. What monitoring and change-control evidence would satisfy the compliance requirement on health-equity outcomes?
Read the worked answer
Define equity slices in advance with compliance and clinical leadership, such as language, age band, disability status, and payer type, while respecting privacy and minimum cell sizes. Monitor approval-letter completeness, citation validity, override rate, and abstention by slice. Do not infer protected status from unrelated data; use only fields that are approved and necessary.
For every model or prompt change, run a pre-deployment evaluation on the equity slices and compare against the current production baseline. Record the evaluation dataset version, slice metrics, reviewer sign-off, and rollback plan. Use the monthly 200-interaction sample to audit real decisions and adverse action reasons, not just offline metrics.
Set thresholds for material degradation and require human review before rollout if a slice falls below baseline. Keep an audit trail linking each change to its evaluation evidence. This gives compliance evidence that changes are tested, approved, and reversible, rather than relying on aggregate satisfaction scores.