Auto-closing duplicates: precision, cost asymmetry, and a confidentiality boundary

The number. 47 auto-closes, 9 reopened → 38 correct, so precision is 38/47 ≈ 80.9%, i.e. roughly a 19% false-positive rate on an irreversible action. Two caveats before using it:

• Sample size. n=47 is small; the 95% Wilson interval spans roughly 67–90%. You cannot distinguish "good enough" from "clearly too low" yet.
• Base rate. 47/1240 = 3.8% of inbound issues are even candidates. Precision on a rare action says nothing about the agent's overall competence, and small precision losses generate a large share of all its errors because the action is rare but high-consequence.

What precision doesn't tell you. It ignores cost asymmetry. A wrong close on a regression or security report costs a reporter's trust and possibly a missed vulnerability; a missed duplicate costs a maintainer a few seconds. Treating those as equal is the error, not the 80.9%.

Design under the 30-minute constraint. Don't add blanket review - the reviewer cannot absorb 1,240 issues. Change the action shape instead:

1. Default to advisory: comment with the candidate duplicate link and apply a possible-duplicate label. No close.
2. Auto-close only when all of: similarity/confidence above a tuned threshold, no security, regression, or confirmed label present, and 72 hours elapse with no objection on the thread. This converts a one-shot irreversible decision into a reversible, time-delayed one, and the 72-hour wait costs nothing.
3. Route the small residual (high-confidence but sensitive-labeled, or reporter pushback) to the reviewer's queue - that queue is what the 30 minutes is for.
4. Monitor reopen rate over a rolling 50 closes with a control limit; if it exceeds ~10%, revert to advisory-only automatically. Instrument it now so the decision is data-driven rather than a future argument.

Classification. This is a policy-boundary failure, not an output-quality failure. The agent produced a well-formed, accurate comment; what failed is that nothing in its execution path was aware that this issue had become confidential. Because the label arrived two minutes before the post, it is also a time-of-check/time-of-use problem: any decision the agent made earlier in the run (or in a warm context) was based on stale state.

Why prompting is not the control. Adding "never comment on security-sensitive issues" to the system prompt makes the model responsible for a security property. Models forget instructions under long context and cannot reliably know label state that changed mid-run. Security properties belong in deterministic code.

Controls that would have prevented it:

1. Pre-execution gate in the tool wrapper, evaluated immediately before the create_comment call: re-fetch current issue labels; if any label is in a deny-list (security-sensitive, embargoed, private), suppress the public write and instead write to a private draft channel for maintainer review. This closes the TOCTOU gap because the check happens at the moment of the side effect, not at planning time.
2. Invalidate in-flight runs on label change. Subscribe the label event to a handler that cancels or re-plans any active run for that issue. Two minutes is well within a typical agent run.
3. Log the label snapshot at write time with every comment the agent posts, so post-incident you can reconstruct what the agent knew rather than what the issue looks like now.
4. Fail closed. If the label fetch errors or times out, do not post.

With those in place, the reviewer's 30 minutes go to the escalation queue rather than to reading every comment - which is the only version of this that scales past week one.