The Proactive Alert Whose Holdout Got Treated
Situation. Support analytics is evaluating a proactive firmware alert for OptiPlex fleets on NIC firmware 4.2.11. The design: 40,000 fleets randomized 70/30 - 28,000 receive the alert, 12,000 are a holdout - and inbound support contacts are counted for 60 days.
Three things go wrong. In week 2, account managers who were never told about the design mention the firmware issue in routine QBRs to 3,100 holdout accounts. Before the send, 900 alert-group customers opt out of marketing email and never receive it. On day 35 the telemetry team splits the single 'connectivity' contact reason into four sub-reasons and rewrites history only for the current quarter.
The sponsor wants one lift percentage with a confidence interval by Friday, and has already told the QBR audience that the alert 'cut contacts by about a third.'
1. Which estimand and analysis population should the lift number use? Quantify how the week-2 QBR mentions and the opt-outs distort a naive alerted-versus-holdout comparison.
Read the worked answer
Use intent-to-treat (ITT) on the original randomized assignment. The decision in front of the sponsor is whether to roll the alert out company-wide, which is exactly the effect of offering it - including fleets that never received it.
Dilution from the QBR mentions. 3,100 of 12,000 holdout fleets (25.8%) were effectively exposed to a weak version of the treatment. If those fleets respond like fully treated fleets, the holdout mean drops by 0.258 x delta, so the observed difference is only 0.742 x delta: a genuine 30% relative reduction in contacts surfaces as roughly 22%. Because a QBR aside is far weaker than a targeted alert, 25.8% is an upper bound on the dilution, so the observed effect is a conservative underestimate - acceptable for a go/no-go.
Do not switch to per-protocol. Dropping those 3,100 breaks the randomization: account managers choose strategic accounts, so the surviving holdout is systematically different and the resulting contrast is not causally interpretable.
Opt-outs. 900/28,000 = 3.2% assigned-but-untreated is ordinary ITT dilution; keep them in the primary analysis. Report a labelled as-treated sensitivity analysis beside the headline, not instead of it.
2. The contact-reason taxonomy changed on day 35. What breaks, and how do you still produce a metric comparable across the full 60 days?
Read the worked answer
What breaks: anything keyed on contact sub-reason. 'Connectivity' before day 35 is not comparable to four sub-reasons after it, and because history was rewritten only for the current quarter, the pre-period baseline outside the test lives on a different schema than the test window. The sub-reason mix chart will show a fake structural break at day 35 that has nothing to do with the alert.
What survives: total inbound contacts per fleet, provided the split preserved the parent mapping and no counting rule changed (deduplication window, one-contact-per-case-per-day, chat/voice inclusion). Verify that before publishing.
Plan: (1) build a sub-reason to parent mapping table and roll the primary metric up to parent level; (2) sanity-check on a fixed panel of ~200 fleets whose counts you can recompute from raw events under both schemas; (3) if the parent mapping cannot be reconstructed for days 1–34, restrict sub-reason analysis to days 35–60 and label it exploratory; (4) recompute the prior-year comparison window under the new taxonomy so the baseline is schema-consistent. Publish contacts per 1,000 fleets per week, never raw counts, because fleet counts drift.
3. Write the two-to-three sentence response you would give the sponsor about the 'about a third' claim.
Read the worked answer
"The randomized test supports roughly a 22% reduction in support contacts, with a 95% interval of about X to Y. The true effect is probably a little larger than 22% because some of the comparison group also heard about the issue from their account manager, but a third sits at the optimistic edge of the data rather than the centre. You will have the final number plus interval on [date]."
Two rules drive that wording. First, never hand a sponsor a point estimate without its interval when the sponsor will repeat it in public - the interval is what stops the number becoming a commitment. Second, correct the earlier claim immediately: a figure already said aloud in a QBR becomes the anchor for every later decision, and waiting until the analysis is final means arguing against a number people have quoted to customers.