The Confidence Interval the Client Doesn't Want
You lead a six-person team contracted to evaluate a municipal workforce program. Your final report estimates the program raised employment by 1.8 percentage points, with a 95% confidence interval of −0.4 to +4.0. The contract requires the report to end with 'a clear recommendation on continuation.' The program director - who is also the contract's technical monitor and controls a renewal worth 40% of your firm's annual revenue - has asked you to drop the confidence interval from the executive summary because it 'confuses council members,' and to call the result 'promising.' The council votes in three weeks. Your firm's principal has told you to 'keep the client happy without lying.' A local reporter has already filed a public-records request for the underlying data.
1. The director's request has two parts. How do you handle each, and what is your strongest argument?
Read the worked answer
Split the request. Removing a confidence interval from an executive summary is presentational and negotiable. Calling a result 'promising' when the interval spans zero is substantive and is not.
Presentational: agree to cut the interval from the summary - but only if a plain-language sentence replaces it. Something like: 'The estimate is positive, but the study is too imprecise to rule out either no effect or a substantial effect.' Council members do not need '−0.4 to +4.0'; they do need to know the study cannot settle the question. That is a concession that costs nothing true.
Substantive: decline 'promising.' A confidence interval spanning zero means you cannot claim the program worked, and the point estimate sits close enough to zero that you also cannot rule out a small harm. 'Promising' asserts a direction the data do not support.
Your strongest argument is not ethics, it is exposure: a public-records request is already filed. If the executive summary says 'promising' and the released data says the interval spans zero, the discrepancy is a story and the council's vote is tainted by it. Keeping the summary consistent with the body is the director's protection too. Make that argument to the director and to your principal, and put it in writing.
2. Can you give the 'clear recommendation on continuation' the contract demands? What should it say?
Read the worked answer
Yes - a clear recommendation about the decision, not about the effect. 'Clear' means unambiguous, not confident. Conflating the two is how evaluators end up overclaiming to satisfy a contract clause.
Recommendation: continue the program at its current scale but do not expand it this cycle, and fund a stronger evaluation, because the present study cannot answer the question the council is actually asking. Continue, because the point estimate is positive and the plausible downside is small; do not expand, because a −0.4 lower bound means a meaningful benefit has not been demonstrated. Those are two different decisions and the council should vote on them separately.
Give the design arithmetic. The current interval is roughly ±2.2 percentage points. Halving that half-width requires on the order of four times the sample, or a phased or randomised rollout that gets you comparison evidence for free. That converts an abstract 'we need better data' into a cost the council can weigh.
What not to write: 'the results are promising and further study is warranted.' It satisfies the contract clause, commits to nothing, and tells the council that a study spanning zero found something.
3. The director escalates to your principal, who tells you to drop the interval. What do you do now?
Read the worked answer
Do not relitigate the statistics. The argument has moved from method to authority, and repeating the method will lose. Escalate the risk framing instead, ascending in order:
- Settle signature authority in writing. Ask who is the named author of record. If your name is on it, you are the one attesting, and that is a fact the principal has to work around rather than a preference you are asserting.
- Re-offer the compromise in writing - interval out of the summary, plain-language uncertainty sentence in. This makes clear you have accommodated, and it puts the refusal to accept a reasonable middle on the record.
- If overruled, protect internal consistency. A summary that contradicts the body is worse than either version alone. Insist the summary carry a pointer to the interval in the body.
- If asked to sign something you believe misrepresents the finding, decline to be the named author and ask that the disagreement be recorded in the file. This is a serious step; take it once, calmly, not as a recurring threat.
Do not mention the reporter's records request as leverage. It is true and it is irrelevant to the professional judgement; using it makes the whole dispute look tactical, and it hands the director a reason to end the conversation.