The Four-Point Slide: Deciding What the Data Actually Say

What the data can support: very little causal inference.

• Sampling error first. With n ≈ 90 and a proportion near 0.75, the standard error is about 4.6 points, so a 95% confidence interval is roughly ±9 points. A move from 79% to 71% is inside that band; a move from 76% to 71% is marginal. Quarter-to-quarter movement of this size is largely noise unless it persists.
• Mode effect. Switching from mail-only to mixed web/mail changes who responds and how they respond. Without a bridge or overlap period, Q3 and Q4 are not cleanly comparable to Q1 and Q2. This alone could explain part of the pattern.
• The comparison is broken. Two units got the intervention and four did not, but results are reported in aggregate. Any unit-level effect is diluted and unmeasurable at this denominator.

Analysis to do in three weeks:

1. Re-cut by unit and by service line, pooling quarters to get adequate n per unit; compare pilot units to non-pilot units pre and post.
2. Build a p-chart or annotated run chart rather than a bar chart of quarters; ask whether any point exceeds control limits or whether there is a run of points on one side of the centerline.
3. Read all nurse-communication verbatims for the period, tagged by unit and theme, to see whether patient language changes.
4. Verify denominators, discharge counts, and exclusions; confirm whether the item set or vendor scoring changed.
5. Add at least one qualitative source (10-15 patient interviews or 20 staff conversations) so the board sees signal, not just percentages.

Framing for the board: "We have four noisy points, a measurement change midstream, and no valid comparison. We cannot yet say whether the pilot helped, hurt, or did nothing. Here is what we will know by [date] and how we will decide."

In the meeting: agree on the goal, separate the two questions.

"We both want nurse communication to improve and we both want it fast. I don't think we have the evidence yet to say the pilot failed - the swing is within sampling error and the survey mode changed. Can we agree to decide on the basis of a defined test rather than this quarter's number?" Naming the shared goal before disagreeing lowers the temperature and avoids a data argument becoming a turf argument, which matters when the counterpart is a physician leader with formal authority.

Why not a compliant script:

• Scripts raise the floor and lower the ceiling; audited compliance tends to produce ritualized speech that patients detect as performance.
• It converts a relationship behavior into a compliance behavior, which invites staff reactance and drives reporting underground.
• It cannot be evaluated, because compliance rate is not patient experience - you would be measuring the intervention, not the outcome.
• It imposes a system-wide fix for a problem that may be local to two units.

Counter-proposal (30-45 days):

1. Diagnostic sprint: 10 patient interviews, 20 staff conversations, and 5 observations on each unit; identify what patients actually experience as "not always."
2. Defined test: standard work (e.g., structured bedside handoff, purposeful hourly rounding, nurse leader rounding) on two units with two matched comparison units, with a pre-specified decision rule - a run of points beyond control limits on a monthly p-chart, plus a process measure of observed practice.
3. Do now, no regret: things with strong face validity and low harm - leader rounding, whiteboards updated, call-light response standard - implemented everywhere while the test runs.
4. Commit to come back on a date with a decision, not just more data.

One page, four blocks.

1. The picture: an annotated run chart or p-chart of nurse communication by quarter, with control limits and vertical markers for the vendor mode change and the pilot start. The visual should make the noise visible without editorializing.
2. The honest claim: "We cannot yet distinguish signal from noise. Three explanations are plausible: normal variation, a measurement change in Q3, or a genuine change in practice. We are testing which." Resist ranking units publicly and resist percent-of-percent language ("a 10% drop in always") which magnifies small denominators.
3. What we did and what we will know: the diagnostic sprint, the defined two-unit test with comparison units, the pre-specified decision rule, and the date the board will get a yes/no answer. Include the process measure, not just the survey - a quarterly survey cannot steer a monthly operation.
4. The ask: agreement to hold off on a system-wide mandate until the test reads out; support for staff time to run the test; and, if the CMO's script idea is to be tried, permission to pilot it on one unit as a third arm so the comparison is fair rather than assumed.

Also include two or three patient verbatims with unit identifiers removed. Boards act on stories as well as numbers, and pairing a verbatim with the chart makes the uncertainty concrete rather than evasive.

What not to do: present a single percentage as a headline, use red/green traffic lights on n=90, or promise a 5-point gain by a date. Overpromising once costs credibility for every later report.

Accept the decision, protect the measurement.

• Implement what is directed, but negotiate three things in the same breath: a written baseline, a named comparison (units not yet mandated, or a staggered rollout), and an agreed review date. A staggered rollout turns a mandate into a natural experiment and costs nothing.
• Keep the script from becoming a compliance audit if at all possible - offer instead to observe practice and pair observations with patient-reported items, so staff experience it as feedback rather than surveillance.
• Document the rationale and the uncertainty in the minutes, neutrally. Not to be right later, but so the organization can learn if the intervention doesn't work.
• Watch for the predictable side effects: staff cynicism ("another flavor of the month"), script fatigue, and reduced willingness to report honestly on the next survey. Build a quick pulse item to detect them.
• If the mandate does produce improvement, say so publicly and credit the CMO. The credibility you build is what allows the next, better-designed test to be approved.