The Baseline That Doesn't Add Up
You are an outside consultant hired by a 40-bed residential support agency. The director says aggressive incidents have "doubled" since a new shift schedule began six weeks ago and wants a written behavior support plan by Friday. You pull the incident records: 34 incidents across six weeks, but two of the fourteen staff filed 21 of them. In week 4 the agency replaced its incident form, and the new version explicitly includes "verbal threats," which the old one did not. The director also hands you a single 20-minute observation summary in which two observers agreed on only 11 of 20 intervals. There is no baseline from before the schedule change. The agency's funder requires a plan within 30 days of any "significant increase" in incidents.
1. Before drafting any plan, what must you establish about the data you have been given?
Read the worked answer
Do not treat the 34 incidents as a baseline. Four separate data-quality problems have to be resolved first:
- Definition break. The form changed in week 4, so weeks 1–3 and weeks 4–6 measure different things. Re-code all 34 narrative entries against a single written definition (with verbal threats included consistently) before plotting anything.
- Reporting source. Two of fourteen staff filed 62% of the incidents. That is either a real exposure difference - those two worked the highest-risk shifts - or a reporting-rate difference. Check shift assignment and incidents per staff-shift-hour before assuming either.
- Measurement error. 11/20 interval agreement is 55%, far below any usable criterion (typically ≥80%). One observation with disagreement that large cannot support a conclusion about rate, and cannot serve as a baseline.
- Denominator. Raw counts ignore census and shift hours. Compute incidents per 100 bed-days, by shift, so the comparison is like-for-like.
Then run three direct-observation sessions with two independent observers and an ≥80% agreement criterion before you characterize the behavior at all.
2. How would you test the claim that incidents have doubled, and what evidence would change your conclusion?
Read the worked answer
The claim is currently unverified, and artifact explanations are the most likely ones. Test in this order:
- Reconstruct a like-for-like series. Re-code weeks 1–6 under one definition; recompute the week 1–3 and week 4–6 rates per 100 bed-days.
- Stratify. Per shift, per staff member, per unit. If the apparent rise lives in one or two reporters on one shift, it is a recording phenomenon, not a behavior change.
- Triangulate with an independent stream. Injury reports, nursing notes, client or family complaints, police calls. Series that move together are more credible than a single form that just changed.
- Ask what actually changed operationally. A new schedule can change staffing ratios at high-risk times. That is a real mechanism worth testing, and it is testable.
What would change my mind: the increase persists after re-coding, appears across multiple independent reporters and shifts, and is corroborated by at least one independent data stream. Absent that, the correct finding is that a change cannot be demonstrated - not that behavior is unchanged.
3. The director wants a plan by Friday and the funder's 30-day clock is running. What do you deliver, and what do you tell each of them?
Read the worked answer
Deliver something honest on Friday; do not deliver a plan built on an invalid baseline.
- Friday, to the director: a two-page data-quality memo with the re-coded series, the stratification, and a safety review of the incidents that did occur - were there injuries, were response protocols followed, what changed after the schedule shift. This meets the director's real need, which is knowing whether clients and staff are safe, without pretending to a functional assessment you cannot yet support.
- To the funder, this week: the 30-day clock is triggered by a "significant increase," and that is precisely what is in dispute. Write to the program officer: describe the mid-period form change, the assessment plan, the date the plan will be submitted, and the interim safeguards in place.
- Do not let a deadline convert a hypothesis into a written finding. A plan that names a function from this data will misdirect the intervention and will be very hard to withdraw later, once staff have built their practice around it.