Behavior-Based Safety: Flat Results at 60 Days
After a 6-hour behavior-based safety observation course for 90 supervisors, you find: reaction scores 4.6/5, post-test mean 91%, observation counts up 40% in the 60 days after, but recordable injuries flat and the audit's 'quality of observation' score unchanged at 62%. The baseline injury rate was already low (2.1 per 200,000 hours). Observations are logged in a system with known duplicate entries. The audit samples 10 observations per site per quarter across 8 sites. Leadership wants to cancel the program or expand it company-wide and will decide next week.
1. What can and cannot be concluded from these five data points?
Read the worked answer
Levels 1 and 2 are solid. Level 3 is plausible but confounded. Level 4 is not measurable at this base rate in 60 days.
Can conclude: supervisors can pass a knowledge test, and more observations are being logged.
Cannot conclude: that the course caused the 40% increase - a jump in logged counts is as likely to be a reporting push or a new expectation as a learning effect, and duplicate entries mean the figure needs de-duplication before it is quoted at all. Nor that the course failed.
The specific trap: a rate of 2.1 recordables per 200,000 hours across 90 supervisors' areas is a handful of events per year. A move from five events to four is noise. Flat injuries over 60 days is an absence of evidence, not evidence of failure.
The finding that matters: observation quality is unchanged at 62%. Counts went up without the behavior changing. That is a level 3 result, and it is the one that should drive the decision - not the injury rate, which cannot answer the question being asked of it.
2. How do you investigate the flat quality-of-observation score before blaming the course?
Read the worked answer
Check whether the measure and the course are talking about the same behavior.
Likely causes, roughly in order:
- Rubric mismatch. Item-by-item, does the audit score what the course taught? If the audit scores 'quality of the feedback conversation' and the course taught 'identification of at-risk behavior,' that is a transfer gap, not a training gap, and no amount of course redesign will move the score.
- Volume incentive. If observation counts are reported upward, supervisors optimize for counts. Quality decays when nothing downstream uses it.
- No feedback loop. Supervisors observe, data disappears, behavior reverts.
- Measurement artifact. 10 observations per site per quarter across 8 sites is 80 observations total. With ordinary inter-rater disagreement on 'quality,' 62% could be stable by construction and insensitive to real improvement.
Do three things: pull the audit rubric and map it against the course objectives, sit in on five live observations with the auditor to see how the score is assigned, and de-duplicate the log. If the rubric and course do not overlap, fix the rubric or the course before touching the training.
3. Leadership wants cancel-or-expand next week. What do you recommend, and what would change your mind?
Read the worked answer
Recommend neither. The framing is a choice between two unsupported conclusions, and both are irreversible.
Propose a bounded test:
- Keep the course. Levels 1-2 are strong and level 4 is unmeasurable at this base rate within 60 days.
- Fix the one known defect - observation quality - and re-measure at the next quarterly audit sample, which is the earliest defensible read.
- Expand only what is already validated - the knowledge component - while holding the coaching-heavy elements until quality moves.
- Clean the data first: de-duplicate the observation log and set a 12-month results window with a comparison group of not-yet-trained supervisors, if the rollout sequence allows one.
What would change this: if the rubric and the course overlap well, and quality stays flat after a real coaching intervention, the course design is wrong and expansion would scale a failure. Say that explicitly in the meeting so the recommendation reads as conditional on evidence rather than as a preference for delay.