When the Graph Says Yes and the Classroom Says No
A behavior analyst consults in a self-contained classroom for students with autism. The target is disruptive behavior during independent work. Baseline data are collected across five days using 10-minute partial-interval recording, yielding a mean of 62% of intervals with disruption. An intervention combining a visual schedule and a first-then contingency is introduced. The graph shows a clear downward trend to 30% of intervals over eight days. However, the teacher reports that disruption "feels worse than ever," and the paraprofessional says the student now cries during transitions. A second observer, present for two sessions, records 55% and 58% of intervals with disruption, while the primary data collector records 31% and 33% for the same sessions.
1. Interpret the discrepancy between the graph and the staff report. What measurement problems could produce this pattern, and how would you test each?
Read the worked answer
The graph and the classroom report are both data, and they disagree. The second observer's numbers are the key clue: a roughly 25-percentage-point gap on the same sessions suggests the primary collector's measurement is not capturing the same events the second observer sees.
Candidate explanations: (1) Observer drift - the primary collector may have gradually redefined disruption (for example, no longer counting crying or protest as disruption). Test by re-training with a written definition and a video or role-play calibration check, then computing interobserver agreement again. (2) Definition ambiguity - if disruption includes crying, the paraprofessional's report makes sense while the primary collector's does not. Test by having both observers independently code the same 10-minute segment and comparing interval-by-interval, not just totals. (3) Reactivity or expectancy - the primary collector knows the intervention is supposed to work. Test with blind coding from video. (4) Interval-recording limits - partial-interval data inflate estimates and are insensitive to rate changes within intervals, so a real change in intensity could be masked. Test by adding frequency or duration measures.
Until agreement is restored, the intervention's effectiveness is unknown, and the team should not conclude the plan is working.
2. The teacher asks you to keep the intervention because the graph looks good and the student "just needs to adjust." How do you respond, and what design would you use to settle the question?
Read the worked answer
The ethical response is to say plainly that the data are not trustworthy enough to support that conclusion, and that crying during transitions is a signal to look harder, not to wait. Continuing an ineffective or possibly harmful plan while reporting success would misrepresent outcomes to the family and the district.
Steps: first, suspend claims of effectiveness and re-establish measurement. Retrain the primary collector, tighten the operational definition, and collect interobserver agreement on at least 25-30% of sessions with a criterion of 80% or better before interpreting the graph. Second, add a direct measure of the concern staff raised, such as duration of crying or latency to transition, so the side effect is quantified rather than debated. Third, use a design that can separate the intervention from observer changes: a withdrawal (ABAB) design if disruption is safely reversible, or a multiple-baseline across work activities or students if withdrawal is impractical. If crying increases when the visual schedule is introduced, that is a treatment-integrity and side-effect question the design should answer. Finally, bring the family into the conversation and document the disagreement and the plan to resolve it.