Unreliable Off-Task Data Before a Placement Change
A middle school behavior team is monitoring 'off-task' behavior for Jamal, a 13-year-old with ADHD, in a 45-minute science class. The paraprofessional uses a 10-second momentary time sampling sheet and records off-task if Jamal is not looking at his assigned work at the beep. After four weeks, the team reports off-task in 62% of intervals and plans to move Jamal to a self-contained setting. A second observer conducted a 15-minute interobserver agreement check and obtained 48% agreement. The teacher says Jamal is 'fine' and that the paraprofessional 'doesn't like him.' The behavior specialist notices the definition does not specify what counts as looking at work and that the paraprofessional often records during teacher lecture, when no independent work is assigned. The class has 28 students and one teacher; the paraprofessional also supervises a study hall.
1. Evaluate whether the data are adequate to support a move to a more restrictive setting.
Read the worked answer
The data are not adequate for a change to a more restrictive placement. The definition, 'not looking at assigned work,' is not operational: it does not specify duration, what counts as looking, or whether lecture periods have assigned work. The paraprofessional recorded during lecture, when independent work was not assigned, so the denominator is invalid for part of the class. Interobserver agreement of 48% is far below acceptable agreement and suggests the observers are not measuring the same thing, not merely that one observer is biased. The teacher's contrary impression and the possible conflict with the paraprofessional are additional reasons to pause. A 62% score derived from unreliable measurement cannot support a high-stakes decision. Before any placement change, the team needs a precise definition, a measurement system matched to relevant instructional periods, and retraining with practice until agreement is consistently high.
2. Propose a corrected definition and measurement plan.
Read the worked answer
I would define off-task as eyes oriented away from the assigned worksheet or screen for more than 3 consecutive seconds during independent work, or engaging in a competing activity such as a phone or drawing instead of the assigned task. Teacher lecture and transitions would be excluded. Measurement would use 10-second partial-interval recording during the 20-minute independent-work block, not the full 45 minutes, with two observers scoring the same live or video session. The paraprofessional would be retrained with a practice video and a criterion check before collecting official data. Interobserver agreement would be collected on at least 20% of sessions. I would also collect permanent products such as work completion and a brief weekly teacher rating. If off-task is low during lecture but high during independent work, the intervention target should be task initiation and help-seeking, not placement.
3. What should the behavior specialist communicate to the team, and on what timeline?
Read the worked answer
I would tell the team plainly but without blame: 'The current data are not reliable enough to make a placement decision; here is the plan to fix measurement over the next two weeks.' I would present the definition, observation schedule, and agreement target, ask the teacher to identify the most concerning periods, and ask the paraprofessional to practice the code. We would schedule a review after three sessions with agreement of at least 80%. In the meantime, no placement change; we could add low-risk supports such as chunked assignments, a checklist, and a brief teacher check-in. I would document the original data, the concern, and the remediation plan. This protects Jamal from a decision based on measurement error and gives the team a fair basis for reassessment.