The Substitute's Numbers
Devon is 8 and receives in-home services four afternoons a week, three hours per session, for aggression toward his brother. The behavior plan calls for event recording: every instance of hitting, kicking, or throwing an object that makes contact is clicked on a handheld counter, and the total is divided by session hours. For six weeks your rates sit between 3.5 and 4.5 per hour. In week seven your car breaks down and a technician hired two weeks ago covers two of your sessions, recording with paper tallies. Her sessions yield 5.2 per hour and 1.1 per hour. The BCBA asks you to assess agreement before the week's data are graphed, and mentions that if agreement is poor she will drop the week from the graph.
1. What can you actually do with the data that already exist, and what would you compute? Be specific about the computation and about what you would report.
Read the worked answer
There is nothing to compute for those two sessions. Interval-by-interval or total-count agreement requires two observers recording the same events at the same time, independently. Your 3.5-4.5/hr sessions and her 5.2 and 1.1/hr sessions are different sessions, different observers, and different days, so any comparison between them is a comparison of conditions, not a measure of agreement.
What you can do:
- Arrange a simultaneous, independent observation - a second observer live or from video - using the same written definition, and compute total count IOA as (smaller count ÷ larger count) × 100. With 14 and 18 instances that is 77.8%.
- Report IOA only for sessions where two independent records exist, and label the graph accordingly.
- Report the two substitute sessions as data with a note about the observer change, so the BCBA can decide whether to graph them, rather than silently treating them as equivalent to your own.
2. A simultaneous observation in week eight gives 14 and 18 instances. Interpret the resulting number.
Read the worked answer
Total count IOA = 14 ÷ 18 × 100 = 77.8%, which falls below the commonly used 80% criterion, so agreement is questionable and the definition or the observers need attention before the data are trusted.
What it does not tell you: it is a session-total comparison only. Two observers can reach 78% on totals while disagreeing about which events counted - one counts a throw that hits the wall and the other only counts contact, one misses low-amplitude hits, one credits a two-hit sequence as one instance. It also rests on a single session; a single agreement estimate is unstable, so you would want several.
A better supplement is exact agreement per event or per short interval (e.g., 10-second bins) so you can locate the disagreement rather than just quantify it.
3. The substitute's 1.1 per hour is far below every other session on record. List three explanations other than "the behavior improved" and say how you would discriminate among them.
Read the worked answer
Plausible alternatives:
- Definitional drift / observer error - she may not count throws that miss, may miss low-amplitude contact, or may under-tally because paper tallies get lost or combined.
- Different conditions - brother absent, different activity, different time of day, session shortened without the divisor being corrected.
- Reduced procedural fidelity or reactivity - a new technician may set up less demanding contexts, or Devon may behave differently with an unfamiliar adult.
How to discriminate: read the session notes for brother's presence and actual session length; recompute rate from raw counts and minutes; review the written definition with her and quiz her with video clips; have a second observer shadow one of her sessions; and compare her counts to yours on the same video-recorded session. Only after these checks would you consider a behavior-change explanation.