The Outlier, the Substitute, and a 60% IOA Session
Priya, a BCBA, oversees a clinic program for a 9-year-old whose aggression is recorded as a frequency count across 2-hour sessions. The last seven sessions read 14, 12, 15, 13, 11, 30, 12. On the 30-count day a substitute technician ran the session. Interobserver agreement was collected in five sessions using the total-count method: 91%, 88%, 60%, 93%, 90%. In the 60% session the primary observer recorded 6 responses and the secondary observer recorded 10. The treatment team wants to drop the 30-count session as an outlier and report the mean of the remaining six sessions in the quarterly report.
1. Evaluate the IOA data. Is the 60% session a problem with the observers, the formula, or the metric itself, and what should happen next?
Read the worked answer
Total-count IOA = smaller count ÷ larger count. The 60% figure is arithmetically correct (6 ÷ 10), so the formula was not misapplied. The metric is the problem.
- Total count is sensitive only to the aggregate. Two observers can disagree on every instance within a session and still score 100% if the totals match. With counts of 10–15, one missed response costs roughly 7–10%, so a 91% score may represent as few as one disagreement.
- 60% is below the conventional 80% criterion. That session's data are not credible, and the target for change is the measurement system, not the datum: retrain both observers on the response definition, run a calibration check (video or in-vivo), and collect IOA again on a comparable session before trusting the counts.
- A better metric here would localize disagreement - exact count agreement per 30-minute block, or convert to partial-interval recording in 10-second bins.
2. Can the team drop the 30-count session and report the mean of the other six?
Read the worked answer
Not as described. Discarding a datum because it inflates or deflates a mean is data manipulation, and it would leave the graph showing a level that did not occur.
A defensible path:
- Look for a documented procedural deviation on that day (substitute technician, different setting, shortened session, changed activity). If a pre-specified invalidity criterion is met, the session can be flagged - but it stays on the graph, annotated, and the exclusion is stated in the report.
- No IOA was reported for that session. Without it, observer error cannot be claimed, only suspected.
- The substitute is better treated as a condition change than as noise: a novel implementer plausibly alters the discriminative stimuli and the establishing operations in the room. Report it as a possible confound.
- If the question matters, re-present the condition (a second session with the same substitute) rather than deleting the point.
3. Describe the visual analysis you would report for these seven sessions and the decision that follows from it.
Read the worked answer
Six points cluster at 11–15 with no trend; a single spike to 30 is not a level change. Standard practice:
- Do not describe a trend without at least three consecutive data points moving in one direction, and do not call a change in level from a single session.
- Do not report a mean that hides the spike; report the range and the session-by-session data.
- Keep the point, note the implementer change on the graph, and continue as scheduled. If 30 recurs when the substitute returns, the variability is a variable to analyze - for example, low rates of attention or a high rate of demands during that technician's sessions - not an outlier to delete.
- State explicitly in the report that the 30-count session lacked IOA and coincided with a change in implementer.