An Improving Trend Produced by an Unreliable Observer
An analyst collects data on aggression in a clinic using 10-s partial-interval recording across 20-minute sessions. Three observers rotate across 24 sessions. Summary sheets show aggression decreasing from 45% of intervals in week 1 to 12% in week 6, and the analyst plans to recommend fading the behavior plan. A colleague computes session-by-session IOA: 91% agreement for Observer A versus B, but only 58% for Observer C - who coded 8 of the 24 sessions, including all of weeks 5 and 6. Observer C was trained two weeks before data collection began, has not been recalibrated, and mentioned that she "sometimes codes a whole minute as one interval when it gets chaotic." The clinical director asks whether the intervention is working.
1. Is the downward trend interpretable? Walk through the evidence.
Read the worked answer
Not as it stands. IOA is a property of the data, not a formality. Observer C coded 8 of 24 sessions, including all of weeks 5 and 6 - precisely the sessions producing the trend - and agreed with other observers on only about 4 of every 10 intervals. The apparent decrease is confounded with observer drift.
Averaging IOA across the study conceals this: a weighted mean looks acceptable because A-versus-B sessions dominate the count, while the low-agreement sessions are the ones driving the effect. The reported shortcut ("code a whole minute as one interval") is a procedural change that systematically lowers the percentage of intervals scored whenever behavior is intermittent - exactly what a downward trend would look like.
Recommendation: suspend the fade decision. Recalibrate Observer C against a calibrated primary observer across three sessions with feedback to criterion, re-collect IOA for weeks 5-6, and re-plot using only sessions meeting a session-level IOA criterion (e.g., ≥80%). Accept the trend only if it survives.
2. What measurement-system concerns exist independent of the IOA problem?
Read the worked answer
Partial-interval 10-s recording is discontinuous and overestimates occurrence and duration: any behavior in an interval counts the whole interval. It is also sensitive to interval size - halving interval length changes the obtained score, so week-to-week comparisons are only valid if interval length is constant. At high rates it compresses real differences between conditions, making a plan look more effective than it is.
For aggression with discrete onsets and offsets, frequency/rate per minute or duration per occurrence is more defensible, and a frequency measure would have made observer C's shortcut immediately visible as a discrepancy.
Also check: response-definition drift (has the topography changed so that a new form is no longer coded?), reactivity when the analyst is observing, and whether the observers have been recalibrated since the definition was written. Measurement problems like these are supervision problems as much as they are statistical ones - the analyst owns the failure to monitor IOA in real time.
3. State a decision rule and describe how you would communicate this to the director.
Read the worked answer
Decision rule (state it in advance, in writing): only sessions with session-level IOA at or above criterion against a calibrated observer enter the analysis. A trend is accepted only if it holds after recalibration and re-collection. If the trend disappears, the plan is revised; if it holds, fading may be considered.
Communication:
- Present the session-level IOA table, not the mean - show that the low-agreement sessions coincide with the trend.
- Name the confound in behavioral terms (observer drift, procedural deviation), not as a personal failing.
- Present the remediation plan with dates: recalibration to criterion, re-collection, reanalysis.
- Be explicit that no clinical recommendation to fade will be issued until data quality is established - this protects the client from premature withdrawal of an effective treatment and from continued exposure to an ineffective one.
Handle the observer issue as supervision, not discipline: retraining with feedback to criterion, and a shared acknowledgment that the analyst should have caught the drift earlier.