A Sharp Drop That Coincides With a New Data Collector
At a home program for a 5-year-old, tantrums are defined as 'screaming and dropping to the floor for at least 10 seconds.' Baseline over five days averaged 8.2 tantrums per day (range 6–11). After a function-based intervention (functional communication training with break requests), the intervention phase shows a decreasing trend, with the last five days averaging 2.4 per day. Interobserver agreement was collected on 25% of sessions: frequency IOA averaged 95%, but duration IOA averaged 62% because two observers disagreed on tantrum offset. Midway through intervention, a new data collector began using a different definition: 'screaming for any duration.' The graph used for the funding report shows a sharp drop exactly when the collector changed. The parent wants to discharge, and the funding reviewer asks for proof of effectiveness.
1. Evaluate the credibility of the apparent treatment effect.
Read the worked answer
The effect is not yet credible. Frequency IOA at 95% is strong, but duration IOA at 62% is unacceptable for a duration measure; observers did not agree on when tantrums ended, so the 2.4 per day may be an artifact of offset rules. More importantly, the sharp drop coincides with a new collector using a broader definition ('screaming for any duration'). A definition change can produce a step change independent of behavior.
What to do: Separate the data by collector and definition. Re-score video if available using the original definition. If the drop occurs only after the new collector, the effect is confounded. Do not claim success to the funding reviewer until the measurement is corrected.
What remains possible: The intervention may be working, but the current graph cannot prove it. Frequency data are more trustworthy than duration, yet frequency also depends on the same definition. Treat the report as preliminary.
2. What measurement and IOA steps should be taken before claiming success?
Read the worked answer
- Operational definition: Restore 'screaming and dropping to the floor for at least 10 seconds' or revise it with the team; write examples and non-examples.
- Observer training: Train all collectors to 90% agreement on two video samples before independent data.
- IOA: Collect on at least 25–33% of sessions, including baseline and intervention; calculate exact-count IOA for frequency and total-duration IOA for duration. Report occurrence and nonoccurrence agreement if using interval data.
- Design: Add phase-change lines and mark the collector change on the graph. If possible, have a secondary observer unaware of condition.
- Stability: Collect at least five sessions per phase after the definition is stable before claiming an effect.
- Social validity: Confirm that the parent's definition of improvement matches the measurement.
Then present a corrected graph with caveats. If the corrected data still show a decrease, the claim is defensible.
3. How should you communicate with the parent and the funding reviewer?
Read the worked answer
To the parent: Be transparent: 'The first graph looked better than the data can support because a data collector changed the definition. We are re-checking before discharge.' Do not blame the parent or hide the issue. Offer a two-week verification period.
To the funding reviewer: Provide the corrected graph, IOA data, definition, collector changes, and a limitations paragraph. State what is known (frequency IOA 95%, decreasing trend by the primary collector) and unknown (duration reliability, definition confound). Propose continuation for two weeks with blinded IOA, or discharge with a maintenance plan if data are stable.
Ethically: Do not overstate effectiveness to preserve funding. If the reviewer requires proof, the honest answer is that the current evidence is inconclusive. Document the communication.