A Flat Response Distribution on Item 12: Too Hard, or Broken?
Setting. A three-teacher team is finalising a 15-item practice booklet for rising first graders. Item 12 shows a row of five frames with a sixth blank. Three features cycle independently. Shape runs a three-step cycle: square, triangle, circle, square, triangle. Fill runs a two-step cycle: shaded, unshaded, shaded, unshaded, shaded. Size runs a four-step cycle: large, medium, small, medium, large. In the tryout with 30 children, the five options drew 7, 6, 6, 6, and 5 responses. The team disagrees about what that means. One teacher says a flat distribution proves the item is too hard for the level. The other says it proves the item is broken and must be cut. The booklet goes to the printer in 40 minutes and the deadline cannot move, so there is no time for a second tryout.
1. Work out the intended answer and state what a random-responding child would produce. What does the observed distribution actually show?
Read the worked answer
Intended answer: a medium-sized, unshaded circle.
Trace each cycle to frame 6:
- Shape, period 3: square, triangle, circle, square, triangle → the next term is circle.
- Fill, period 2: shaded, unshaded, shaded, unshaded, shaded → the next term is unshaded.
- Size, period 4: large, medium, small, medium, large → the cycle repeats, so the next term is medium.
So the item is determinate. It is not ambiguous, and it is not broken in the sense of having more than one defensible answer.
Now the distribution. Thirty children, five options, random choice predicts 6 responses per option. Observed: 7, 6, 6, 6, 5. The largest deviation from chance is one response. No option attracted even a quarter of the group, and the correct option did not stand out at all.
That is what chance-level responding looks like. The item is not separating children who can reason from children who cannot; it is separating nothing from nothing. With 30 children you would expect the correct option to draw at least 12–15 if the rules were reachable at this level.
2. One teacher says 'too hard', the other says 'broken'. Which diagnosis is right, and why does the distinction matter here?
Read the worked answer
Both are half right, and the difference changes what you do.
'Broken' means the item is unfair or underdetermined: more than one answer fits, or the answer depends on a convention the child cannot know. That is not the case. The three cycles are shown in full, the continuation is forced, and the distractors can all be traced to specific rule failures. The item is internally sound.
'Too hard' means the demand exceeds what the children can hold at once. That is what the flat distribution shows. A child has to track three cycles with periods 3, 2, and 4 simultaneously and combine them at one point. Working memory, not logic, is the bottleneck - and for this age group three simultaneous cycles is well past what a single item should require.
The distinction matters because the remedies are opposite. A broken item gets cut. An over-demanding item gets simplified and kept, because the underlying design is good. Concluding 'broken' would throw away a well-constructed item whose only fault is that it asks for one rule too many.
3. Recommend a specific revision that fits the 40-minute deadline and keeps the item in the booklet.
Read the worked answer
Remove one cycle and hold that feature constant.
The cleanest cut is size: draw all six frames at the same medium size. Shape keeps its three-step cycle and fill keeps its two-step cycle, so the answer becomes a medium, unshaded circle. Two interacting cycles with periods 3 and 2 still produce a genuinely non-obvious answer - a child cannot get there by pattern-matching adjacent frames - but the load drops from three simultaneous rules to two, which is where the rest of the booklet sits.
Then rebuild the distractors so each isolates one failure: right shape with the wrong fill (circle, shaded); wrong shape with the right fill (square, unshaded); the previous frame repeated (triangle, unshaded); and right shape and fill with a size change, so the size error stays available even though size is no longer a rule.
Two honest caveats to record with the file. First, this revision is untested, so flag item 12 for review after the first real administration rather than treating it as settled. Second, keep it late in the booklet - a hard two-rule item belongs at the end, not in the first five.