The Recertification Cohort That Passes but Cannot Screen
A training centre runs an 8-day initial AVSEC screener course. The current written examination uses a 60-item multiple-choice test drawn from the course slides; the pass mark is 70%. Over the last four intakes, 94% of candidates passed on the first attempt, and the centre's management cites this as evidence of course quality. However, three months after graduation, supervisors at two airports report that new screeners routinely fail to identify cluttered X-ray images, do not apply the correct hand-search procedure when an alarm is unresolved, and cannot explain why a certain item is prohibited, only that it is. The training manager has one week and a fixed budget before the next intake. The regulator's oversight visit is in six weeks.
1. Diagnose the most likely gap between the current assessment and the stated performance problem, and justify your diagnosis with reference to assessment validity and learning outcomes.
Read the worked answer
The pass rate is near-ceiling while post-course performance is poor, which is the classic signature of an assessment that measures recall of slide content rather than job performance. The written test samples declarative knowledge (definitions, lists) and is aligned to what was taught, not to what must be done. Three separate construct-validity failures are visible in the supervisor reports: (1) image interpretation is a perceptual skill assessed only by recognizing pictures already seen in class, so the test cannot detect transfer to novel clutter; (2) procedural execution (unresolved alarm hand-search) is a psychomotor and decision skill never sampled by a paper test; (3) the prohibition rationale is being memorized as a list, so the learner cannot generalize to a new item. The correct diagnosis is that the course's terminal objectives were never written as observable performances, so the assessment could not be blueprinted to them. The fix is not to make the written test harder, but to add performance-based assessments tied to the real tasks, then rebuild the written items to sample the underlying knowledge that supports those tasks.
2. Propose a prioritized remediation plan for the one week available, and explain what you would defer to the six-week window.
Read the worked answer
Within one week, prioritize the highest-risk gaps that can be assessed with existing resources. First, add a practical image-interpretation check: 20 unfamiliar X-ray images with mixed clutter, scored pass/fail against a written key, with a criterion referenced standard (for example, all threat items detected and no more than two false alarms). Second, add a scenario-based oral or practical station for unresolved alarms, scored with a short behaviourally anchored checklist covering identification, containment, hand search, and escalation. Third, rewrite the written items so that at least half require application or justification (for example, 'why is this item prohibited and what is the screening consequence') rather than naming. Defer to the six weeks: full blueprint reconstruction, item analysis and bank expansion, instructor calibration on the new practical stations, and a formal validation report for the regulator. Communicate the interim changes to the regulator proactively, framing them as a corrective action with evidence, rather than waiting for the oversight visit to discover the mismatch.
3. What evidence would you present to the regulator to show the revised assessment is fit for purpose?
Read the worked answer
Present a chain of evidence rather than a single statistic. Include: the task analysis or job performance list that the course and assessment are mapped to; an assessment blueprint showing the proportion of items and stations per domain and per cognitive level; the practical station scoring rubrics with inter-rater agreement data from instructor calibration; item difficulty and discrimination statistics for the written test; a sample of candidate performance across cohorts showing the distribution, not just the pass rate; and a follow-up sample of supervisor observations at three months post-course. The narrative should state explicitly what each instrument can and cannot measure, and how the combination covers knowledge, skill, and judgement. Where data are not yet available, state the timeline and the interim controls.