Designing an Instructor Competency Check That Means Something
A national AVSEC training organisation has 22 instructors delivering a standardized screening course across five sites. Instructor selection currently relies on a written exam plus five years of screening experience. Site managers complain that some instructors with strong records produce weak graduates, while one instructor with only three years' experience consistently produces the best results. The organisation wants a competency framework and an assessment that predicts classroom effectiveness, and it must be defensible to the regulator. There is no observation scheme in place, and instructors are evaluated once at hiring and never again.
1. Propose the competency domains for an AVSEC screening instructor and justify each with the performance it predicts.
Read the worked answer
Four domains are needed. (1) Technical and regulatory competence: current knowledge of screening procedures, prohibited items, and the applicable regulations, because an instructor who is out of date teaches errors that screeners then apply. (2) Instructional delivery: the ability to explain, demonstrate, question, and manage a classroom, because content knowledge alone does not transfer. (3) Assessment literacy: the ability to write and score performance-based checks, give feedback, and recognize invalid assessment, because graduates are only as good as the checks that certify them. (4) Professional conduct and safety culture: consistency, integrity in scoring, and willingness to report and correct errors, because instructors model the behaviour screeners will copy. Experience years is a weak proxy; it should be a prerequisite for technical currency, not the selection criterion. The framework should define observable behaviours at three levels (developing, competent, exemplary) so it can be assessed, not just described.
2. Design an assessment process that predicts classroom effectiveness and can be defended to the regulator.
Read the worked answer
Use multiple instruments and multiple observers. Include a knowledge check on current regulations and procedures, a live teaching observation scored with a behaviourally anchored rubric (for example, clarity of explanation, use of questions, correction of errors, time management), a practical demonstration of a screening task, a review of the candidate's own assessment materials against a quality checklist, and a structured interview on handling a difficult classroom or an ethical pressure. Calibrate observers before use, record inter-rater agreement, and require at least two independent observers for a pass decision. Include a candidate portfolio of graduate outcomes as supporting evidence, not as the sole basis. Defensibility comes from the explicit link between each rubric dimension and the performance it predicts, documented observer training, and a review cycle. Reassessment should occur annually and after any significant regulatory or equipment change.
3. How would you validate that the framework actually predicts effectiveness, given only 22 instructors?
Read the worked answer
With 22 instructors, formal psychometric validation is limited, so use a pragmatic triangulation. Collect the competency assessment results, then compare them with independent outcome measures: graduate practical assessment scores, supervisor observations at three months, and graduate pass rates on performance-based checks. Look for consistent rank ordering rather than statistical significance, and examine outliers in both directions to see whether the rubric missed something. Run a small criterion study over two cohorts, and use the results to revise rubric dimensions that do not discriminate. Supplement with a qualitative review of the two instructors cited as contrasting cases to check whether the framework captures what actually differs. Report limitations honestly to the regulator and set a three-year review with accumulating data.