The 75 Percent That Passed
You write a 20-question pre-solo knowledge check for three students. Question 4 asks for "the most important preflight action," and the correct choice is the only long, richly detailed option. Questions 7 through 9 test turbine-engine theory, which you have never taught and which is irrelevant to your training aircraft. Question 12 offers options containing "always" and "never." Your stated pass mark is 70 percent. One student, Priya, scores 75 percent - but she missed all four weight-and-balance questions, a subject you covered for two full lessons and which the school requires before solo.
1. Evaluate this knowledge check against the characteristics of a good test, and say whether the 75 percent is usable.
Read the worked answer
Validity (does it measure what was taught?): weak. The turbine items test material outside the objectives, so part of the score measures the wrong thing.
Reliability (does it measure consistently?): weak. Question 4 cues the answer by length; question 12 offers absolute words that let a test-wise student eliminate options without knowledge. Both produce scores that would change if the items were rewritten, so the number is not dependable.
Objectivity: questionable. "Most important preflight action" depends on the instructor's opinion unless the criterion is stated; two instructors could defend different answers.
Usability: fine - short, easy to administer and score.
Comprehensiveness: poor. W&B - a required, heavily weighted topic - is only four items, and three items are wasted on turbine theory.
Verdict: 75 percent clears 70 percent only by luck, and the margin is smaller than the measurement error. Do not treat the score as a decision; treat it as a symptom.
2. Priya missed every weight-and-balance question. How should that drive your decision, and what do you do next?
Read the worked answer
This is the core of criterion-referenced assessment: the question is not "what score did she get?" but "has she met each required objective?" She has not met the W&B objective, and the total score hides it by averaging a deficiency against strengths.
Actions:
- Do not sign off the solo knowledge requirement yet; the omission is a required element.
- Reteach W&B to the specific errors - talk through her missed items with her; the wrong answers are diagnostic, not just points lost.
- Retest W&B with new items, not the same four, so you are measuring learning rather than memory of the key.
- Fix the instrument for the other students too: delete the turbine items, rewrite question 4 so the correct option is not identifiable by length, remove the absolute-word cue from question 12, and build the check from the lesson objectives so every required topic carries proportional weight.
Avoid: dropping the W&B items to raise her average, or accepting 75 percent because it is above the line. Either move is grade inflation and it transfers an unassessed risk to a solo student.
3. Write a short guide your other instructors could use to judge whether a test item is sound.
Read the worked answer
Item-quality checklist:
- Tied to an objective. If it isn't in the lesson, it doesn't belong on the check.
- One defensible answer. State the criterion if the stem says "most" or "best."
- No structural cues. Options should be similar in length, grammar, and detail; the correct one must not be the longest or the most technical.
- No absolutes as shortcuts. "Always" and "never" let a student eliminate without knowing the material.
- Plain language. Test the aviation knowledge, not the reading level.
- Independent items. One item should not reveal the answer to another.
- Sampled proportionally. Weight items by the importance and time spent on each objective - a table of specifications, built before the questions.
- Scored against a standard, not a curve. Every student should be able to reach 100 percent, and a miss on a required element triggers remediation regardless of the total.