The Prep Course That Raises Scores Without Changing Graduation Rates
A training battalion adds a three-week "language aptitude foundations" block to its pre-accession pipeline. It is built from publicly described test structure: students drill invented-language rule sets, practise stress-pattern discrimination on synthetic audio, and learn answer-elimination habits for the visual rule section. Across two cohorts (n=41), the mean composite rises 11.4 points between a pre-test and a post-test, and nine soldiers who had previously scored below the command's 90-point line cross it. Six months later the language school reports that graduation and washback rates for soldiers from this battalion are statistically indistinguishable from those of soldiers who never took the block. The battalion commander calls the course a demonstrated success and wants to expand it to 200 soldiers per cycle, which costs two instructor slots and roughly 600 training hours. The registrar asks for your read before the funding decision.
1. How do you reconcile a real score gain with no change in course outcomes?
Read the worked answer
The gain is real. The question is what construct it belongs to.
The test is a predictor, not the criterion. Its value is entirely in how well it forecasts performance in a long, hard language course. If scores rise and graduation rates do not, one of two things is true: either the test's predictive power has been weakened, or the soldiers were always going to graduate and the original scores were misclassifying them.
Which explanation fits? Look at where the 11.4 points came from and how the block was built. Synthetic stress-pattern drills and answer-elimination habits are format-training. Format training raises scores by making the instrument more familiar; there is no reason it should raise language-learning ability, and the outcome data say it did not. That is the first explanation, not the second.
Consider the second explanation seriously anyway. If the block genuinely reduced test anxiety, or gave soldiers a first exposure to analytic grammar outside their native language, the original scores may have been low estimates and the post-scores closer to the truth. Under that reading the block has screening value - it prevents false negatives - but it still has no demonstrated training value, which is what the expansion is being sold on.
The tension: the same 11.4 points support either story. Only the outcome data can separate them, and they do not support a training effect.
2. The commander wants to expand to 200 soldiers per cycle. Give a recommendation with conditions.
Read the worked answer
Recommendation: do not expand on the current evidence. Fund a smaller evaluation instead, and keep the course running at current scale only if a decision is needed before the evaluation finishes.
Reasoning. The stated justification is score gain. Score gain is an input measure. The two instructor slots and 600 hours are being traded for an outcome that has been measured and has not moved. Expanding now would spend the full cost to learn what a smaller study could tell you cheaper.
What the evaluation needs: a prospectively assigned comparison group, pre-test on both groups, the same post-test, and - the only number that matters - graduation and washback rates tracked through the language course. Track also whether coached soldiers are being seated in courses they would not otherwise have qualified for, because that is the mechanism by which a coaching program can quietly raise attrition.
Conditions under which expansion becomes defensible: coached soldiers graduate at a measurably higher rate, or the same rate at lower cost, or the block is shown to prevent false negatives (soldiers who would have been wrongly excluded now pass and succeed). Any one of those justifies a limited expansion with continued tracking.
The honest framing for the commander: the course may be doing something useful. The current evidence cannot tell you which of three useful things it is doing, and 200 seats per cycle is an expensive way to keep not knowing.
3. A staff officer proposes adopting the block's own tests as the unit's selection measure, arguing they are 'aligned with training.' Evaluate.
Read the worked answer
This inverts the logic of a predictor.
Alignment with training is a virtue in a test only when the training is the criterion. If the unit's goal is to predict success in a 63-week language course, a measure that correlates with that course's outcomes is valuable. The block's own tests correlate with the block. They have no demonstrated relationship to anything downstream - that is precisely what the six-month follow-up failed to show.
It also creates a closed loop. Teach to your own test, select on that test, and you will select for people who are good at your test. The loop is self-confirming and tells you nothing about the world. Commanders will see rising scores and read them as rising quality.
There is also a fairness cost. A locally built, locally taught measure is only available to soldiers who attend the block. Anyone who arrives from another installation, or who cannot be scheduled into the three weeks, is measured on something else. You have replaced a standardized instrument with a local one and made eligibility depend on scheduling.
Defensible alternative: keep the block as preparation if you like, but keep selection on the standardized measure with its published thresholds, and invest in the outcome tracking that would let you justify a change later. If the standardized measure is genuinely inadequate for your population, the answer is a validation study against your own graduation data - not a substitution by assertion.