The pilot trap: what a successful trial actually proves
Public agencies have made the pilot the default unit of innovation, and it is a poor guide to scale. The reason is not that pilots test bad ideas. It is that a pilot is staffed by the unusually committed, funded from a discretionary budget that never has to survive an annual plan review, and observed by people who know they are being observed. Those three conditions are the pilot. None of them survives replication. The honest conclusion is not that pilots fail, but that they answer a narrower question than the one we ask of them: can this work at all? Decisions, unfortunately, need a different question: will this still work when the staff are ordinary, the budget is contested, and nobody is watching? The remedy is unglamorous. Cost the model at full market salaries, run it through the procurement rules that will actually apply at scale, and publish the failure modes alongside the results. Teams that do this look slow in the review meeting. They also scale more often, and their pilots stop being exhibits.
1. State the author's central claim and explain how it differs from the claim that pilots fail.
Read the worked answer
The central claim is that a pilot answers a narrower question than the one decision-makers are actually asking. It establishes whether an idea can work at all under favourable conditions; a scaling decision needs to know whether it will still work under ordinary conditions. These are different questions, and a pilot's success is not evidence about the second.
The support is a three-part mechanism: self-selected committed staff, funding insulated from annual plan review, and the observer effect ('observed by people who know they are being observed'). The author then asserts that none of the three survives replication.
The difference from 'pilots fail' is explicit in the text: the honest conclusion is not that pilots fail. If pilots failed, the prescription would be to stop running them. Instead the author prescribes costing at market salaries, applying real procurement rules, and publishing failure modes - all of which keep pilots and change what they measure. So the author is reformist, not dismissive; a candidate who reads this as an attack on pilots has missed the pivot sentence that the whole paragraph is built around.
2. Which of these would the author most likely endorse, and why would the others be rejected? (a) A pilot run by volunteers on donor money proves the model works. (b) A model should be costed at full market salaries before it is scaled. (c) Pilots should be abandoned in favour of direct scaling. (d) Publishing failure modes slows teams down and should be avoided.
Read the worked answer
The author endorses (b). It is almost a paraphrase of the stated remedy: 'cost the model at full market salaries' and 'run it through the procurement rules that will actually apply at scale'. Both are ways of importing the conditions of scale into the pilot itself.
(a) is rejected by the mechanism. Volunteers and donor money are two of the three favourable conditions the author says constitute the pilot. Success under them is precisely the evidence that does not transfer.
(c) contradicts the text. The author says the conclusion is not that pilots fail, and the remedy is a better-designed pilot, not no pilot.
(d) inverts the prescription. The text says teams should 'publish the failure modes alongside the results', and concedes only that such teams 'look slow in the review meeting' - an appearance the author plainly regards as misleading, since the same teams 'scale more often'. The apparent cost (looking slow) is set against the real benefit (scaling more often), and the author comes down on the side of the benefit.
3. What does 'contested' mean in this context, and what is implied by the closing phrase that such teams' pilots 'stop being exhibits'?
Read the worked answer
'Contested' here means disputed and liable to be challenged - a budget that must be argued for line by line in an annual plan and can be cut, as opposed to the discretionary funding a pilot enjoys. It signals the absence of guaranteed money, not the presence of hostility; the point is that ordinary programmes must win their funding repeatedly, while pilots do not.
'Stop being exhibits' carries the author's attitude. An exhibit is something put on display to demonstrate a desired conclusion - a showpiece for a review meeting, a photograph for a press note. The implication is that many pilots function as political theatre whose real product is a favourable impression, and that the author regards this as the root of the scaling problem. Once a team costs its model honestly and publishes failure modes, the pilot becomes an instrument that produces knowledge rather than a display that produces approval.
The tone is therefore sceptical of the theatre but sympathetic to practitioners and prescriptive about the fix - not cynical. 'They look slow' concedes that the author's preferred path carries a real reputational cost inside organisations, which is why the sentence reads as advocacy rather than scolding.