Evaluating a Seminar-Budget Pilot
The teaching committee at a specialized master's program proposes moving 15% of the first-year lecture budget into seminars capped at 20 students. It cites a pilot in which students in seminar sections scored 6 points higher on a standardized reasoning test than students in lecture sections. The committee concludes that the budget shift will improve reasoning across the entire cohort.
Three details complicate the inference. First, seminar seats in the pilot were voluntary, and volunteers had entry scores about 5 points higher than non-volunteers. Second, the pilot replaced only two of twelve lecture sections, so seminar students may have benefited from extra faculty attention rather than from the seminar format itself. Third, the test consists entirely of multiple-choice logic items and includes no essay or data-interpretation tasks. A dean objects that the test measures only one narrow kind of reasoning.
1. State the committee's main conclusion and identify one unstated assumption the argument relies on.
Read the worked answer
Main conclusion: shifting 15% of the first-year lecture budget to small seminars will improve reasoning across the whole cohort.
The argument relies on the assumption that the 6-point pilot difference was caused by the seminar format rather than by pre-existing differences between students or by other features of the pilot sections. The setup gives direct reason to doubt this: seminar seats were voluntary and volunteers had higher entry scores, and only two of twelve sections were replaced, so extra faculty attention is a plausible alternative cause.
A second assumption is that the standardized test captures the kind of reasoning the program cares about across the cohort. The dean's objection targets this assumption, because the test contains only multiple-choice logic items and no essay or data-interpretation tasks.
2. Does the dean's objection undermine the committee's conclusion? Explain what it does and does not establish.
Read the worked answer
It weakens the conclusion but does not by itself defeat it. The dean establishes that the outcome measure is narrow: a multiple-choice logic test does not measure essay-based argumentation or data interpretation. If the committee's claim is that the budget shift improves 'reasoning' broadly, then evidence from only one narrow component is incomplete.
However, the objection does not show that seminars fail to improve the measured logic skills, nor that the measured skills are irrelevant. It also does not address the selection and attention confounds, which are separate and arguably more serious threats to the causal claim. The strongest reading is that the dean's point limits the scope of the conclusion: the pilot supports, at most, a claim about multiple-choice logic performance, not about all reasoning. To defend the broader conclusion, the committee would need evidence from additional outcome measures.
3. What evidence would most strengthen the committee's causal claim that the seminar format itself improves reasoning?
Read the worked answer
The strongest evidence would come from a design that removes the selection and attention confounds. For example, randomly assign eligible first-year students to seminar or lecture sections, keep the instructor, content, contact hours, and assessment identical across conditions, and compare outcomes. If seminar students still outperform lecture students under random assignment, the causal claim is much stronger.
It would also help to hold faculty attention constant: for instance, compare a seminar with 20 students to a lecture section that receives an equivalent additional small-group session, so that 'extra attention' is not confounded with 'seminar format.' Finally, the outcome measures should include essay and data-interpretation tasks, not only multiple-choice logic items, if the conclusion is about reasoning across the cohort. A replicated pilot across more than two sections would further reduce the chance that the result reflects one instructor or one unusual section.