The Library's Extended-Hours Pilot Memo
A city library system ran a six-month pilot extending three of its eleven branches to 10 p.m. on weekdays. The memo announcing results reads in part: "Average daily visits at the pilot branches rose 18 percent. The pilot was therefore a success and should be extended to all branches." A staff analyst notes that the three pilot branches were the busiest in the system, that the comparison used the same branches' prior-year figures rather than the other eight branches, and that the 18 percent figure counts every person entering the building, including those attending free evening lectures that were also introduced during the pilot. The board meets in two weeks and has budget for either a full rollout or a one-year extension of the pilot at the same three branches.
1. State the memo's main claim and its main supporting evidence. Then explain why the main claim overreaches relative to that evidence.
Read the worked answer
Main claim: The extended-hours pilot was a success and should be extended to all eleven branches.
Supporting evidence: Average daily visits at the three pilot branches rose 18 percent over the pilot period.
The claim overreaches on two fronts. First, the evidence establishes only that visits rose at three branches; the conclusion jumps to all branches. Second, the 18 percent rise is a raw visit count, not a measure of demand for extended hours specifically. Because free evening lectures were introduced at the same time, some unknown share of the increase may be attributable to the lectures rather than to the later closing time. A defensible conclusion would be narrower: "Visits rose at the pilot branches, and the extension merits further testing." The gap between what was measured and what is claimed is the core of the problem.
2. The analyst raises three separate concerns. Identify each and explain what specific additional comparison or data would address it.
Read the worked answer
Concern 1 - Selection of branches. The three pilot branches were already the busiest, so they may not represent the other eight. Fix: report baseline visit rates for all eleven branches and note whether the pilot branches differ systematically (location, transit access, staffing).
Concern 2 - Wrong comparison group. The memo compared pilot branches to their own prior year, which cannot separate the pilot from citywide trends (population growth, a new transit line, seasonal shifts). Fix: compare the change at pilot branches against the change at the eight non-pilot branches over the same period - a difference-in-differences comparison.
Concern 3 - Confounded measurement. Total entries include lecture attendees, so the 18 percent does not isolate the effect of hours. Fix: report entries separately for lecture nights versus non-lecture nights, or exclude lecture attendees, or report after-8-p.m. entries specifically.
Each concern is distinct: one is about generalizability, one about causal attribution, one about what the metric actually counts. A strong answer keeps them separate rather than lumping them as "the data is flawed."
3. Given the board's two options, which would you recommend and why? Frame your answer around what the evidence can and cannot support.
Read the worked answer
Recommendation: extend the pilot at the same three branches for one year, with measurement fixes.
The evidence supports a modest, local conclusion: visits rose at these branches during a period when hours were extended. It does not support a system-wide rollout, because (a) the pilot branches are unrepresentative, (b) there is no comparison group, and (c) the visit metric is confounded by the lecture series. A full rollout would commit budget across eight untested branches on the strength of a number that cannot bear that weight.
A one-year extension lets the library (1) add the eight non-pilot branches as a comparison group, (2) separate lecture attendance from general visits, and (3) test whether the effect holds at lower-traffic branches by extending hours at one or two of them. This converts an ambiguous result into decision-grade evidence at low cost. The key reasoning: when evidence is suggestive but confounded, the right move is usually to buy better evidence, not to scale the intervention.