A difference-in-differences with pre-trends that disagree
A business-support programme was rolled out in 2022 to firms in 6 of 12 districts, chosen by an administrative decision that analysts suspect was correlated with local economic conditions. Firm-level data are available for 2019, 2021 and 2023. Mean annual profit, in thousands of euros, is: programme districts 40 (2019), 46 (2021), 55 (2023); comparison districts 30 (2019), 32 (2021), 38 (2023). There are about 4,000 firms, roughly 340 per district. A colleague has reported a difference-in-differences estimate using only 2021 and 2023 and describes it as 'the programme effect'. A second colleague proposes adding a control for whether a firm received a follow-on export grant in 2023, arguing that this 'absorbs other shocks'. The ministry wants a number it can put in the budget note, and the permanent secretary has asked whether the estimate is robust enough to defend in parliament.
1. Compute the available difference-in-differences estimates and explain why they differ. What does the spread tell you about the identifying assumption?
Read the worked answer
The three estimates.
- 2021 base: (55 − 46) − (38 − 32) = 9 − 6 = +3.0.
- 2019 base: (55 − 40) − (38 − 30) = 15 − 8 = +7.0.
- Two-way fixed effects with all three years effectively compares post to the average of the two pre-periods: treated pre-mean 43, comparison pre-mean 31, so (55 − 43) − (38 − 31) = 12 − 7 = +5.0.
Why they differ. Parallel trends requires the change in the untreated outcome to be the same in both groups. In the pre-period the programme districts grew by 6 (40 to 46) while comparison districts grew by 2 (30 to 32). Trends were already diverging before any treatment, so the assumption is violated on the face of the data. Choosing the base year chooses how much of that divergence is loaded into the estimate: the 2021 base attributes the 2019–2021 divergence to a pre-existing difference and leaves 3; the 2019 base attributes it to the programme and yields 7.
What the spread means. An estimate that moves by a factor of more than two depending on a defensible specification choice is not robust. The honest statement is that the data cannot separate the programme from a pre-existing growth differential, and the number quoted should be a range with the assumption stated, not a point estimate. The budget note should say 3–7 with the lower end preferred, or withhold a number pending better design.
2. Evaluate the proposal to control for receipt of a follow-on export grant in 2023. What would you advise instead?
Read the worked answer
Why it is a bad control. The export grant is measured after the programme began. If the programme increased firms' capacity or credibility, it may have caused grant receipt, so conditioning on it removes part of the effect being estimated (over-controlling). If instead both the grant and profits respond to unobserved firm quality or to the same local demand shock, the grant is a collider or a proxy for the confounder, and conditioning induces bias of unpredictable sign. Either way, the coefficient on the programme becomes a conditional effect within grant-recipient and non-recipient strata, which is not the policy parameter the ministry wants.
What to do instead. (1) Treat the grant as an outcome in a separate regression, to describe a mechanism, not as a control. (2) Model the confounder directly if it is observable: local GDP growth, sector composition, pre-period firm size and age, and baseline profit trajectories. (3) Use a design that does not rely on parallel trends across all 12 districts: an event-study with several pre-periods and formal tests of pre-trend coefficients, a synthetic control built from the six comparison districts, or a matched DiD on pre-period growth. (4) If the administrative selection rule is known (for example a scoring threshold), exploit it with a regression discontinuity rather than comparing districts.
Advice. Report the mechanism analysis separately and pre-register the specification. Coefficients from a specification chosen after seeing the results will not survive scrutiny in parliament.
3. The treatment is assigned at district level and there are 12 districts. Assess the inference problem and propose concrete remedies.
Read the worked answer
The problem. With treatment assigned at the district level, errors within a district are correlated: firms in the same district share local demand, local government, and the same shock. Treating 4,000 firms as independent observations understates standard errors, often by a factor of two or more, and produces confidence intervals that are far too narrow. Clustering at the district level is the minimum correction, but 12 clusters is small: cluster-robust standard errors are downward biased and the t-distribution approximation is poor, so a nominal 5% test can reject far too often.
Remedies. (1) Wild cluster bootstrap with Rademacher weights, imposing the null, which performs much better with fewer than 20 clusters. (2) Randomisation inference: with six treated districts out of twelve there are C(12,6) = 924 possible assignments, so the exact p-value can be computed by re-assigning treatment labels and comparing the observed estimate with the distribution. This does not require large-sample approximations and is the most defensible option here. (3) Report randomisation-inference confidence intervals by inverting the test. (4) Be transparent about the effective sample size: the design has 12 units of treatment variation, not 4,000, and precision cannot be manufactured by adding firms.
Presentation. Report the point estimate with the randomisation p-value and a confidence set, and state explicitly that the design is powered to detect only large effects. With a standard deviation of district-level shocks unknown, a power calculation should be run before any claim of 'no effect' is made.