Can We Say the Campaign Caused 12%?
Daniel's capstone team analysed 18 months of weekly sales for a mid-size retailer. In the eight weeks after a March email campaign, weekly sales were 12% higher than in the previous eight weeks. The campaign went only to loyalty program members in 3 of the retailer's 9 regions, and the retailer launched a sitewide 10% discount in the same week. The sponsor wants the board slide to read 'the campaign increased sales by 12%' and needs it Friday. Meanwhile the analysis is fragile: the notebook hardcodes C:/Users/daniel/Desktop/sales_final_v3.csv, there is no random seed anywhere, one teammate cleaned the region names by hand in Excel without saving the mapping, and the charts were pasted into slides as screenshots. A reviewer will try to rerun everything in three months.
1. The sponsor's requested wording is not supportable. Explain what the analysis actually shows, propose a stronger design that could be run with the available data, and draft the sentence you would put on the slide.
Read the worked answer
What the current analysis shows. A before-and-after comparison with no control group. Weekly sales rose 12% after the campaign; that rise is consistent with the campaign, with the concurrent 10% sitewide discount, with seasonality (March is often a promotional month), with a regional mix shift, and with ordinary noise in eight data points. It is not evidence of cause. The design has three defects: no counterfactual, a co-occurring treatment, and a pre-period chosen because it was convenient.
A stronger design the data supports. The campaign ran in 3 of 9 regions - that is a natural comparison group. Run a difference-in-differences: change in the 3 treated regions minus change in the 6 untreated regions, same pre/post windows. This removes any shock common to all regions, including the sitewide discount, which is the largest confounder. Then check the parallel-trends assumption by plotting the pre-period series for both groups; if they were diverging before March, the estimate is not credible. Consider matching regions on baseline sales and store count, and run a placebo test on an earlier, campaign-free month to see whether the method produces spurious 'effects'.
Slide sentence. 'Sales in campaign regions rose 12% after launch; comparable non-campaign regions rose X%, implying an estimated campaign effect of Y% (95% CI ...). The campaign coincided with a sitewide discount, so this is an association under a parallel-trends assumption, not a proven causal effect.' If the diff-in-diff estimate is small or negative, say so - that is the useful answer.
2. Enumerate the reproducibility defects and specify the concrete changes needed so a reviewer can rerun the analysis in three months.
Read the worked answer
Defects and fixes, one for one:
- Hardcoded absolute path. Move to a relative path resolved from the project root, or a small
config.yamlwith adata_dir. The reviewer's machine has noC:/Users/daniel/Desktop. - No random seed. Set seeds for every stochastic step (
random, NumPy, and any model). Better, seed once in a shared module so it cannot drift between notebooks. - Manual Excel cleaning with a lost mapping. This is the most serious one: an undocumented transformation means no one can reproduce the region names, and the diff-in-diff depends on region assignment. Recover the mapping, commit it as a CSV, and apply it in code with an assertion that every source value maps to exactly one target value.
- Screenshots in slides. Generate figures from code, save as vector files into an output directory, and reference those files - so the chart in the deck is provably the chart the code produces.
- Missing environment. Pin dependencies in
requirements.txtorenvironment.yml; unversioned library upgrades silently change groupby and plotting defaults.
Project hygiene. Put everything in Git with meaningful commits; add a README stating run order and the expected final numbers; keep raw data immutable and write intermediate outputs to a derived/ folder so each stage is inspectable. A reviewer should be able to run one command and see the same 12%, or see it change and know exactly why.
3. The board audience is non-technical and time-poor. How should Daniel present the estimate and its uncertainty without either overclaiming or burying the finding?
Read the worked answer
Lead with the decision, not the method. The board is deciding whether to fund another campaign. The slide's headline should be the best estimate and its range in plain language - 'Campaign regions grew about Y% more than comparable regions; the plausible range is A% to B%' - with the treated-versus-comparison chart directly beneath it, two lines, same axis, a vertical marker at launch. One chart, one sentence, one caveat.
Handle uncertainty by showing it, not by hedging in prose. A range plus the comparison lines does the work that a paragraph of caveats cannot. Avoid p-values and the word 'significant' - in a boardroom it means 'large', not 'unlikely under the null'. If asked 'could it be noise?', the honest answer is the interval and the placebo test result.
Put the caveats where they belong. The sitewide discount, the parallel-trends assumption, and the treatment assignment belong in a footnote and a backup slide, not in the headline - but they must exist, because the first question a sharp board member asks is 'what else happened in March?'. Prepare that answer before the meeting.
Say what would settle it. Recommend that the next campaign be launched in a random subset of eligible regions with the discount held constant, and state the measurement plan. Offering a cheap way to get a cleaner answer is more persuasive than defending a weak one, and it converts a fragile capstone result into a decision the board can act on.