Milk collection forecast: a 0.96 R² that failed in the first live week
A WBCHSE vocational team built a model to forecast how many litres of milk a block-level dairy cooperative should collect each morning. Their table had 720 daily records from January 2022 to December 2023 with columns: date, litres collected, day of week, festival flag, rainfall and local temperature. Rainfall was missing on 61 days. The team filled every missing rainfall value with the overall column mean, shuffled the rows with a random 80/20 train-test split (random_state = 0), trained a random forest and reported R² = 0.96 on the test set. On that basis the cooperative agreed to use the model. In the first live week it over-predicted Monday collection by 18% and under-predicted the two festival days by about 30%. The chilling tank holds 5,000 litres; surplus milk spoils within a day and a shortfall means rationing to 40 village societies.
1. Identify the three workflow defects that best explain what happened, and link each defect to a specific symptom in the evidence given.
Read the worked answer
Defect 1: random shuffle on time-ordered data. Daily records are autocorrelated: 14 January and 15 January are nearly the same. A random 80/20 split puts days from the middle of the series into the test set, so the model is tested on days whose neighbours it has already seen. This is leakage, and it explains the gap between R² = 0.96 and the live errors: the score measured interpolation inside 2022-23, not forecasting an unseen tomorrow.
Defect 2: mean imputation of rainfall. Rainfall is not missing at random; it is missing on particular days, and rainfall is one of the few genuine drivers of collection. Replacing 61 values with one constant tells the model that those days were average, when they may have been the wettest. The model then has no way to lower the forecast on wet mornings, which is consistent with large errors concentrated on weather-driven days and on Mondays after a wet weekend. Computing the mean over all rows, including test rows, adds a second small leak.
Defect 3: metric and decision mismatch. R² rewards explaining variance across the whole year, dominated by the high-volume winter flush. What the cooperative loses is measured in litres per day, especially on the few festival days (roughly 15-20 in 720). With so few examples and a loss that pulls predictions toward the mean, the forest shrinks festival spikes, giving the 30% under-prediction. No naive baseline (same weekday last week, or last year same week) was computed, so nobody knew whether the model beat a wall calendar.
2. Rewrite the validation and deployment plan. Specify the split, the handling of rainfall, the metric, the baseline and at least two operating guardrails.
Read the worked answer
Split. Use a forward-in-time split: train on 2022, validate on the first half of 2023, test on the second half; then retrain on all data. Confirm with rolling-origin (walk-forward) validation rather than a single cut. Keep festival days in the test set deliberately and report them separately.
Rainfall. Do not use a column mean. At prediction time the cooperative only has a forecast, so train on forecast rainfall, not observed rainfall, or on an indicator ('heavy rain expected') plus the forecast amount. During training, impute with the previous day's observation and add a missing-indicator column. Drop any row where the target is missing.
Metric. Report MAE in litres and MAPE per weekday, plus a separate festival-day table. Report the maximum shortfall in litres, because the cost is asymmetric: an 8% over-collection spoils milk, a 8% shortfall breaks commitments to 40 societies. Consider pinball loss at the 0.6 quantile so the model aims slightly high on ordinary days and clearly high before festivals.
Baselines. Compare against (a) last year same weekday, (b) mean of the last four same weekdays. If the model does not beat both on MAE, it is not ready.
Guardrails. (1) Clamp the daily order to within plus or minus 10% of the seasonal-naive value until eight consecutive weeks show MAE under 4%; anything outside the band is reviewed by the collection officer. (2) Publish a prediction interval, not a point number, so the officer orders the upper bound before a festival. (3) Log every forecast, the actual, and the override reason; retrain quarterly and re-run the walk-forward test before each retrain.
3. The secretary asks: 'The computer said 96 out of 100. Why should I not trust that number?' Write the short answer you would give, in plain language, including what the team will do next.
Read the worked answer
Say this: 'That 96 was measured on days the model had effectively already seen. We shuffled two years of daily records before splitting them, so the test days sat next to their own neighbours in the training set. It is like marking a student on questions taken from the same page he just read. The live week is the first honest test, and it says our Monday figure is 18% too high and our festival figure 30% too low.'
Then state the concrete change, because a plain-language answer without a fix is just an apology. 'We will retrain so that the model is always tested on months that come after the months it learned from, we will use the rainfall forecast instead of a monthly average, and we will judge it in litres per day against a simple rule: last year, same weekday. Until it beats that rule for eight weeks, the officer keeps final say and no order moves more than 10% from the simple rule.'
Add the cost framing so the number is understood in the cooperative's units: a 5,000-litre tank means a 10% over-order on a 4,200-litre day is roughly 420 litres at risk of spoilage, which is more informative than any R². Finish by offering the walk-forward table for the last three months, so the secretary can see the errors himself rather than take a single figure on trust.