The Recommendation Engine That Learned the Wrong Lesson
A district pilots a system that recommends practice problems to students. It was trained on historical data: which problems students attempted and which they completed successfully. After a month, teachers notice that students who were previously high-performing get harder and harder problems, while students who struggled get easier and easier ones, and the gap widens. The vendor says the model is 'just reflecting the data.' A teacher notes that the historical data came from a period when only advanced students were given the hard problems in the first place, and that 'completed successfully' was recorded only if a student submitted before a deadline that many students missed due to unreliable internet at home.
1. Explain how the training data can produce a widening gap even if the model has no explicit rule about student ability.
Read the worked answer
The model learns the pattern in the labels, and the labels are contaminated by two confounds:
- Selection in the inputs. Historically, hard problems were given mostly to advanced students, so 'hard problem → success' appears in the data mainly for strong students. The model learns that association and serves hard problems to students who resemble that group, reinforcing the pattern.
- A biased success label. 'Completed successfully' required submission before a deadline that some students missed for reasons unrelated to ability (unreliable internet). Those students are labeled failures, so the model learns to route them to easier content.
The feedback loop closes because the model's recommendations generate the next round of data. Nothing in the model needs a rule about ability; the data-generating process already encoded one, and the model faithfully copies it.
2. Propose one change to the data and one change to the system design that together would reduce the harm, and explain the limit of each.
Read the worked answer
Data change: separate mastery from access. Replace 'submitted before deadline' with a measure of correctness that is not confounded by connectivity (e.g., correctness on problems actually attempted, with missed deadlines excluded rather than counted as failure). This removes the label bias - but it does not remove the selection bias in which problems were offered historically.
Design change: ensure exploration. Deliberately serve some harder problems to all students regardless of predicted performance, and use those outcomes to update recommendations. This counters the self-reinforcing loop - but it costs some short-term accuracy and only helps if the new outcomes are recorded fairly.
Neither change alone is sufficient: the data fix addresses the label, the design fix addresses the feedback loop. Together they attack both mechanisms.
3. The vendor argues the system is not responsible because it 'only reflects the data.' Evaluate this argument.
Read the worked answer
The argument confuses describing a pattern with endorsing it. A model that reflects historical data also acts on that data, and its actions generate new data - so it does not merely observe the past, it shapes the future. Once deployed, the system is a cause, not a mirror.
Responsibility follows from the ability to choose: the designers chose the label definition, the training period, and the recommendation policy. Each is a design decision with foreseeable consequences. 'The data made me do it' is not a defense when the data's flaws were knowable and the system amplifies them.