The Data Analyst's Dashboard and the Appearance of Objectivity
A city council uses a dashboard to allocate youth-services funding across 12 suburbs. The dashboard ranks suburbs by 'need index', computed from reported incidents, service wait times, and population. A councillor notices that the two suburbs with the highest need scores have the fewest reported incidents, because residents there distrust police and rarely report. A staff analyst says the index is 'objective because it is quantitative'. A community group says the index systematically underfunds the suburbs that need most. The council has funding for 4 suburbs and must decide whether to keep the index, adjust it, or replace it with qualitative assessments.
1. Assess the analyst's claim that the index is 'objective because it is quantitative'.
Read the worked answer
The claim confuses measurability with objectivity. A quantitative index is objective only in the thin sense that its arithmetic is reproducible. Its inputs are chosen, weighted, and interpreted by people, and those choices embed value judgements.
The index uses 'reported incidents'. Reporting depends on trust in institutions. Where trust is low, incidents are under-reported, so the indicator measures reporting behaviour as well as underlying need. Using it as a need measure therefore introduces systematic bias against exactly the suburbs with least trust. The analyst's claim that the index is objective obscures this: the number looks neutral, but its construction is not.
Objectivity in a stronger sense requires transparency about choices, validation against independent measures, and acknowledgement of known biases. The index fails these. The analyst should say: 'the index is transparent and reproducible, but its indicators are proxies with known limitations', not 'it is objective'.
2. The community group claims the index 'systematically underfunds' the neediest suburbs. What would have to be true for this claim to be correct, and how could the council test it?
Read the worked answer
For the claim to be correct, three things must hold:
- Reported incidents understate need in low-trust suburbs relative to high-trust suburbs (differential reporting).
- The index weights reported incidents heavily enough that this understatement materially lowers the suburb's rank.
- No other indicator (wait times, population) compensates sufficiently.
The council could test this by:
- Comparing reported incidents with independent need measures (e.g., school welfare referrals, emergency-department presentations, survey-based need estimates) in each suburb.
- Conducting a sensitivity analysis: recompute the index with incidents excluded or down-weighted, and see whether the ranking of the four funded suburbs changes.
- Consulting residents in low-trust suburbs about barriers to reporting.
If the ranking flips under plausible alternative measures, the claim is supported. If it does not, the claim is weaker. The test is empirical and should be done before the funding decision, not after.
3. Recommend whether the council should keep, adjust, or replace the index, and justify your recommendation with explicit reasoning about trade-offs.
Read the worked answer
Recommendation: adjust the index rather than keep or replace it wholesale.
Reasons for adjusting: the index has real virtues - it is transparent, comparable across suburbs, and reproducible. Replacing it with purely qualitative assessment would lose comparability and invite discretionary bias, which may be worse. Keeping it unchanged ignores a demonstrated differential-reporting problem.
Adjustment should include: (i) down-weighting reported incidents or replacing them with a measure less sensitive to trust, such as survey-based need; (ii) adding an explicit correction for under-reporting where evidence supports it; (iii) publishing the weights and the sensitivity analysis so the value judgements are visible; (iv) reviewing the index annually with community input.
Trade-offs: adjustment is slower and requires better data; a corrected index may still embed assumptions. But these costs are lower than the cost of systematically underfunding the neediest suburbs, and lower than the cost of abandoning a comparable measure. The council should fund the 4 suburbs using the adjusted index, and document the change so the decision is defensible.