Fare Evasion Risk Scoring at a Transit Agency
A metropolitan transit agency's revenue protection team proposes buying a vendor's "fare evasion propensity" score to prioritize fare inspections. The score uses trip history, time of day, station entries, payment method, and a "neighborhood context" input derived from census tract. In a pilot on three lines, inspector teams targeted by the score issued 2.4x more citations per inspector-hour. However, 71% of citations went to riders from the six lowest-income ZIP codes in the region, which contain 29% of ridership. Fare evasion is estimated at $28M a year against a $1.9B operating budget. The vendor's validation used prior citations as the label. The board wants a recommendation in 30 days, and the contract contains a renewal clause.
1. Assess whether the model is valid for the purpose it is being sold for. Address the label and the features separately.
Read the worked answer
The label first. Prior citations are the output of enforcement, not of evasion. Training on them teaches where inspectors have been, and then the score sends inspectors to the same places - a self-reinforcing loop that will look more accurate every quarter while measuring only its own history. This is construct invalidity before it is anything else. The 2.4x citation lift is therefore not evidence of predictive skill; it may be evidence that the model efficiently re-finds the same riders.
The features second. Census-tract "neighborhood context" is, in most U.S. metros, a direct proxy for race and income. Including it means the allocation of enforcement is driven by where a rider lives, whatever the stated intent. Time-of-day and station entries are defensible only if linked to observed evasion rather than to patrol patterns.
What validity would require. An enforcement-independent target: evasion measured by random proof-of-payment audits, fare-gate sensor counts, or controlled surveys. Features should be rider behaviors plausibly connected to the act of non-payment - not residence. Until that exists, the vendor has shown a model can predict citations, not that it can find evaders.
2. Design the equity analysis you would run. Which results would change your recommendation, and which would not?
Read the worked answer
Specify the analysis before seeing the results, so the results cannot pick the standard.
Measures. By score decile and by census-tract income quintile: inspection rate, citation rate, and - critically - the hit rate (citations per inspection). If the score is well-calibrated across groups, hit rates should be similar at similar scores. A systematically higher citation rate among low-income riders at the same score implies either miscalibration for that group or enforcement intensity inflating the label.
Also compute the demographic composition of the top-k inspection list against ridership share, and benchmark against a random-inspection sample.
Results that would change my view. A calibration gap across income groups after controlling for observed evasion; or a top-k list concentrating on tracts holding 29% of ridership with no correspondingly higher measured evasion. Either would make the disparate-impact objection concrete rather than speculative.
Results that would not settle the matter. Equal hit rates and proportional allocation would remove the statistical objection but leave the purpose question - whether non-payment should be policed person-by-person at all - which is a policy judgment the board must make explicitly, not one the analysis can make for it.
3. The board wants a recommendation in 30 days and the vendor contract is up for renewal. What do you recommend, and how do you handle the pressure?
Read the worked answer
Recommend against deploying the score for punitive enforcement in its current form, and deliver that in writing with the reasoning and the alternatives attached.
The obligations are concrete. Competence: the model does not measure what it is sold to measure. Avoiding harm: enforcement exposure falls hardest on people least able to absorb fines. Transparency: riders cannot contest a score they cannot see. None of these is satisfied by a 2.4x citation lift.
The constructive counter-proposal. Use analytics on the system rather than the rider: deploy inspectors to stations with high evasion measured by sensors or surveys, invest in fare gates and in fare-capping or means-tested discount programs, and decriminalize marginal non-payment where the cost of collection exceeds the loss.
If the agency proceeds anyway. Require an independent bias audit, targeting at the station level rather than the person, human review before any citation issues, demographic monitoring with published results, and a sunset clause.
On the contract. Flag that renewing during an unresolved validity question is a governance risk. Recommend the 30 days be used for a scoping and audit decision, not a signature - and put that advice in the record, because the pressure to sign will outlast the meeting.