Model Validation After Telework and a New BRT Line
A regional travel demand model was calibrated to a 2015 household travel survey and 2018 traffic counts. Post-pandemic, observed AM-peak vehicle trips at 12 screenlines run 9% to 14% below model estimates, while core-area transit boardings are 6% above them. An 8-mile bus rapid transit line opened 14 months ago; the model forecast 12,400 daily riders and observed ridership is 9,100. The MPO must certify the model for its 2050 plan, and a consultant has proposed simply re-validating to 2019 conditions because 2019 is the last "normal" year.
1. What are the plausible sources of the two errors, and how would you test each?
Read the worked answer
The two errors point in opposite directions, which is diagnostic: the model is probably overgenerating peak-period auto travel while underperforming on transit attractiveness in the core.
Highway overprediction candidates
- Trip generation rates carried over from a 2015 survey that did not reflect current telework; test by comparing observed work-trip productions from a current survey or employment/payroll data against model productions by district.
- Peak-period assignment assuming pre-pandemic temporal profiles; test with time-of-day count profiles.
- Land-use inputs that lag actual development or occupancy.
Transit shortfall (9,100 vs 12,400) candidates
- Network coding: BRT speeds, dwell time, signal priority, or headways coded more favorably than delivered. Compare coded run times to AVL data - this is the most common and most fixable cause.
- Transfer penalties and fare assumptions.
- Station access/walk catchment overstated.
- Demand suppression from perceived reliability, safety, or cleanliness that a utility-based model does not capture.
Test protocol: hold out a subset of screenlines and stations, compute GEH and percent-difference against acceptance thresholds, run a before/after comparison on the BRT corridor with a parallel control corridor to isolate corridor-specific from regionwide effects, and back-check boarding counts by station rather than only the line total. Station-level error patterns (uniform vs. concentrated at transfer points) distinguish a coding problem from a behavioral one.
2. How should the BRT forecast error be treated in the 2050 plan?
Read the worked answer
The line has 14 months of data, which is enough to detect bias but not enough to fully re-estimate a model. Treat it as evidence to be quantified, not as a reason to abandon modeling.
Steps:
- Diagnose before adjusting. If the error is coding (run times, headways), fix the network and see how much ridership gap closes before touching behavioral parameters. Adjusting parameters to compensate for a coding error corrupts the model for every other corridor.
- If the residual is behavioral, apply a documented, bounded calibration factor and state its basis - for example, an incremental logit or elasticity adjustment derived from the observed ridership ratio, with a confidence range.
- Carry the uncertainty forward. Report the 2050 ridership for other planned transit projects as a range that reflects the observed forecast bias, not as a single point. If the model overpredicted by roughly a quarter on a well-specified BRT project, planning partners are entitled to know that.
- Do not re-validate to 2019 alone. A 2019 base year will reproduce a world that no longer exists and will re-import the very peak-period bias now visible. Re-validate to 2019 and to current conditions, and document the difference as a scenario rather than concealing it.
3. The consultant argues that because the model meets screenline criteria overall, certification should proceed. How do you respond?
Read the worked answer
Meeting aggregate criteria is necessary but not sufficient. A model can pass overall screenline thresholds while being systematically wrong in the direction that matters for the decisions the plan will support.
A defensible position for the MPO:
- Ask for segmented performance. Report validation statistics by facility type, by area type, and by time period, not just pooled. A 9-14% underprediction concentrated on radial freeways is a planning-relevant bias even if the pooled statistic passes.
- Ask for transit validation separately. Most highway-oriented validation protocols say nothing about boarding accuracy; the BRT error demonstrates why a transit-specific check is needed.
- Insist the limitation is disclosed. Certification with documented, quantified limitations is normal practice. Certification with a known directional bias left unmentioned is not.
- Tie certification to a corrective action plan with milestones before the plan's adoption and before the next project-level forecast.
The planner's job is not to block certification; it is to ensure the decision-makers who rely on the model know what it is and is not good at. That is a professional judgment the MPO should own rather than delegate to the consultant's acceptance criteria.