Scaling Broker Scorecards Without Overclaiming Causality
A commercial insurer gets 70% of GWP through 1,200 brokers. Eight months ago it launched a portal with monthly broker scorecards showing quote-to-bind, retention, and cross-sell. Only 35% of brokers are monthly active. Observational data show active brokers have 18% higher retention and 12% higher quote-to-bind than inactive brokers. The distribution head wants to mandate the portal and tie broker bonuses to scorecard use. Large brokers say the data is theirs and want opt-in. The insurer has no new IT budget and a compliance review pending on sharing broker-level performance data.
1. What is wrong with using the 18% and 12% differences to justify a mandate? How would you test causality?
Read the worked answer
The comparison is selection-biased. Active brokers may already be larger, more digital, better at retention, or serving better risks. The 18% and 12% differences may reflect broker quality, not portal use. Mandating the portal based on that comparison risks a costly program with no causal lift.
Test causality with a randomized invitation to the portal at the broker level, or a matched difference-in-differences design if randomization is impractical. Pre-register metrics: activation, quote-to-bind, retention, premium, and business mix. Compare invited versus not-invited brokers over at least six months. If large brokers cannot be randomized, analyze them separately and do not extrapolate.
2. Design a rollout and incentive plan that respects broker data concerns and avoids gaming.
Read the worked answer
Start with a voluntary opt-in pilot, randomized at broker level, with aggregate benchmarks and minimum cell sizes so no broker sees another broker's data. Use a data-sharing agreement and compliance sign-off before sharing broker-level performance. Give large brokers a separate pilot track with their own data controls.
Do not tie bonuses to scorecard use until causal lift is shown. Instead, reward participation in the pilot and use scorecards for coaching. After the pilot, tie incentives to outcomes such as retention or quote-to-bind, not to portal logins. Monitor for cherry-picking, mispriced risks, and brokers abandoning difficult segments.
3. What would make you stop the program?
Read the worked answer
Stop or redesign if the randomized pilot shows no statistically credible lift in quote-to-bind or retention after six months, if broker attrition rises materially, or if the compliance review cannot support the data-sharing model. Also stop if the program causes adverse selection or if large brokers credibly threaten to move profitable business.
The decision should be based on causal impact and broker economics, not on portal activity alone.