Simulation Testing
This domain tests how you evaluate a proposed decisioning change against historical data before it reaches customers.
What simulation does
- Replays a period of recorded interactions through an alternative configuration, such as changed arbitration weights, a revised strategy, a new action or an adjusted contact policy.
- Compares simulated results with what actually happened: propensity, accept rate, number of actions delivered, and value or revenue estimates.
- Runs without touching production decisions and without updating adaptive models; the models score with the state they had at the time.
- Requires that the data the alternative needs was actually captured historically; a new predictor that was never recorded cannot be simulated.
Worked example
A retail bank wants to know whether raising the value weight for card offers would improve revenue. The consultant copies the current configuration, raises the value lever, and simulates over 90 days of stored interactions. The simulation shows more high-value card actions delivered but a slightly lower overall accept rate and a net revenue increase. Reasoning: the comparison is simulated versus actual over the same historical window, so the deltas are attributable to the configuration change, not to a different customer mix.
Traps that cost marks
- Confusing simulation with champion/challenger or A/B testing, which run live traffic.
- Expecting models to learn during a simulation; they do not.
- Ignoring sample size or choosing a period with unusual traffic.
- Simulating a change that depends on data never captured in interaction history.
- Forgetting to compare against the actual baseline, so the result has no meaning.