Insurance Migration: Is 1,100 Transactions per Minute Enough?
An insurance company is migrating a policy administration system to a new Java stack. Production statistics for the busiest hour show 12,000 policy lookups, 1,800 quote calculations, 300 endorsements, and 120 payments. Each quote calculation performs a 2.5-second mainframe call. The test team builds a script with 50% lookups, 30% quotes, 15% endorsements, 5% payments because those are the 'most important' user journeys. The test environment has 4 application servers versus 12 in production and a database with 10% of production rows, but the same schema and indexes. The test executed 2,000 virtual users for 30 minutes and achieved 1,100 transactions per minute with p95 of 4.2 seconds. The project manager wants to sign off because the target was 1,000 transactions per minute.
1. Should the project manager accept the 1,100 transactions/minute result as evidence that the 1,000 transactions/minute target is met? Justify your answer with the workload and environment facts.
Read the worked answer
No. The result is not representative enough to support sign-off, even though the raw number exceeds the target.
First, the workload mix is wrong. Production mix is 12,000 lookups, 1,800 quotes, 300 endorsements, and 120 payments out of 14,220 transactions: about 84.4% lookups, 12.7% quotes, 2.1% endorsements, and 0.84% payments. The test mix is 50% lookups, 30% quotes, 15% endorsements, and 5% payments. The test overweights quotes, endorsements, and payments and underweights lookups. Because each quote performs a 2.5-second mainframe call, the test may be heavier in one expensive path but lighter in the dominant lookup path. The direction of the error is not obvious.
Second, the environment differs: 4 application servers versus 12, and 10% of production rows. Throughput may be limited by the smaller application tier, or the smaller database may be faster due to cache and index effects. Neither is production-like.
Third, 'transactions per minute' is undefined. If a transaction is a business operation, the mix determines the cost per transaction. A 30-minute run is also short for steady-state confidence. The p95 of 4.2 seconds has no stated SLA. The PM should treat the result as a preliminary indication, not acceptance evidence, and require a corrected workload model and a defensible environment scaling position.
2. How should the team derive a representative workload model from the production statistics? Show the transaction mix and explain how think time and arrival rate fit in.
Read the worked answer
Derive the model from the busiest-hour counts, not from importance rankings.
Total = 12,000 + 1,800 + 300 + 120 = 14,220 transactions/hour.
- Lookups: 12,000 / 14,220 = 84.4%; rate = 200/min = 3.33/s.
- Quotes: 1,800 / 14,220 = 12.7%; rate = 30/min = 0.50/s.
- Endorsements: 300 / 14,220 = 2.1%; rate = 5/min = 0.083/s.
- Payments: 120 / 14,220 = 0.84%; rate = 2/min = 0.033/s.
- Total: 237/min = 3.95/s.
The model should preserve both the mix and the arrival pattern. Use production logs to determine session paths, inter-arrival times, and think times. For a closed model, calculate required virtual users with Little's Law: N = X × (R + Z). For an open model, generate arrivals at the measured rate and shape. Keep the 2.5-second mainframe call in the quote path, or virtualize it with the same latency distribution if the mainframe cannot be loaded. Define each business transaction separately, so a slow quote does not get hidden inside an average with fast lookups. Include peak-hour bursts rather than a flat average, and state whether the target is sustained or peak throughput.
3. Which environment differences can be compensated for and which invalidate the result? Recommend a risk-based decision.
Read the worked answer
The 4-versus-12 application-server difference can sometimes be compensated by scaling per-server results, but only if the application scales near-linearly and the per-instance configuration is the same. That assumption is risky: shared caches, load-balancer behavior, session affinity, and database contention often break linearity. A safer approach is to test at production scale, or at least at a scale whose bottleneck is the same tier.
The 10% database row count is harder to compensate. Query plans, index depth, cache hit ratios, and statistics can change materially with data volume. A smaller database can make the test look faster than production, or in some cases slower if statistics are skewed. Same schema and indexes are necessary but not sufficient. Use a production-sized or statistically representative data set, or test at both 10% and full size to understand sensitivity.
The 2.5-second mainframe call dominates quote response. If it is stubbed with unrealistic latency, quote results are invalid. If it is real but the mainframe is shared with production, the test may disturb production or be throttled.
Risk-based decision: do not sign off on the current evidence. Rerun with the corrected mix, a production-like data volume, and either full application scale or a documented scaling justification. If full scale is impossible, sign off only for a limited scope, state the residual risk, and add production canary monitoring with rollback criteria.