Scaling mainframe automation causes timeouts and memory growth
A logistics company uses Pega Robotics unattended robots to process 3,000 shipment updates per hour against an IBM mainframe emulator. The solution ran on four VMs with five robots each and met the target initially. After adding ten more robots to handle peak season, throughput fell to 1,800 per hour, 25% of transactions timeout, and memory usage grows by about 300 MB per robot per hour. Operators report that emulator sessions remain open after a robot finishes a batch, and some sessions are reused by the next robot with the previous user's credentials. The mainframe team sees a rising number of orphaned sessions and a licensed session limit of 120. The infrastructure team proposes adding eight more VMs. The business needs 4,000 per hour for the next six weeks and cannot change the mainframe application.
1. What is your diagnosis, and what evidence would you collect?
Read the worked answer
The pattern points to session lifecycle failure, not CPU shortage. Memory growth of 300 MB per robot per hour, orphaned sessions, and session reuse with the previous user's credentials all suggest emulator sessions are not being closed and objects are not released. The 25% timeout rate and the licensed session limit of 120 reinforce that. I would collect per-robot session counts over time, heap and handle counts, runtime logs with session IDs, mainframe orphan reports, and transaction durations. I would test one robot with explicit session close and compare memory after 100 transactions. I would also check whether the added robots increased concurrency beyond what the mainframe or emulator license can support. Adding VMs would not fix a leak; it would create more orphaned sessions and hit the license ceiling faster. The diagnosis should be proven with a controlled run before changing architecture.
2. What architecture and runtime changes would you make?
Read the worked answer
Make session ownership explicit. Each robot should create its own emulator session with its own credential from the vault, and close it in a guaranteed cleanup block even on exception. Do not reuse a session across robots or leave it for the next batch. Add health checks that detect orphaned sessions and terminate them safely. Introduce a session pool with a hard maximum based on the 120-session license, and throttle work accordingly. Set memory and handle thresholds that trigger a graceful restart between batches. Review the global container and any shared objects for references that keep sessions alive. Load-test at 4,000 per hour with the new lifecycle before peak. The target is not just more robots; it is a bounded, observable session model. If the emulator supports it, use unique session IDs and log them for reconciliation.
3. How would you respond to the infrastructure team's proposal to add eight VMs?
Read the worked answer
I would explain that the bottleneck is likely session leakage and concurrency, not VM capacity. Adding eight VMs could increase parallel sessions and make the orphan problem worse, while the license limit of 120 caps useful concurrency. Instead, propose a short proof: run the session cleanup and pooling change on the existing four VMs, measure throughput, memory, and orphan count for 24 hours, then decide on capacity. Build a capacity model from transaction duration and licensed sessions: if each transaction holds a session for 30 seconds, 4,000 per hour needs about 34 concurrent sessions plus headroom, which is within 120 if sessions are released. If the pilot shows a real CPU constraint, add VMs after the leak is fixed. This response respects the infrastructure team's concern but avoids scaling a defective design.