Order Submission Degrades After a Routine Release

The server metrics are the strongest clue: low CPU with very high latency means requests are waiting, not computing. That points at thread starvation around an external call, not at a slow query or a code-level loop.

Leading hypothesis: the timeout change from 10s to 45s converted a fast failure into a long block. Each Submit Order request now holds a front-end thread for up to 45 seconds waiting on SAP. Under normal order volume this saturates the worker pool, so unrelated screens also queue behind it. The five-minute timer competes for the same pool, which explains why the problem is intermittent rather than constant - it spikes when the timer and a traffic burst coincide.

Check in this order:

1. Service Center, Monitoring, Threads - are threads in a waiting state, and is the wait time consistent with the new 45s ceiling? This alone confirms or kills the hypothesis.
2. Integration logs / General Log filtered to the module - measure SAP response times directly for calls made before and after 14:00, isolating the deploy from SAP itself.
3. Slowest screens and screen timings - confirm the delay is concentrated in screen actions that call the integration, and that unrelated screens degrade only under load (the signature of pool exhaustion).
4. Timers - check the timer's duration and overlap at the time of the spike.
5. The site property's other effects - a timeout is sometimes the only thing that was hiding a long-standing slow SAP call.

Rule out the entity index as a cause: an index change does not produce multi-second waits with idle CPU. Do not spend the first hour in database query plans.

Everything hinges on whether the timeout change alone explains the regression, because that is reversible in seconds without discarding the release.

Immediate action: revert the site property to 10 seconds and re-measure for five minutes. If latency and error rates recover, you have both a mitigation and proof of cause. Keep the new screen live; marketing keeps its announcement and the cart abandonment stops.

If reverting does not recover it: the deploy likely reset integration connection state or the new screen calls the integration more often than the old one. In that case take the integration call out of the synchronous path rather than rolling back - for example, switch the screen action to enqueue the order and return immediately, accepting a brief degradation in order confirmation UX in exchange for a working site.

Present to the bridge:

• Latency before and after 14:00, plus the thread-state snapshot showing waits, not CPU burn.
• SAP response-time distribution from integration logs, showing SAP is slow but not slower than yesterday.
• The one-line configuration delta between releases.
• The finding that the 10s timeout had been silently absorbing this latency for months - the release exposed a pre-existing SAP latency problem rather than creating one.

Roll back only if the evidence shows the new screen drives materially more integration traffic or the entity change introduced blocking. Rollback also redeploys roughly 20 modules at peak hours, which carries its own risk and resets caches again. Do not recommend rollback as the reflexive option when a site-property revert is available and testable in minutes.

Fix the architecture first, then the observability that let this reach customers unnoticed.

1. Take the SAP call off the request thread. Screen actions must not block on a remote system with variable latency. Submit the order, persist it, return a confirmation with a status, and process the SAP posting asynchronously via a BPT or timer. This removes the coupling between SAP latency and portal capacity permanently.
2. Give the integration its own execution capacity. Move the timer's SAP traffic to a dedicated timer or separate module so batch work cannot starve interactive users. Add rate limiting and a circuit breaker so that a degraded SAP produces fast failures after a threshold instead of sustained 45-second waits.
3. Stop tuning timeouts as a fix. A 45s client timeout is a symptom mask. Define an explicit per-call budget and an explicit user-facing failure mode.
4. Alert on user-visible signals, not just infrastructure. Page on Submit Order p95 and error rate, not on CPU. The portal was visibly broken for two hours before anyone acted.
5. Use canary or blue-green style promotion for customer-facing changes, or at minimum deploy mid-morning with an engineer watching the dashboards for 30 minutes.
6. Treat site-property changes as production changes. They bypass build validation and are among the most common causes of unheralded behaviour shifts; require them in the change record.

Ordering matters: items 1 and 2 remove the failure class; 4 and 5 shorten the next incident regardless of cause.