Full-stack blind spot: fast API, slow checkout button
A retail bank's mobile web checkout shows a p75 Largest Contentful Paint of 4.8 seconds on 4G, while server-side traces show 180 ms p95 for the checkout API. The front-end team uses Real User Monitoring from 2% of sessions; the backend uses 10% head sampling. The page loads a third-party fraud SDK, a tag manager, and four marketing pixels. The CDN caches static assets for 24 hours, and a recent release changed the checkout page to load the fraud SDK synchronously before the payment form. Customer complaints mention a blank payment button for 3-6 seconds. The SRE team's dashboards show all services green, and the error budget is untouched. The bank has a change freeze in 10 days and must decide whether to roll back, change the SDK loading strategy, or add more backend instrumentation.
1. Diagnose why full-stack observability is failing here and identify the missing telemetry.
Read the worked answer
The observability is not full-stack; it is two disconnected stacks. The backend trace starts when the API receives a request, so it measures 180 ms of server work after the browser has already spent seconds parsing, executing third-party scripts, and blocking the main thread. The RUM sample at 2% may never capture the affected 4G sessions, and even when it does, there is no trace context linking the browser action to the API trace. The service dashboards are green because they measure availability and server latency, not user-perceived performance. The error budget is untouched because no errors occurred; the page was slow.
The missing telemetry is browser-side: resource timing for the fraud SDK, tag manager, and pixels; long-task and main-thread blocking data; JavaScript errors; LCP element attribution; and the release version. The team also needs CDN cache status and the HTML version served, because a 24-hour cache can keep an old synchronous loader alive after a fix. Synthetic checks from a throttled 4G profile would reproduce the blank button. Finally, the browser should inject traceparent into checkout API calls so a slow user session can be joined to the server trace. Without that, the bank cannot prove whether the delay is network, third-party script, CDN, or API.
2. Recommend the immediate action before the freeze and justify it with risk.
Read the worked answer
The immediate action should target the synchronous fraud SDK, because the timing aligns with the release and the symptom is a blank payment button while the API is fast. The lowest-risk fix is a feature flag that loads the fraud SDK asynchronously or after the payment form is interactive, with a timeout and a fallback path so checkout still works if the SDK is slow. If the fraud check must complete before submission, load it after first render but before the user can submit, and preconnect to its origin. This preserves fraud coverage while unblocking the main thread.
Rollback is the alternative if the team cannot safely change loading behavior within 10 days. Rollback is fast and reversible but may remove a fraud control or a compliance requirement, so it needs risk acceptance from the fraud owner. Adding backend instrumentation is not justified: the API is 180 ms and the delay is before or around the payment form. More backend spans would produce more green dashboards without improving the user experience.
Whatever is chosen, verify with a 4G synthetic check and RUM segmented by release and SDK version. A change freeze does not prohibit a targeted, flagged fix; it prohibits unverified change.
3. Define an SLO and alerting approach that would have caught this.
Read the worked answer
The SLO should be user-centric, not server-centric. Define a checkout performance SLO such as: 95% of mobile checkout sessions have the payment button interactive within 2.5 seconds and LCP within 2.5 seconds, measured at p75 from RUM. Add a second condition that the checkout API remains under 500 ms p95 so the SLO does not hide a backend regression. Segment by device class, network type, release, and CDN cache status; aggregate-only SLOs hide the 4G cohort.
Alert on multi-window burn: page when 5% of sessions exceed 4 seconds for 10 minutes, or when the fast-burn rate would exhaust 2% of the monthly budget in an hour. Use synthetic checks from throttled 4G every few minutes as a leading signal, and annotate deploys and feature-flag changes so the release is visible on the chart. Correlate RUM sessions to backend traces through traceparent where possible, and alert on third-party script duration separately so the fraud SDK owner is paged rather than the platform team.
The SRE error budget should include performance, not only errors. A service can be 100% available and still fail the user-perceived SLO, which is exactly what happened here.