Green Synthetic Tests, a 14% Conversion Drop in Brazil

An API test proves a URL answers from a cloud region with a fixed request shape. It never runs JavaScript, never loads the third-party payment SDK, never uses a Brazilian mobile network, and never runs on a slow device. Green means the endpoint is up; it says nothing about whether checkout works.

Look in this order:

1. RUM sliced by country. Checkout view filtered to Brazil against the rest of the world. Inspect loading_type and resource errors for the payment SDK's domain - a third-party JS failure that raises an error and never resolves the form matches 'spinning' exactly.
2. The funnel. View of checkout → click 'Pay' → confirmation. Where the drop-off appears localises the failure to render, click handler, or network.
3. Session replay filtered to Brazilian sessions with errors, to see the actual UI state.

Then add a browser test covering the real checkout flow, including third-party JS. API tests structurally cannot catch this class of failure, so no amount of tuning their regions or frequency will. If the affected user base isn't covered by a public location, run that browser test from a private location in the region or from CI in a service running there.

Separate the two knobs. RUM session collection (views, actions, errors, resources) and session replay sampling are configured independently, and they are usually conflated into one 'RUM is expensive' complaint.

• Keep session collection at 100% during the investigation - it is cheap relative to replay and it is what gives you per-country and per-ISP slices.
• Drop session replay to 10-20% overall, and use replay retention filters so sessions with errors or frustration signals (rage clicks, dead clicks, error clicks) are kept. For a spinning form, the error-click and dead-click sessions are the ones you need.
• If Brazil volume is high and replay is the only way to see the bug, raise replay sampling temporarily rather than the global rate, investigate for a week, then drop back. Sampling is per-application, so this is a change you can make and reverse.
• Longer term, instrument the payment SDK call as a custom action with a timeout threshold so 'SDK took more than 8s' becomes a metric you can alert on, instead of a replay you have to watch.

Then re-audit: which applications actually consume replay volume, and do web and mobile need the same rate? Most RUM bills are set once at adoption and never revisited.

Public synthetic locations cannot reach an internal domain - egress is blocked and DNS resolves internally. Deploy a private location: run the private location container inside the same network, with outbound HTTPS to Datadog and reachability to staging, then attach the browser test to it.

Judgement calls:

• A private location is infrastructure you own and must monitor. An unmonitored private location that dies produces 'no data' alerts, not test failures, and the difference is easy to miss at 3am. Run a second runner in another availability zone and alert on location health separately.
• Test failures must be distinguishable from staging outages. Keep an API-level smoke test alongside the browser test so two independent signals disagree before you page anyone.
• Credentials belong in secure global variables, not inline in the test script, and the staging test should run at a lower frequency than production - it is a pre-merge gate, not a customer-facing SLI.
• Don't make the private location a production SLI source unless it sits in the same failure domain as production; otherwise you have built a monitor that measures your VPN.