4.2 TB a Day, an Exhausted Quota and a PCI Obligation

Three layers, three different bills:

• Agent processing rules (logs_config.processing_rules with exclude_at_match / sample) run before the log leaves the host. The volume never reaches Datadog, so this is the only lever that reduces ingestion - the dominant cost.
• Pipelines and processors run in Datadog after ingestion. They tidy attributes and shrink indexed size, but the bytes were already shipped and billed.
• Index exclusion filters stop logs being indexed. That reduces the indexing bill, not the ingestion bill, and it is what most people reach for first.

So: scope an Agent exclusion rule to service:checkout-api, service:cart-sync, service:price-cache that drops DEBUG and keeps WARN/ERROR, plus a 1-in-100 sample rule so a statistical trace survives for the day debug output is the incident.

Guardrails: confirm the service attribute is resolvable at that stage of the Agent pipeline for these workloads - if not, match on the log path or a content regex. Test regexes against real lines in staging, roll out to 5% of hosts first, and watch the per-service ingestion graph for 24 hours. Do not blanket-drop DEBUG across the other ~110 services; that is where unknown unknowns live, and the audit says the problem is three services, not everything.

Archives are for the obligation; indexes are for the next two weeks of work.

• Configure a log archive to object storage containing all checkout-related service logs, with the bucket lifecycle policy aligned to 12 months (and note the lifecycle policy is the thing that actually enforces retention - Datadog stops managing the data once it is archived).
• Keep live indexes short, e.g. 15 days, so the monthly quota is never reached and logs stop disappearing mid-month.
• When audit or a postmortem needs an old window, rehydrate that time range into a rehydration index and query it there.
• Review the index daily quota setting itself: hitting it silently drops the rest of the day, which is precisely the failure described.

Be explicit about the trade-off. Archived logs are not full-text searchable until rehydrated. Rehydration takes minutes to hours depending on range size, costs money, and the rehydration index has its own retention. If legal needs sub-minute retrieval of an arbitrary day, that is a different and more expensive design than 'we can produce it within a business day'. Get that expectation agreed in writing before the first audit request.

Verify: archive a day's logs, rehydrate a 1-hour slice, and confirm the count matches the index count for the same hour.

Agent exclusion helps only where the rule matches, and DEBUG lines are not the only place a PAN can appear - error stack traces, request dumps and third-party callbacks leak too.

Do:

• Enable Sensitive Data Scanner with the built-in credit-card/PAN rules against the checkout log stream, action redact (or hash if you need to correlate the same card across events without exposing it).
• Confirm redaction applies to the archive as well, by enabling the archive redaction option - do not assume it is inherited. If it isn't, 12 months of unredacted card numbers sit in object storage, which is a worse finding than the one you started with.
• Scope the rules by service tag and pipeline. Scanning every event from every service is expensive and slows ingestion.
• Re-test the security detections: Cloud SIEM rules read the same processed logs, so verify redaction doesn't blind a rule that matches on PAN or BIN patterns.

Residual risk to state plainly: scanning happens inside Datadog after transmission, so the numbers left the host unredacted and crossed the network. If that is unacceptable for the PCI scope, the control of record has to be at the application or in Agent-side scrubbing - the scanner is a safety net, not a compliance boundary.