OTCA: Span Metrics Cardinality Blows Up the Backend

Span Metrics Cardinality Blows Up the Backend

A platform team enables the Collector's spanmetrics connector to derive RED metrics from traces. They send everything - including http.route for a public API with 30k distinct paths (user IDs in the path), http.status_code, http.method, and tenant.id (bounded at ~500 tenants). Within a week the backend's metrics store rejects writes with 'too many series', and query latency for dashboards climbs from 1s to 30s. The Collector's memory also climbs steadily. The team's fix is to raise the metrics backend's series limit and add more Collector memory.

1. Compute/estimate the cardinality explosion and explain why raising limits and memory is the wrong fix. Identify which dimension is the primary culprit and why.

Read the worked answer

Cardinality math: spanmetrics emits, per (service, operation, status_code, method, tenant, ...), a calls counter and a duration histogram. Series count ≈ product of distinct values of each attribute, per metric.

http.route with user IDs in path: 30,000 distinct values. This is unbounded by nature - it is per request, not a route template.
http.status_code: ~5. http.method: ~4. tenant.id: 500.
Rough series for the calls counter ≈ 30,000 × 5 × 4 × 500 ≈ 300 million series. Even restricting to one service, the route dimension alone is 30k × the rest.

Why the limits/memory fix fails:

The series count is structurally too large; raising the backend limit just moves the cliff and increases cost nonlinearly. Dashboards still scan too many series → slow queries.
Collector memory grows because the connector holds aggregation state per series in memory; more series = more state, and more memory just delays OOM while worsening GC pressure.

Primary culprit: http.route is not normalized. The instrumented value is the raw path with user IDs, so every request creates a new series. The correct dimension is the route template (/users/{id}), which has a small fixed set. This is the single change that collapses 30k → ~dozens.

2. Design the fix: what to change at instrumentation, at the connector, and what to monitor. Include the trade-off of dropping `tenant.id` from metrics.

Read the worked answer

At instrumentation (root cause): emit http.route as the template, not the raw path. With auto-instrumentation, confirm the framework provides the route pattern (e.g. Spring @RequestMapping template, Express req.route.path). If the framework can't, use a span processor/attribute processor to rewrite the path to a template before the connector sees it. Keep the raw path out of metrics entirely (it may live on spans/logs if needed for debugging, subject to sampling).

At the connector:

Set dimensions explicitly to a bounded allowlist: http.method, http.status_code, http.route, and optionally tenant.id.
Drop unbounded attributes (user.id, raw url.path, http.target) from the connector's dimensions.
Consider histogram.explicit.buckets tuning to reduce histogram series cost.
Add a metricstransform/filter after the connector if any residual high-cardinality attribute slips through.

Trade-off on tenant.id: including it multiplies every series by 500. It is bounded (acceptable) but expensive. Options: (a) keep it only on a separate, coarser metric (e.g. per-tenant request count without route), or (b) drop it from span metrics and derive tenant views from traces/logs, or (c) keep it but accept the 500× multiplier and budget for it. State that 500 tenants × ~50 routes × 5 codes × 4 methods ≈ 500k series per metric - feasible but must be a conscious decision.

Monitor: otelcol_connector_spanmetrics_* series counts if exposed, Collector memory (otelcol_process_rss), and the backend's active-series gauge per metric. Alert on series growth rate, not just absolute count, to catch a new unbounded attribute early.