Bigtable Hotspot: 180,000 Writes/s Into One Row-Key Range

This is a row-key design problem, and more nodes will not fix it. Bigtable distributes data by row-key range across tablets; load balancing follows key boundaries. Every write for a device lands in one contiguous lexicographic region, and the busiest customer's keys cluster together, so one tablet (and effectively one node's worth of work) absorbs 62% of ingest while 11 other nodes idle. Doubling nodes adds capacity to ranges that are already at 20% utilization.

The proof is in the visualizer: a single dark band with idle neighbours is the signature of a hot contiguous range, not of insufficient total capacity. The first question in any Bigtable capacity incident is 'is the load spread across keys', not 'how many nodes do we have'.

What doubling does buy: nothing for throughput, and it doubles cost. It may briefly reduce latency as the hot tablet gets scheduled onto less contended hardware, which is exactly the kind of false positive that leads teams to keep adding nodes.

What must change: the key. Two options, often combined:

1. Promote a high-cardinality field to the key prefix. If metrics carry a metric-name or shard identifier, key on that first (shard#deviceID#reverseTimestamp) so writes spread across many ranges. Choose a field with high cardinality and even distribution, not something correlated with device ID.
2. Salt the prefix with a small hash bucket count (e.g., 16-64 buckets of hash(deviceID) % N). This spreads the hot customer across buckets. Cost: a latest-100 read now requires N parallel prefix scans merged client-side, which raises read amplification by roughly N. Size N as a deliberate tradeoff - 16 buckets typically fixes a hotspot without destroying read patterns; 256 does not.

Also verify the write pattern: 180,000 writes/s of single rows are fine for Bigtable, but per-row write cost rises with row size. Batch writes per device per interval into one row per timestamp bucket if the reads can tolerate it - this cuts write operations dramatically.

Use Bigtable multi-cluster routing with clusters in at least two regions, and be explicit that replication is eventually consistent. Multi-cluster routing sends writes to a single cluster for a given row key (so writes remain consistent) while reads can be served from the nearest cluster. For 'latest metrics for this device' - a read that must reflect recent writes - use single-cluster routing for that read path or accept bounded staleness; for 30-day dashboard scans, nearest-cluster reads are fine.

Sequence the work so capacity and keys change in a controlled order:

1. Re-key the table with the salt/shard prefix into a new table. Do not attempt to mutate keys in place - Bigtable keys are immutable; you are writing a new table and backfilling.
2. Dual-write from the ingest service to old and new tables for a validation window; compare latest-100 results for a sample of devices between the two.
3. Cut reads over, then writes, then delete the old table after the retention window.
4. Re-measure CPU per node before buying nodes. With the load spread, the same 12 nodes may handle 180,000 writes/s comfortably; buy only what the measurement demands.

Hitting the read targets:

• Latest-100 (8,000/s): with a salted key this becomes N parallel prefix scans; cap N and merge in the client, cache the last result per device for a few seconds if the dashboard tolerates it, and use a row-key layout that makes the newest entries contiguous within each bucket.
• 30-day dashboard scans (40 s): this is the wrong workload for Bigtable. Range scans across 30 days for a fleet are an analytical query. Export metrics to BigQuery (Bigtable change streams → Dataflow → BigQuery, or batch export to GCS/Parquet) and let dashboards query partitioned, clustered BigQuery tables. Keep Bigtable for point lookups and recent windows only. That removes most of the scan load from Bigtable entirely and makes the capacity question much easier.

Monitoring: treat key distribution as a first-class SLI.

• Per-node CPU and per-range utilization, not just cluster-average CPU. A healthy cluster has a reasonably flat distribution; alert when any node exceeds 2x the cluster median for 15 minutes. Cluster averages hid this incident for weeks.
• Write latency p99 per row range - the hot range degrades first.
• Hot-key cardinality: the ingest service should emit the top-N most frequent key prefixes to Cloud Monitoring. A shift in prefix concentration is the leading indicator, and it is cheap to produce.
• Tablet count and split behavior for the hottest ranges; repeated splits that immediately re-saturate mean the key space, not the node count, is the problem.
• Per-tenant ingest rate if the product is multi-tenant: a single customer at 62% of writes is a capacity and fairness risk regardless of key design, and that customer should be visible in a dashboard before it becomes an outage.

Ingestion design changes:

• Aggregate before writing where the read pattern allows: buffer 2 seconds of metrics in the ingest tier and write one batched row per device per interval. Fewer, larger writes reduce per-write overhead and node CPU, and the latest-100 read pattern is barely affected.
• Enforce key hygiene in code, not in a wiki. The row-key builder should be a shared library that applies the salt automatically, so a new service cannot reintroduce a prefix hotspot. Include a unit test asserting key distribution over a synthetic device population.
• Per-tenant admission control: rate-limit or isolate the noisiest tenant with a dedicated table or cluster if the contract requires it, so one customer cannot consume the shared cluster's write budget.
• Retention discipline: Bigtable is expensive per GB. Set garbage-collection policies on column families so the 30-day window does not become 3 years, and keep long-horizon history in BigQuery/Cloud Storage where it belongs.