AWS Developer: Throttling at 40% of Provisioned Capacity

Throttling at 40% of Provisioned Capacity

A telemetry pipeline writes to a DynamoDB table events with partition key deviceId and sort key ts. Provisioned capacity is 2,000 WCU with auto scaling set to a 2,000–8,000 range. About 50,000 devices report every 30 seconds. During a firmware rollout, 300 devices begin sending every 50 milliseconds instead of every 30 seconds. CloudWatch shows ThrottledRequests at 3–8% on the table while ConsumedWriteCapacityUnits sits near 40% of provisioned - yet auto scaling never scales out to the 8,000 maximum. The AWS SDK retries (default, 3 attempts) turn 200 ms writes into a 2-second p99, and the ingestion Lambda hits timeouts. That Lambda also runs at 900 concurrent executions during peak against an account limit of 1,000.

1. Why is the table throttling when it is only using 40% of its provisioned write capacity, and why does auto scaling stay flat?

Read the worked answer

This is a hot-partition problem, not a table-capacity problem. DynamoDB divides a table into partitions, and a single partition is capped at roughly 1,000 WCU and 3,000 RCU. All writes for one deviceId land on one partition. When 300 devices jump from 1 write per 30 s to 20 writes per second, a handful of partitions need far more than 1,000 WCU while the table as a whole is barely warm.

Auto scaling is doing the right thing. It targets consumed capacity against a target utilization, and the table average is low, so there is nothing to scale. Even at 8,000 WCU the offending partitions would still be capped. Adaptive capacity can isolate a hot partition, but it cannot overcome a design where one key holds a disproportionate share, and it is not a substitute for a fix.

Confirm before changing anything: enable CloudWatch Contributor Insights for the table and look at the top partition keys - you should see the 300 offending deviceIds dominating. Then:

Shard the hot keys: append a suffix, deviceId#<0..9>, chosen at random or by hash at write time. Reads fan out across the shards and aggregate. This trades read amplification for write headroom.
Separate the hot producers: route high-frequency devices to their own table or a Kinesis Firehose path with different capacity, so normal devices are unaffected.
Aggregate at the edge: devices sending every 50 ms should not produce one item each. Batch on the device or ingest into a stream and write one rolled-up item per second per device. That is a 20× reduction with no schema risk.

2. The SDK's default retries turned a 200 ms write into a 2-second p99 and pushed the Lambda into timeouts. What is the correct retry strategy here?

Read the worked answer

Retries are amplifying the throttle. When writes fail fast, three immediate retries multiply the request rate, the hot partition stays saturated, and each retry consumes a concurrency slot for longer - the failure feeds itself.

What to change:

Use exponential backoff with jitter, and for the AWS SDK switch to adaptive retry mode, which applies a client-side rate limiter when it detects throttling. Default legacy/standard backoff is too aggressive for a saturated partition.
Raise maxAttempts moderately (for example 5–6) with full jitter, so retries spread across clients instead of retrying in lockstep. Never retry a ValidationException or a conditional-check failure - only throttling and 5xx are retryable.
Give the Lambda timeout room for the whole retry envelope. If the function timeout is below the worst-case total backoff, the invocation dies mid-retry and SQS/stream redelivery starts the cycle again. Prefer a shorter SDK retry budget and a function timeout comfortably above it.
Detect and shed. If a single device is emitting 20 writes per second, that is a producer bug: rate-limit or dirty-limit per device at ingestion, and return success so the device does not amplify. Persist unretryable failures to a DLQ rather than dropping them.

After the fix, re-measure p99 - under healthy conditions a DynamoDB single-item write should be low single-digit milliseconds, so a 2-second p99 is retry noise, not DynamoDB latency.

3. The ingestion Lambda runs at 900 concurrent executions against a 1,000 limit. What are the options, and what are the trade-offs during this incident?

Read the worked answer

The function is consuming nearly the whole account concurrency pool, so every other function in the account - including the ones you need for recovery and monitoring - is at risk of throttling. During an incident, that means the console's own tooling can fail.

Options:

Reserved concurrency on this function (for example 700). Immediately caps the blast radius and guarantees headroom for everything else. Downside: the function is now hard-capped, so if it is already at 900 the backlog grows faster. That is the honest trade - you are choosing a controlled backlog over a cascading failure.
Move to an SQS-buffered consumer. Put the queue between the producers and the writer, then control drain rate explicitly with batch size and reserved concurrency. Producers get a fast 200, the consumer is decoupled from the burst, and a DLQ gives you a place for poison records.
Reduce invocation cost per event. Larger batch sizes and pre-aggregation (see the hot-key fix) cut the number of concurrent invocations needed for the same write throughput. This is the only option that lowers the requirement rather than redistributing it.

Do not raise the account limit as the first move - the underlying problem is a hot partition and an unbounded producer, and a bigger concurrency pool would simply drive more throttled writes into the same 300 partitions. Fix the data model first, then right-size concurrency against a measured target drain rate.