Right-Sizing Throughput for a Bursty Telemetry Fleet
Contoso operates 8,000 leaf devices that report every 10 seconds during business hours (08:00–18:00 local) and every 5 minutes otherwise. Each message is 1.5 KB and devices use MQTT over TLS. The team provisioned a Standard S1 hub (400 messages/second throttle, 4,000 messages/day free tier not applicable) in a single region. During the first month, peak-hour telemetry arrives at roughly 1,300 messages/second because device clocks drift and many devices retry aggressively after transient failures. The operations team reports frequent 429 throttling errors, and the business wants no more than 10% message loss during peaks. Backend processing currently reads from the built-in event hub endpoint with a single consumer group shared by two Azure Functions instances.
1. Should the team scale the S1 hub up to a higher tier, add units, or both? What is the most defensible first action, and what evidence supports it?
Read the worked answer
The core constraint is the unit-level throttle on the S1 tier, not the tier ceiling alone. S1 allows 400 messages/second of ingress per unit. The observed 1,300 msg/s peak is more than three times the S1 capacity of a single unit, so simply adding units (S1 supports scaling units) would require at least 4 units to cover the peak, and the team should verify the tier's maximum unit count first.
A defensible sequence:
- Quantify the real peak. Confirm 1,300 msg/s is genuine business traffic versus retry amplification. If retries account for a large share, fixing retry behavior reduces the required capacity and cost.
- Fix retry storms first. Aggressive retries multiply load exactly when the hub is throttled. Exponential backoff with jitter and client-side rate limiting is cheaper than over-provisioning.
- Then scale. If the corrected peak is still above 400 msg/s, scale out units or move to a tier whose per-unit and total throughput fit the corrected peak with headroom (for example 20–30%).
Scaling to a higher tier without fixing retries risks paying for capacity that amplifies itself. The exam-style judgement is: measure, remove self-inflicted load, then size to the corrected peak with headroom, and confirm the chosen tier's per-unit and maximum throughput limits support it.
2. The two Functions instances share one consumer group. What problem does this create, and how should the team change the design?
Read the worked answer
A consumer group in the built-in Event Hubs-compatible endpoint is a single logical cursor over the partition stream. When two independent readers share one consumer group, they compete for partition leases; each partition is owned by one reader at a time. The result is uneven distribution, frequent lease stealing, and potential duplicate or skipped processing depending on checkpoint behavior. It does not provide clean parallel fan-out.
Correct design choices:
- One consumer group per independent consumer application. If two Functions need the same data for different purposes, give each its own consumer group. If they are instances of the same function scaling out, sharing a consumer group is correct and expected - the Functions host coordinates partition ownership across instances.
- Partition count matters. The built-in endpoint's partition count is fixed at hub creation, so parallelism is capped by partitions. If the team needs more concurrent readers than partitions, they must plan partition count up front or route through a downstream service.
- Checkpointing. Ensure checkpoints are stored reliably so a restart resumes rather than reprocessing the whole retention window.
The judgement: distinguish "scale-out instances of one consumer" (shared group is fine) from "two different consumers" (separate groups required). The setup implies two Functions instances of possibly different logic, so the team must clarify intent before changing anything.
3. What throttling and monitoring signals should the team watch to know whether the fix worked?
Read the worked answer
The team should track both hub-side throttling metrics and end-to-end delivery evidence:
- Throttling errors (429) and throttled message counts on the hub - should drop toward zero at peak.
- Ingress message rate versus the tier limit - confirm corrected peak sits below the limit with headroom.
- Device-side retry rates and backoff behavior - verify the retry amplification is actually gone, not just shifted.
- Message loss / gap detection - compare device-sent sequence numbers against backend-received counts to measure the real loss rate against the 10% target.
- Consumer lag and checkpoint age - ensures backend keeps up and the shared-consumer-group issue is resolved.
- Partition distribution - detect hot partitions or uneven lease ownership.
Judgement point: 429s disappearing is necessary but not sufficient; the business cares about loss. Sequence-number reconciliation is the metric that actually proves the 10% objective is met.