Bucketing 90 Days of Sensor Telemetry
A telemetry platform ingests readings from 25,000 sensors. Each sensor sends one reading every minute. Readings are kept for 90 days, then deleted. The current schema stores one document per reading with sensorId, timestamp, temperature, pressure, and status. That creates about 36 million inserts per day and roughly 3.2 billion documents at steady state. The main dashboard queries the last 24 hours for one sensor and renders a chart; an alerting job computes five-minute averages for each sensor; and engineers occasionally query a single sensor over a seven-day window. The current compound index on sensorId and timestamp is growing quickly, and deletes are expensive. The team wants to reduce document count and index size while preserving per-sensor time-range queries. They can tolerate a few seconds of delay for alerts.
1. Design a bucketed document schema and choose a bucket width. Show the expected document count reduction and the index you would create.
Read the worked answer
Use one document per sensor per hour. At one reading per minute, an hourly bucket holds 60 readings. Document count becomes 25,000 sensors x 24 buckets = 600,000 documents per day, or about 54 million at 90 days, versus 3.24 billion one-reading documents. That is roughly a 60x reduction. Each bucket is small enough to stay far below 16 MB, likely 6 to 10 KB with four numeric fields per reading.
Schema:
{ _id, sensorId, bucketStart,
count,
readings: [ { t, temperature, pressure, status } ] }
Index { sensorId: 1, bucketStart: 1 }. The dashboard query becomes a lookup for one sensor and bucketStart greater than now minus 24 hours, then filtering or unwinding the readings array. The seven-day query touches about 168 buckets instead of 10,080 individual documents.
Hourly buckets are a deliberate compromise: day buckets hold 1,440 readings and make each update rewrite a much larger document, while minute buckets would not reduce document count. If sensors can burst above one reading per minute, account for that with a count limit or the outlier pattern rather than letting buckets grow unbounded.
2. How would you implement retention, late-arriving readings, and five-minute averages? Discuss tradeoffs.
Read the worked answer
Retention: put a TTL index on bucketStart with expireAfterSeconds set to 90 days. TTL deletes whole hourly buckets, not individual readings, so the delete workload and index churn drop sharply. The TTL monitor is asynchronous, so allow a small grace period beyond 90 days.
Late readings: insert with an upsert on { sensorId, bucketStart }, using $push and $sort on readings.t. Make retries idempotent by checking for an existing reading identifier or timestamp before pushing. If the bucket has already expired, either reject the reading or keep a short extra retention window for late data. Do not let a late write create a second bucket with the same key; a unique index on { sensorId, bucketStart } prevents that.
Five-minute averages: compute them on read from one hourly bucket because 60 readings is cheap to aggregate. If the alerting job runs frequently, store a fiveMinuteAverages array in the bucket via the computed pattern. That saves CPU but adds write complexity and contention on the current bucket. Given the few-seconds tolerance, on-read aggregation over the current and previous bucket is simpler and less error-prone. Monitor bucket size and split or limit when a sensor produces outliers.
3. Compare this manual bucketing design with using MongoDB time-series collections. When would you choose one over the other?
Read the worked answer
Time-series collections automate bucket creation and storage optimization, provide time-based indexes, and support TTL-style expiry. For pure telemetry with high insert rates and time-range queries, they are usually the better default on supported server versions because they reduce custom code and storage overhead.
Choose manual bucketing when you need explicit control over bucket boundaries or want to store precomputed fields inside the bucket, such as five-minute averages or status counters. Manual bucketing also works on older server versions and can be easier to reason about when the application already has custom rollup logic. It is also useful when readings need more complex per-bucket updates than the time-series implementation handles well. The tradeoff is that the application must manage upserts, idempotency, bucket growth, and TTL behavior. For this workload, time-series would be the first choice; manual hourly bucketing is the fallback or the choice when the precomputed aggregates are a hard requirement.