IoT Sensor Data: Sampling and Storage Trade-off
A factory installs 200 temperature sensors on machines. Each sensor reports one reading per second, 24 hours a day. Each reading is stored as a record of about 100 bytes. The plant manager wants to keep every raw reading for 5 years for warranty disputes, and also wants a dashboard showing the average temperature per machine per hour. The database server has 2 TB of usable disk. A vendor offers compression that reduces raw records to about 25 bytes on average.
1. Calculate the raw storage requirement for 5 years without compression and determine whether the server suffices.
Read the worked answer
Readings per day = 200 sensors × 86,400 s = 17,280,000 readings/day.
- Bytes/day = 17,280,000 × 100 = 1,728,000,000 ≈ 1.73 GB/day.
- Per year ≈ 1.73 GB × 365 ≈ 631 GB/year.
- Five years ≈ 3.15 TB.
The 2 TB server is insufficient - the requirement exceeds capacity by roughly 1.15 TB before any indexes or overhead. The plan as stated fails on arithmetic alone.
2. Apply the vendor's compression and evaluate whether the plan now fits, including the dashboard requirement.
Read the worked answer
With 25-byte records:
- Bytes/day = 17,280,000 × 25 = 432,000,000 ≈ 0.43 GB/day.
- Five years ≈ 0.43 GB × 365 × 5 ≈ 0.79 TB.
That fits within 2 TB with headroom for indexes and the hourly aggregates. The dashboard does not need to scan raw data: precompute hourly averages per machine (200 machines × 24 h × 365 × 5 ≈ 8.8 million aggregate rows, tiny). So the design is: compressed raw readings for disputes, plus a small aggregate table for the dashboard. The compression is what makes the 5-year retention feasible.
3. The plant manager worries that compression could lose data needed for a warranty dispute. How do you address this?
Read the worked answer
Lossless compression preserves every value; the concern is about lossy schemes or downsampling. Clarify which the vendor offers and require lossless for the raw tier.
To be safe: keep raw readings lossless-compressed, and verify by spot-checking that decompressed records match originals after a sample period. If the vendor's 25-byte figure assumes lossy aggregation, reject it for the dispute tier and reserve aggregation only for the dashboard. The dispute use case needs exact historical values at the recorded timestamps; the dashboard needs trends. Separating the two tiers resolves the tension without compromising either.