Process and Manage Data
What you must know
Data processing covers ingestion, transformation, storage, and hot-path versus cold-path analysis. Know IoT Hub built-in endpoint versus custom endpoints, and how consumer groups let multiple readers share an Event Hubs-compatible endpoint without competing. Know Azure Stream Analytics windowing (tumbling, hopping, sliding, session) and reference data joins.
Know Azure Functions triggers and bindings for IoT Hub and Event Hubs, and the checkpointing model that prevents reprocessing. Understand Blob Storage tiers and lifecycle management for cold telemetry, and Azure Data Explorer or Synapse for analytics at scale.
Traps
- Using the same consumer group for two readers - this causes contention and missed messages.
- Assuming the built-in endpoint retains data indefinitely - retention is limited.
- Confusing tumbling (non-overlapping, fixed) with hopping (overlapping) windows.
- Forgetting that Stream Analytics output ordering and late-arrival tolerance settings affect results.
Rule of thumb
One consumer group per independent reader. If two services read the same stream, give each its own consumer group; otherwise one starves the other.
Worked example
A Stream Analytics job needs a 5-minute average that updates every minute. Use a hopping window of 5 minutes with a hop size of 1 minute, not a tumbling window, because tumbling windows would not overlap and would update only every 5 minutes.