Streaming Sensor Data with Event Hubs and Delta Lake
A manufacturer ingests 50,000 events/second from factory sensors into Azure Event Hubs. A Databricks job reads the stream and writes to Delta Lake. The job currently uses foreachBatch with MERGE into a target table and runs every 10 seconds. After two weeks, the job lags by 40 minutes during peak shifts, and the target table has 1.2 million small files. Duplicate device readings appear because the producer retries. The team wants exactly-once semantics and lower latency.
1. Diagnose the lag and small-file problem. What architectural changes do you recommend?
Read the worked answer
The 10-second trigger with MERGE creates many small commits, and each MERGE rewrites files. The lag comes from per-batch overhead plus file proliferation. Recommend: (1) increase trigger interval or use availableNow/continuous with a larger maxOffsetsPerTrigger; (2) use foreachBatch to append raw events to a Bronze Delta table with mergeSchema off, then run a separate compaction job (OPTIMIZE with ZORDER or auto-compaction) to coalesce files; (3) enable Delta optimizeWrite and autoCompact; (4) partition Bronze by ingestion date/hour, not by device. This separates ingestion from curation and reduces small files.
2. How do you achieve exactly-once when the producer retries and may send duplicate events?
Read the worked answer
Event Hubs gives at-least-once delivery, not exactly-once. Use the event's unique DeviceId + SequenceNumber or a producer-supplied EventId as a deduplication key. In the Silver layer, MERGE on that key with WHEN MATCHED THEN UPDATE or INSERT only if not exists. Store the checkpoint in a durable location so the stream resumes without reprocessing. Alternatively, use Delta's idempotent writes with txnAppId and txnVersion in foreachBatch to make each batch idempotent. Document that end-to-end exactly-once requires the producer to supply a stable key.
3. What monitoring would you add to catch lag before it reaches 40 minutes?
Read the worked answer
Track Event Hubs IncomingMessages, OutgoingMessages, ThrottledRequests, and consumer group lag via Offset vs LastEnqueuedSequenceNumber. In Databricks, monitor numInputRows, inputRowsPerSecond, processedRowsPerSecond, and batch duration in the Spark UI or via metrics sink. Alert when lag exceeds 5 minutes or when batch duration exceeds the trigger interval. Also monitor Delta table file count and OPTIMIZE history. Set alerts in Azure Monitor and a dashboard for the on-call team.