Real-Time Delivery Operations Versus the Hourly Warehouse

The requirement is five-minute detection of late deliveries, not a live map of every courier. That is an alerting and exception problem, not a sub-second visualization problem. I would choose a micro-batch pipeline every two to three minutes from Kafka into a curated operations table, with a stream-processing job only for the SLA alert path if micro-batch latency proves insufficient. This meets the five-minute requirement with far less operational complexity than full streaming, which matters with one Kafka-fluent engineer and a budget already 20% over. The curated table would hold active orders, promised time, current status, and lateness minutes; the dashboard reads that table, not the warehouse. I would keep the hourly warehouse for historical analysis. The COO's real-time map is the wrong first build: a map of 1,200 couriers does not tell an operator whom to call. The architecture should follow the intervention, not the visualization.

I would design around an exception queue, not a map. The landing view lists orders at risk of breaching 15 minutes late, sorted by minutes late and order value, with courier, city, restaurant, and last event. Each row shows the recommended action: call courier, reassign, or notify customer. A map can be a secondary drill for a selected order, not the default screen. Freshness should be visible as a timestamp and a status indicator, and the page should auto-refresh every two minutes without losing the operator's place. I would separate live operations from weekly performance reporting; mixing them creates a page that is neither fast nor analytical. I would also add a threshold alert to the operations channel so intervention does not depend on someone watching a screen. The success measure is time from late flag to action, not number of couriers displayed.

I would treat event time and processing time separately. Each event gets a unique event_id and order_id; the pipeline deduplicates on event_id and keeps the latest state per order using event_timestamp, not arrival order. A watermark or lateness window handles events that arrive after the batch closes; late events update the order state and are flagged rather than silently dropped. The curated table should be an upsert keyed by order_id, so reprocessing is idempotent. I would add data-quality monitors for duplicate rate, events older than the watermark, orders with no courier assignment after a threshold, and state regressions such as delivered followed by picked up. For the alert path, I would accept at-least-once delivery and make the alert handler idempotent, which is simpler than exactly-once guarantees. Finally, I would run a parallel comparison against the warehouse for two weeks before peak season, because streaming errors often appear as quiet state drift rather than loud failures.