Real-Time Alerts and Exact Daily Money From the Same Event Stream

Ingestion: Azure Event Hubs. It ingests millions of events per second per namespace, partitions the stream for parallel consumers, retains events so a downstream failure can be replayed, and decouples 12,000 chatty producers from consumers. Event Hubs Capture can also drop every raw event into storage automatically, which satisfies the three-year archive requirement without writing a consumer to do it.

Alerting: an Azure Stream Analytics job. The rule is a windowed count grouped by city and error code, output to a function or action group. Stream Analytics is a managed, SQL-like query over the stream with built-in windowing, late-arrival and out-of-order tolerance, and no clusters to size or patch.

Why not Spark Structured Streaming, despite it being capable? Three reasons grounded in this brief, not in general preference:

• Skill: the team has no Spark experience. A working Spark job that nobody on the team can debug at 03:00 is an outage waiting to be scheduled.
• Operations: Spark requires a cluster, checkpoint storage, and cluster lifecycle management - a permanent piece of infrastructure for one alert rule.
• Cost: a continuously running cluster is a poor match for a budget described as tight.

Why not Azure Functions alone? Functions scale well but give you no windowing or state management; you would build the counting and de-duplication yourself, which is precisely the part that is hard to get right.

The requirement is 'more than 15 bikes in one city reporting the same error code', detected within 60 seconds.

• Tumbling window (1 minute, non-overlapping). Each minute is evaluated exactly once, so one condition produces at most one alert and the arithmetic is easy to defend. Cost: detection can lag by up to the window length, and a burst straddling a boundary is split across two windows and may never cross the threshold. This is the right default when you pair it with a short window and a de-duplication step downstream.
• Sliding window (for example 5 minutes, evaluated every 30 seconds). Overlapping windows mean the alert fires as soon as the condition is true and keeps firing while it stays true. Cost: duplicate alerts, so you must add suppression (only notify when a city/error pair transitions from below to above the threshold). Better for detection latency, worse for alert fatigue.
• Session window. Groups events separated by gaps of inactivity - a natural fit for an error episode that starts, cascades, and subsides. Cost: the threshold question becomes fuzzy, because the window length is data-driven and you cannot state 'in the last minute' precisely.

Recommendation: tumbling 1-minute windows for the count, plus a downstream de-duplication state so the same city/error pair does not page twice within the suppression period. Move to a sliding window only if operations confirms it will accept duplicate-suppressed alerts in exchange for sub-minute detection. Whatever the choice, the Event Hub partition count must be at least the streaming-unit parallelism you want, or the throughput ceiling, not the query, becomes the limit.

Why streaming cannot be the source of record: the daily job would close the day shortly after midnight while events for the last six hours of that day are still in flight. The totals would be wrong, and worse, wrong in a way that only shows up in a payment dispute weeks later. Widening the late-arrival tolerance in Stream Analytics helps, but tolerance is a heuristic, it adds latency, and it still cannot promise completeness for a financial figure.

Reconciliation design:

1. Archive the raw stream immutably. Event Hubs Capture into the lake, partitioned by date and hour, untransformed. This is the replayable source of truth.
2. Define a settlement window. Re-run the day's aggregation at, say, T+8 hours, after the six-hour late-arrival tail has closed, and produce the finance figure from that batch run over the archive - not from the streaming output.
3. Make the batch job idempotent. Key the output by rental ID and upsert or overwrite the day's partition, so a re-run corrects the number rather than doubling it. Streaming outputs that are at-least-once must never be summed directly into a financial total without this.
4. Keep the streaming path for what it is good at: near-real-time monitoring and operations alerts. Different consumer, different guarantee, different latency.
5. Reconcile, do not assume. Compare the batch total against the payment gateway's settled count and surface the delta as a metric; a persistent non-zero delta means an event schema or capture problem, which is exactly what you want to know before an auditor asks.

Archive the raw events, unchanged, partitioned by arrival date and hour, with a lifecycle policy that keeps recent months on a hot tier and moves older data to cool or archive tiers. Three years at this volume is the largest storage item in the design, so tiering is not optional.

Also produce a curated Parquet (or Delta) copy for the data science team: flattened, typed columns, partitioned by date and city, and compacted so that a scan does not open a million tiny files. Raw JSON is the replay source; Parquet is what people query. Conflating the two means either slow analysts or an un-replayable archive.

What I would deliberately not do:

• Do not delete or overwrite the raw zone after transformation. When the flattening logic turns out to be wrong - and it will - the raw zone is the only way to re-derive history. This is the single most expensive mistake available in this design.
• Do not archive only the curated data, however convenient it looks. You would be archiving your current interpretation of the events rather than the events.
• Do not store the archive in the same container or account with the same access policy as the live pipeline output. A transform bug that writes to the wrong path must not be able to reach the retained history.
• Do not apply the retention policy to the streaming output topic as a substitute for archive storage; event retention in Event Hubs is measured in hours to days, and it is a replay buffer, not a data lake.