Designing a Mixed Batch and Streaming Architecture for Fleet Telemetry
Northline Logistics tracks 48,000 delivery vehicles. Each vehicle emits a GPS record every 10 seconds, producing about 3 TB of raw JSON per day. The company runs a 12-node Hadoop cluster with 2 TB free HDFS capacity. Today a nightly Hive batch job computes delivery KPIs; dispatchers see results only the next morning. Operations now wants fleet alerts (geofence breach, engine overheating) within 5 minutes, while finance still needs monthly billing reports over 24 months of history. The team is fluent in SQL, has one part-time Java developer, and cannot expand the cluster this quarter. The CTO asks you to propose an ingestion and processing architecture using the Hadoop ecosystem, and to explain which workloads should stay batch and which need streaming. She also wants to know whether the existing Hive tables should be reused for the alerting path.
1. Select an ingestion and processing architecture from the Hadoop ecosystem for the two workloads. Justify each component and state what runs batch versus streaming.
Read the worked answer
Use a two-path architecture rather than forcing one engine to serve both SLAs.
Streaming path (5-minute alerts): ingest vehicle events into Kafka, which buffers bursts and decouples producers from consumers. Run a Spark Streaming or Flink job that evaluates geofence and temperature rules over a short window and writes alerts to a low-latency store or message topic. The 5-minute SLA is achievable because the job processes only the current window, not the full history.
Batch path (billing and KPIs): land raw events in HDFS, then process with Hive or Spark SQL on a schedule. Billing needs complete, reconciled daily data, so batch is appropriate; monthly reports over 24 months are inherently historical scans.
Why not Hive for alerts: Hive is optimized for throughput, not per-event latency. A Hive query has planning and execution startup overhead and typically scans partitions; it cannot reliably deliver 5-minute freshness under load. Reusing Hive tables for alerting would also mix append-heavy event data with reporting schemas.
Keep the SQL team productive by expressing both paths in SQL where possible: Spark Structured Streaming supports SQL-like operations, and Hive remains the reporting layer. The single Java developer maintains connectors and rule logic, not an entire custom platform.
2. Explain why the 5-minute alerting path should not simply query the existing Hive tables, using concrete latency and capacity reasoning.
Read the worked answer
Three constraints make direct Hive querying unsuitable.
Latency structure: Hive compiles a query, negotiates YARN containers, and schedules tasks. Even a small query can take tens of seconds to minutes just to start, leaving little headroom inside a 5-minute window. Alerting needs continuous processing, not repeated ad hoc queries.
Scan volume: at 3 TB/day, a 5-minute slice is roughly 10 GB of raw JSON. If the Hive table is not partitioned and bucketed for event-time lookups, each alert query may scan far more than that slice. With 48,000 vehicles, per-vehicle queries would multiply the scan cost.
Concurrency and state: geofence and overheat rules require maintaining state across events (for example, temperature above threshold for N consecutive readings). Hive is not designed for long-running stateful operators; a streaming engine handles windows and state natively.
A better pattern is to write a compact, deduplicated alert stream from the streaming job, and let Hive consume that stream for audit and reporting. This preserves SQL access without putting Hive on the critical alert path.
3. Finance wants 24 months of history on a cluster with only 2 TB free. Propose a storage plan and justify trade-offs.
Read the worked answer
First check the arithmetic: 3 TB/day × 730 days is about 2.2 PB of raw history. The cluster has 2 TB free, so retaining raw JSON for 24 months on this cluster is impossible. The plan must separate raw retention from reporting retention.
Raw layer: keep a short operational window, for example 7–30 days, compressed and partitioned by date. This supports replay and incident investigation. Even 30 days is about 90 TB, so confirm total cluster capacity, not just free space; if capacity is insufficient, land raw data in object storage and keep only recent partitions in HDFS.
Reporting layer: retain aggregated daily and monthly billing facts for 24 months. Daily aggregates are orders of magnitude smaller than raw events. Store them as partitioned ORC tables, compressed, with only the columns finance needs.
Archive: move older raw partitions to cheaper object storage with lifecycle rules. Keep Hive external tables pointing at the archive for rare audits.
The trade-off is that raw event replay beyond the retention window requires restoring from archive, which is slower. That is acceptable because billing uses reconciled aggregates, while raw replay is an exception path.