Choosing storage and compute for mixed analytical workloads
A startup is building its first data platform on Alibaba Cloud. It has three workloads: (1) a nightly ETL that joins 500 GB of raw logs with reference data and writes a cleaned table; (2) an interactive BI tool where ~20 analysts run ad-hoc queries throughout the day, expecting sub-10s responses on the cleaned table; (3) a small real-time feed of app events that must be queryable within seconds for a live operations screen. The team has one engineer, a modest budget, and no existing platform. They propose putting everything in one large always-on compute cluster and pointing all three workloads at it.
1. Evaluate the single always-on cluster proposal against the three workload profiles.
Read the worked answer
One always-on cluster for all three workloads is a poor fit:
- Nightly ETL (batch, 500 GB join): bursty, heavy, latency-tolerant. It wants elastic compute that can scale up for the run and release resources. An always-on cluster sized for peak ETL wastes money the other 22 hours.
- Interactive BI (20 analysts, sub-10s): needs consistently provisioned, low-latency compute with concurrency. If it shares a cluster with the ETL, the nightly job will contend for resources and BI latency will spike during the run. Conversely, sizing the cluster for BI peak leaves it idle overnight.
- Real-time feed (seconds): batch/interactive compute is not the right engine; it needs a streaming path with a serving layer that supports fast point/range reads.
Putting all three on one cluster couples their failure domains and performance: an ETL spike degrades BI, and a BI-heavy day starves ETL. It also forces a single sizing decision that is wrong for at least two workloads. The engineer's mental model (one cluster = simple) trades operational simplicity for cost and performance, and doesn't even address the real-time requirement.
2. Propose a workload-to-service mapping and justify each choice by workload characteristics.
Read the worked answer
Map by latency, burstiness, and data shape:
- Nightly ETL → elastic batch compute (e.g., MaxCompute-style). Pay for compute during the run, scale to the job, release after. 500 GB joins are well within batch engines; latency tolerance means no need for always-on capacity.
- Interactive BI → a provisioned interactive query/serving layer (separate from batch) sized for concurrency, with the cleaned table materialized for fast scans. Keep it isolated from ETL so nightly jobs don't degrade analyst latency. Use result caching/aggregation tables for common dashboards.
- Real-time feed → streaming ingestion + a low-latency serving store. Events land in a streaming pipeline, are processed into a serving table optimized for the live screen, and optionally archived to storage for later batch analysis. Seconds-level freshness requires a streaming path, not batch.
- Storage: a shared object/storage layer as the source of truth so each compute engine reads the same cleaned data; avoid copying data into each engine's silo.
Justification is workload-driven: bursty batch → elastic; steady low-latency concurrency → provisioned isolated; seconds freshness → streaming. The one-engineer constraint argues for managed services over self-managed clusters, and for starting with the ETL+BI pair before adding streaming if budget is tight.
3. The engineer worries that separate services mean more operational burden for one person. How do you weigh that against cost and correctness?
Read the worked answer
Acknowledge the real constraint - one engineer - but reframe the tradeoff:
- Managed services reduce, not increase, operational burden compared with self-managing one large cluster (patching, scaling, failure recovery). The burden is in coordination, which can be minimized with a shared storage layer and a single scheduling/orchestration tool.
- Cost: one always-on cluster sized for peak is the most expensive option per unit of useful work because most capacity idles. Elastic batch + right-sized interactive + a small streaming path is typically cheaper and matches spend to demand.
- Correctness: isolating workloads prevents cross-contamination (ETL spikes breaking BI latency) and gives each workload the engine it needs. A single cluster cannot give sub-10s BI and seconds-level streaming simultaneously without over-provisioning.
- Pragmatic sequencing: don't build all three at once. Start with shared storage + elastic batch ETL + a right-sized BI layer; add streaming when the live screen is actually needed. This keeps the operational surface small now and grows it deliberately.
The decision rule: operational simplicity is achieved by managed services and few moving parts, not by forcing incompatible workloads onto one engine.