I85PREP
⌘ K
DataX study materials
DataX study resources

DataX: Data Platforms and Tools

DataX · Work through your exam, one topic at a time.

All guide sections

Data Platforms and Tools

Data Platforms and Tools tests whether you can explain what each layer of a modern data stack does, and choose configuration that makes queries fast and cheap without breaking correctness.

Core rules and concepts

  • Warehouse, lake, lakehouse: a warehouse stores curated schema-on-write tables on MPP compute with strong SQL and transactions; a lake stores raw files cheaply, schema-on-read; a lakehouse adds table formats - Delta Lake, Apache Iceberg, Apache Hudi - bringing ACID transactions, time travel and schema evolution to object storage.
  • Columnar storage and MPP: columnar formats (Parquet, ORC) read only referenced columns and compress well; MPP engines parallelise across nodes. Wide CSV scans and full-table aggregation are the usual cost drivers.
  • Partitioning and pruning: partition on a low-to-medium cardinality column used in filters (usually date), then cluster or bucket on high-cardinality join keys. Partitions must be coarse enough to avoid small files - target files in the hundreds of MB, not thousands of tiny ones.
  • Compute/storage separation lets you scale engines independently, but you pay for idle compute when auto-stop is misconfigured.
  • Orchestration and metadata: schedulers (Airflow, DolphinScheduler, Azkaban) express dependencies and retries; a catalog/metastore holds schemas, partitions and lineage. Synchronisation tools such as DataX move data between sources and the platform in batch; they are not transformation engines.
  • Performance levers: partition pruning, avoiding SELECT *, pre-aggregation, bucketing for joins, skew handling (salting, broadcasting small tables), and small-file compaction.

Worked example

A 5 TB daily table in CSV is queried by date range and customer_id, scanning two years each time. Switch to Parquet with Snappy compression, partition by dt daily, and bucket by customer_id where the engine supports it. A one-month query now prunes to that month's partitions and reads three columns, cutting scan volume by orders of magnitude. A compaction job keeps each partition's files at a few hundred MB.

Traps

Partitioning by a high-cardinality ID such as order_id; leaving thousands of small files from streaming ingestion; using CSV for analytical workloads; applying OLTP row-store tuning to OLAP; assuming a sync job's success proves the data landed correctly - count and reconcile.