Data Platforms and Emerging Technologies
Data Platforms and Emerging Technologies tests whether you can select and govern platform components - storage layers, compute engines, catalogues, pipelines and newer AI-facing services - and explain the trade-offs behind the choice. Questions target patterns such as lakehouse and data mesh, metadata and lineage, orchestration and reliability, cost control, and the governance demands of machine learning and generative AI.
Platform patterns
- Data lake stores raw files cheaply without transactional guarantees; warehouse gives governed SQL and performance at higher cost per terabyte; lakehouse adds a table format with open metadata and ACID transactions over object storage, so one copy serves BI and ML.
- Medallion layers: bronze (raw, immutable), silver (cleaned, conformed), gold (business-level marts). Each layer has an owner and a quality gate.
- Data mesh devolves ownership to domains publishing data products with contracts, backed by a self-serve platform and federated governance; it fails without real platform capability and enforceable contracts.
- Data fabric centralises metadata, lineage and policy to connect distributed stores.
Operations and emerging technology
Orchestration must be dependency-aware and idempotent so a failed task reruns safely; monitor freshness, volume and schema drift, not only job success. Infrastructure as code makes environments reproducible. Cost is a design property: storage tiering, partition pruning and switching off idle compute are governed decisions, not finance clean-up. For ML and generative AI, register models and prompts, track training-data lineage, evaluate bias and drift, and apply the same classification and access rules to feature stores and vector indexes that hold embedded sensitive text.
Worked example
A retailer wants one platform serving finance reporting and a demand-forecasting model, with strict audit requirements. Reasoning: a warehouse-only design cannot cheaply hold the raw event history the model needs; a lake-only design cannot guarantee the SQL consistency finance requires. A lakehouse with an open table format satisfies both, provided bronze is immutable, silver applies deduplication, and gold marts are contract-tested. Catalogue and lineage are mandatory because auditors must trace a reported figure back to source events. Compute is separated from storage so the forecasting cluster scales without duplicating data.
Traps that cost marks
- Calling a folder of Parquet files a lakehouse without a table format.
- Assuming data mesh removes the need for central standards.
- Monitoring job success while ignoring freshness and schema drift.
- Letting the ML team copy sensitive features into an ungoverned store.
- Treating cloud cost as purely a finance concern.