The Data Lake With No Shore
A mid-size logistics company built a data lake three years ago on the promise of "democratizing data." Today it holds about 4,000 tables across 40 schemas, ingested from 60 source systems by a team of four engineers. There is no data catalog. Table names follow no convention: ship_v2_final, ship_v2_final_use_this, tmp_customer_2021. The BI team maintains 300 reports, of which the head of operations says "maybe 40 get used." Two analysts recently built competing churn models on different customer tables and presented conflicting numbers to the executive team in the same week. The CFO has frozen new ingestion projects until "we know what we have." The engineering team wants to rebuild the lake from scratch with a modern lakehouse architecture. The BI team wants a semantic layer. Both want 12 months and most of the data budget.
1. The CFO asks you to recommend between the lakehouse rebuild and the semantic layer. How do you structure the decision?
Read the worked answer
Framing this as a binary choice between two 12-month programs is the first thing to challenge. The symptoms point to a governance and inventory gap that neither architecture solves on its own.
Diagnose before prescribing:
- The conflicting churn models are not an architecture failure - they are a missing-definition and missing-catalog failure. A lakehouse with no catalog reproduces the same problem in newer technology.
- 260 unused reports suggest the BI layer is not the binding constraint either; the constraint is that nobody knows which assets are authoritative.
- 4,000 tables from 60 systems with four engineers is an ingestion-discipline problem. Rebuilding will re-ingest the same sprawl unless intake is governed.
How I would structure the decision:
- Inventory first, cheaply. Automated scanning to classify tables by last-query date, downstream dependency, and owner. This is weeks, not a year, and it turns "4,000 tables" into a ranked list.
- Retire before you rebuild. Anything with no query in 12 months and no downstream dependency gets archived with owner sign-off. The rebuild scope shrinks.
- Establish a thin governance layer now: naming standards for new tables, a required owner field, and a small set of certified datasets for the metrics that reach executives. This is the semantic layer's real value, and it can start before any platform decision.
- Then decide the platform on evidence: if the retirement pass leaves a manageable estate, the lakehouse rebuild may be unjustified; if the underlying storage and compute are genuinely limiting, it is.
What I would tell the CFO: neither proposal should be funded as a 12-month blank check. Fund a 6-8 week inventory and retirement pass with a named owner, then re-scope. The rebuild and semantic layer are likely both partly needed, but the sequence and size should follow the inventory, not precede it.
2. How would you prevent the "two competing churn models" incident from recurring, given that you cannot stop analysts from building models?
Read the worked answer
You cannot and should not stop analysts from building models. The failure was not that two models existed - it was that both reached the executive team with equal apparent authority and no way to tell them apart.
Controls that address the actual failure:
- Certified vs. exploratory distinction. Define a small set of certified datasets and metrics that carry an owner, a documented definition, and a review. Anything not certified is explicitly exploratory and cannot be presented to executives as a headline number without going through review. This is a labeling and process control, not a technical one.
- A single definition for "customer" and "churn." The two analysts almost certainly disagreed on the population and the churn window before they disagreed on the model. Publishing those two definitions resolves most of the conflict before modeling starts.
- A lightweight model registry. Not gatekeeping - a record of who built what, on which dataset, with which definitions, and what it concluded. When two results conflict, the registry shows immediately whether the difference is data, definition, or method.
- A pre-presentation check for executive-facing numbers. If two teams are presenting related metrics in the same week, someone owns reconciling them first. The incident was as much a coordination failure as a data one.
What I would avoid: a heavyweight approval process for all analysis. That pushes modeling underground and produces the same conflict with worse documentation. The control should sit at the point where a number gains authority - the executive presentation - not at the point where analysis begins.