CDESS common mistakes
CDESS · Understand the traps and how to avoid them.
When asked to design an ingestion pipeline, the candidate jumps straight to writing transformation logic and never states partitioning, file format (Parquet/Avro), or compaction strategy, so the answer describes a working job that would produce a small-file swamp.
Candidates port habits from application development where correctness of the transform is the whole deliverable; they forget that on a lakehouse the layout of the written data is part of the engineering requirement and is what the question is really testing.
What to do instead
Before writing any transform, force yourself to state four decisions explicitly: target table format, partition spec (and why those columns), target file size, and the maintenance job (compaction/rewrite) that keeps file sizes healthy. Practice by taking any pipeline prompt and writing this one-paragraph 'physical design' preamble first, then the logic.
Treats idempotency as an afterthought: writes an overwrite-or-append job that double-counts on retry, and cannot explain how a re-run after partial failure avoids duplicates.
In batch tutorials jobs run once and succeed; candidates never internalize that schedulers retry, tasks fail mid-write, and exactly-once semantics must be designed rather than assumed. They also conflate 'the job succeeded' with 'the data is correct'.
What to do instead
For every write, name the mechanism that makes re-runs safe: deterministic partition overwrite (dynamic partition overwrite of the affected partitions), MERGE on a business key, or a transactional table format with snapshot isolation. Be ready to describe the failure scenario (task dies after writing half a partition) and how your design recovers from it.
Describes streaming as 'batch but faster' and never addresses event time versus processing time, watermarks, or late-arriving data, so a windowed aggregation silently drops or misattributes records.
Candidates who learned SQL on static tables carry the assumption that all rows for a window are present when the window closes; the mental model of unbounded, out-of-order input is genuinely new and easy to skip under time pressure.
What to do instead
For any windowed question, explicitly state: the event-time column, the watermark bound and the lateness it tolerates, what happens to records later than the watermark (dropped, side output, or re-emitted correction), and the output mode. Rehearse one concrete example with a record arriving 10 minutes late and trace exactly where it lands.
Chooses a delivery guarantee by buzzword ('exactly-once') without connecting it to the sink: claims end-to-end exactly-once while writing to a sink that has no transactional or idempotent support.
Exactly-once is marketed as a engine feature, so candidates repeat it as a property of the whole pipeline. They miss that the guarantee is only as strong as the weakest component between source offset commit and sink commit.
What to do instead
Always decompose the claim: source checkpointing/offset management, engine state and checkpoint storage, and sink semantics (transactional, idempotent upsert, or at-least-once). State what breaks if the sink is non-transactional, and name the fallback (dedup on a key downstream, or idempotent MERGE).
Assumes Iceberg's hidden partitioning and metadata mean no maintenance is needed, and cannot explain what happens to query planning and scan performance after thousands of small commits.
The table format's abstractions (partition evolution, snapshots, manifest lists) are presented as solving layout problems automatically, so candidates stop thinking about manifest count, snapshot accumulation, and file sizing as operational concerns.
What to do instead
Memorize the maintenance triad and when each applies: compaction/rewrite of small data files, manifest rewriting to keep planning cheap, and snapshot expiration plus orphan file cleanup. Be able to say which one fixes slow planning versus slow scans versus runaway storage cost.
Writes a MERGE or delete against an Iceberg table without considering row-level delete files and their effect on read amplification, then is surprised that read performance degrades after many updates.
Candidates treat Iceberg as 'a table with ACID' and stop at the transaction guarantee; the copy-on-write versus merge-on-read trade-off, and the need to periodically rewrite delete files, is a detail they have read once and never applied.
What to do instead
For update-heavy workloads, state the write mode explicitly (copy-on-write for read-heavy, merge-on-read for write-heavy), and pair it with a rewrite-data-files job that compacts delete files. Be ready to explain why a table with many small deletes is slow to read even though each individual write was fast.
Answers a catalog or metastore question by naming a component but not the security plane: cannot say how a query against a table is authorized, where the policy lives, and what happens when the same table is accessed from a different engine.
Administration questions feel like 'name the service' questions, and candidates memorize topology diagrams without tracing a single request through authentication, authorization, and audit.
What to do instead
Practice tracing one request end to end: client authenticates (who), the service checks a policy (which authorizer, which resource hierarchy), the access is audited (where the log lands). Repeat for two different engines on the same table so you can articulate where enforcement is centralized versus per-engine.
Confuses authentication with authorization when troubleshooting access failures, and proposes loosening a role or group binding before establishing whether the principal was even authenticated correctly.
Both failures surface as 'access denied', so candidates jump to the permission they can see (a role grant) rather than the identity chain (Kerberos ticket, token, group membership resolution) that may be silently failing.
What to do instead
Adopt a fixed triage order and say it out loud: identity proven? group membership resolved? policy allows this resource for that action? audit log confirms which of the three failed? Then fix only the failing layer, citing the evidence you would look for in each case.
Ignores the interaction between the streaming layer and the lakehouse: proposes writing raw events straight into a table with a commit per micro-batch and no thought to commit frequency, small files, or downstream reader isolation.
Domains are studied separately, so candidates answer streaming questions with streaming tools and lakehouse questions with table tools, and never design the seam where high-frequency writes meet a table format with per-commit metadata costs.
What to do instead
When a pipeline crosses domains, explicitly design the boundary: micro-batch or checkpoint interval, target file size after each commit, compaction cadence, and how readers see consistent snapshots while writes continue. State the cost of committing every few seconds versus every few minutes.
Gives a generic answer to a capacity or configuration question ('add more nodes', 'increase memory') instead of naming the specific resource, the metric that shows it is the bottleneck, and the configuration that changes it.
Under time pressure candidates retreat to safe-sounding generalities; they also lack hands-on familiarity with which metric (GC time, shuffle spill, queue backlog, manifest count) diagnoses which bottleneck.
What to do instead
For each tuning question, answer in the form: symptom metric, likely bottleneck, specific configuration to change, and the trade-off it introduces. Build a short personal table of five or six metric-to-cause pairs and rehearse applying it to unfamiliar scenarios rather than memorizing one fix.