Debugging and Deploying
Packaging and promotion
Production code should be a wheel or a repo file, versioned in Git, tested outside the notebook. Declarative Automation Bundles (databricks.yml with resources and targets) describe jobs, pipelines, and their dependencies so the same definition is validated and deployed across dev/staging/prod - bundle validate, bundle deploy, bundle run. Bundle variables and target-specific overrides are how environment-specific paths and cluster settings are injected; the trap is hard-coding a dev path and promoting it.
Debugging signals
Errors surface as a Py4J Java error with a nested root cause - read the Caused by chain, not the top line. AnalysisException: cannot resolve column means schema mismatch (case sensitivity, wrong table version, missing evolution). Executor OOM points to skew, large collects, or whole-group pandas UDFs; driver OOM points to collect(), broadcasting too large a table, or returning large results. Spark UI stages show spill and skew; the pipeline event log shows data-quality and expectation failures; driver and executor logs are distinct places to look.
Reliability practices
Idempotent writes (merge or replaceWhere), checkpointed streams, retries with backoff for transient cloud errors, and repair runs for failed tasks. Avoid catching exceptions so broadly that the job reports success. Use dbutils.notebook.exit() carefully - it's how tasks signal values and failures across the DAG.
Rule of thumb: if it isn't in Git and isn't a bundle, it isn't deployed - and if the stack trace's top line is vague, the answer is almost always in the deepest Caused by.