Airflow Fundamentals common mistakes
Airflow Fundamentals · Understand the traps and how to avoid them.
Writing a DAG that runs work at module import time (for example calling a function or hitting an API at the top level of the DAG file) instead of inside operator tasks.
The DAG file is parsed repeatedly by the scheduler's DagFileProcessor in a fresh process; top-level side effects execute on every parse cycle, causing duplicate work, slow parsing, and confusing behavior that looks like 'the DAG runs itself'.
What to do instead
Keep the module body declarative: build the DAG object and instantiate operators only. Put all real work inside the operator's callable or execute method, and pass parameters via op_kwargs or XCom rather than computing them at import.
Assuming the scheduler executes tasks itself, or that the webserver triggers task execution directly from the UI.
The UI and CLI talk to the metadata database, not to executors; many candidates conflate the webserver with the scheduler and expect a 'Run' click to do the work in-process.
What to do instead
Memorize the split: scheduler parses DAG files and creates DagRuns/TaskInstances in the metadata DB; the executor (Local, Celery, Kubernetes) actually runs tasks via workers; the webserver only reads/writes metadata; the triggerer handles deferrable operators. Trace any question about 'who runs the task' to the executor and worker.
Using XCom to pass large payloads such as DataFrames or file contents between tasks.
XCom values are serialized and stored in the metadata database (with a size limit and performance cost), but the API looks like an ordinary function return value, so candidates treat it as shared memory.
What to do instead
Push only small primitives (IDs, paths, dates). For large data, write to external storage (S3, GCS, a shared volume) and pass the path or key via XCom. Recall that XCom pull requires the task instance context or a template, and that pulling by key needs the producing task_id.
Setting a schedule_interval or start_date such that a DAG run's logical date is confused with its execution time, and expecting a run to fire at start_date.
Airflow schedules the run for the end of the interval; the run covering [start_date, start_date + interval) is created only after that interval closes, and catchup/backfill semantics compound the confusion.
What to do instead
Remember logical_date is the start of the data interval, not the wall-clock trigger time. To run immediately for ad-hoc testing, trigger manually or use a schedule with a past start_date plus catchup=False. Check data_interval_start/data_interval_end in templates.
Relying on execution order implied by the order operators are declared in the file, rather than explicit dependencies.
The DAG file's top-to-bottom order has no runtime meaning; only >>, set_upstream/set_downstream, or list expansion define edges. Candidates who test with a single linear file often pass locally then fail on branching questions.
What to do instead
Always wire edges explicitly (a >> b, or a >> [b, c] >> d). When reading a DAG snippet, reconstruct the graph from the dependency lines only, and remember that tasks with no path between them can run in parallel.
Using a Sensor without a timeout or with the default reschedule/poke mode misunderstood, causing a worker slot to be blocked indefinitely.
Sensors in poke mode occupy a worker slot for their whole wait; the default timeout is generous (7 days) and mode defaults vary by version, so candidates assume sensors are 'free'.
What to do instead
Set timeout and mode explicitly: mode='reschedule' frees the slot between pokes, and poke_interval controls frequency. Pair with an SLA or timeout so a stuck sensor fails rather than hangs. Know that reschedule mode requires the scheduler to re-queue the task.
Hardcoding connection credentials or environment-specific values in DAG code instead of using Connections and Variables.
Tutorials often show literal hostnames and passwords, and the UI/CLI surfaces make Variables and Connections easy to overlook; candidates then miss questions about where secrets belong or how to reference them.
What to do instead
Create Connections (conn_id) for external systems and reference them via the operator's conn_id or BaseHook.get_connection. Use Variables for non-secret config, read with Variable.get (and template {{ var.value.x }}). Remember secrets belong in a secrets backend, not in the DAG file or the Variable table for sensitive data.
Misreading CLI commands, using 'airflow tasks run' or 'airflow dags trigger' with wrong arguments, or expecting the CLI to validate the DAG's logic.
The CLI has many subcommands (dags list, dags trigger, tasks test, tasks run, dags backfill) with subtly different semantics; candidates blur 'test' (runs locally, ignores dependencies and DB state) with 'run' (executes a real task instance).
What to do instead
Learn the distinctions: 'airflow tasks test' executes a single task locally without recording state or checking upstreams; 'airflow tasks run' runs a real task instance; 'airflow dags trigger' creates a DagRun; 'airflow dags list-import-errors' surfaces parse failures. Match the command to the intent in the question.
Debugging a failing DAG by only looking at the DAG's overall state in the UI, missing the per-task logs and the import-error view.
A DAG can show as failed because one leaf task failed, or not appear at all because of a parse error; the UI's grid/graph view and the 'Import Errors' page are separate from task logs.
What to do instead
Work the triage path: check DAGs list for import errors first, then open the specific DagRun, select the failed TaskInstance, and read its Log tab (or 'airflow tasks logs'). Distinguish scheduler-level problems (DAG missing) from task-level failures (traceback in logs).
Treating a DAG as a place for long-running streaming or transactional logic, or assuming Airflow retries tasks exactly-once.
Airflow orchestrates, it does not process data; tasks can be retried and may partially complete, so non-idempotent tasks duplicate side effects. Candidates who miss this pick 'use Airflow to process the stream' style answers.
What to do instead
Design tasks to be idempotent (overwrite partitions, upsert by key, use unique run identifiers). Use Airflow to trigger external systems (Spark, dbt, warehouses) rather than doing heavy compute in PythonOperator. Know that retries re-execute the whole task, and use the logical date/run id to make writes safe.