Airflow DAG Authoring common mistakes
Airflow DAG Authoring · Understand the traps and how to avoid them.
Writing the DAG's schedule_interval (or schedule) as a naive datetime or a cron string while assuming the interval start equals the run's logical date, then being surprised that execution_date/logical date is the start of the window rather than the moment the run starts.
Candidates carry over intuition from cron or from 'run at time T' systems; Airflow's data-interval semantics mean a run covering 2024-01-01T00:00 to 2024-01-02T00:00 has logical date 2024-01-01T00:00 and only starts after the interval closes.
What to do instead
Practice mapping any schedule to its data interval: compute the interval, state the logical date, and state the earliest possible start. Use {{ data_interval_start }} and {{ data_interval_end }} in templates instead of arithmetic on ds, and remember that a daily job scheduled for midnight processes the previous day's data.
Putting heavy work at DAG-parse time: opening database connections, calling APIs, running queries, or reading large files at module level so every scheduler parse does it again.
Candidates treat the DAG file like a normal script and want 'setup' done once; they don't realize the scheduler re-parses files repeatedly (every min_file_process_interval by default) in the parsing process, not the worker.
What to do instead
Keep top-level code limited to imports, DAG construction, and operator instantiation with static arguments. Move all I/O into operator execute bodies or hooks inside tasks; if metadata must be fetched at parse time, cache it in a file or Variable and refresh it in a separate task or a separate process.
Assuming task order is determined by the order operators are written in the file, or connecting dependencies only in one direction and expecting a fan-in to wait correctly.
The DAG's structure is defined solely by >> and set_upstream/set_downstream (or the newer dependency APIs); file order is irrelevant, and a common slip is writing a >> b >> c but leaving a parallel branch d unconnected to the join.
What to do instead
Draw the intended graph, then express it explicitly with bitshift chains, lists, and cross_downstream/chain as needed. After writing, inspect the rendered graph view or run a quick structural check asserting expected upstream/downstream sets for each task.
Using a fixed, non-idempotent output path or an append-mode write so backfills and retries duplicate or corrupt data.
Candidates think of a task as running once, forgetting that Airflow can run the same logical date again after a clear, a retry, or a backfill, and that parallel runs may overlap.
What to do instead
Make every task idempotent: partition outputs by logical date or data interval (for example, a path containing {{ ds }}), write with overwrite or MERGE semantics, and design deletes/inserts to be safe to repeat. Test by clearing a task and rerunning it twice.
Passing large payloads through XComs (DataFrames, file contents, big lists) and expecting them to be fast and reliable.
XComs are convenient and look like normal function returns, so candidates use them as a general data channel; in reality they are serialized into the metadata database and subject to size limits and performance costs.
What to do instead
Use XComs only for small control values (IDs, paths, counts). Write large data to object storage or a database in one task and pass the URI or key forward; if the default SQLite/Postgres backend is a concern, configure a custom XCom backend for larger objects.
Writing PythonOperator callables that take no arguments, or that try to accept arbitrary kwargs, and then failing when Airflow passes context or when pulling an XCom.
Candidates forget that providing context requires declaring **context (or specific keys) in the signature, and that op_kwargs/op_args are separate from template-rendered arguments.
What to do instead
Declare the exact context keys needed, e.g. def run(ds, ti, **context), and use ti.xcom_pull(task_ids=...). Remember that templated fields are rendered before the callable runs, so prefer passing rendered values via op_kwargs when possible.
Setting retries on the DAG default_args but not accounting for retry_delay, retry_exponential_backoff, or max_retry_delay, and setting sla on the task instead of the DAG-level sla parameter or vice versa.
Default arguments are easy to set once and assume they propagate everywhere, and SLA configuration is genuinely split: sla is a task-level argument, while the SLA miss callback is DAG-level. Candidates also confuse retries with the task timing out.
What to do instead
Set default_args for retries and retry_delay, override per task where needed, and add execution_timeout for runaway tasks. Put sla=timedelta(...) on the task and sla_miss_callback on the DAG; verify with a deliberately failing task that retries occur and the callback fires.
Building dynamic DAGs by generating tasks in a loop with names based on runtime data, or creating DAG objects dynamically without stable dag_id values, causing duplicate or missing DAGs and scheduler confusion.
Candidates want one task per file or per table and reach for loops and dynamic task creation; they don't realize the DAG structure must be stable at parse time and that dag_id collisions overwrite each other.
What to do instead
For a known, bounded set of items, generate tasks at parse time with deterministic IDs (for example, f"process_{table}") and use dynamic task mapping (expand/partial) for runtime-sized collections. Keep dag_id values unique and derived from stable config, not timestamps.
Assuming a mapped task's downstream receives a list it can iterate directly, or using expand with a non-list and expecting per-item tasks without checking the mapping length.
Dynamic task mapping creates one task instance per input element and the XCom is aggregated as a list, but candidates often write downstream code that assumes a single value or forget that an empty input produces zero mapped instances.
What to do instead
Use ti.xcom_pull(task_ids='mapped_task', map_indexes=...) or pull the aggregated list explicitly, and test with zero, one, and many inputs. Guard downstream logic against empty mapping results and use partial() for constant arguments.
Debugging by running the whole DAG locally with airflow dags trigger and waiting, rather than testing a single task with airflow tasks test or a unit test of the callable.
Candidates treat the scheduler as the debugging loop, which is slow and obscures whether the failure is in the task logic, the dependency graph, or the scheduler; they also skip checking that the DAG imports cleanly with a parse test.
What to do instead
Use airflow tasks test <dag_id> <task_id> <logical_date> for isolated execution, run python -c 'import dag_module' or a DagBag load to catch import errors, and unit-test Python callables directly with mocked hooks. Inspect task logs and the rendered template view for templating errors.