Airflow DAG Authoring: Operators and Task Construction

Choosing and configuring operators

An operator is a template for a task; instantiating it creates a task in the active DAG. Know the families: action operators (PythonOperator, BashOperator, SQLExecuteQueryOperator), transfer operators, sensor operators (ExternalTaskSensor, FileSensor), and the decorator style (@task, @task.bash).

Key constructor arguments to recognize: task_id (required), retries, retry_delay, execution_timeout, pool, queue, depends_on_past, trigger_rule, do_xcom_push, and op_kwargs/op_args for Python callables.

Traps candidates hit

• python_callable must be a reference, not a call. PythonOperator(..., python_callable=my_func()) executes at parse time and passes the return value. Correct: python_callable=my_func.
• BashOperator uses bash_command, and a trailing space is required if the command is templated with {{ ... }} and followed by nothing else - a classic gotcha in older versions.
• Sensors block a worker slot. Prefer deferrable sensors (deferrable=True) or mode='reschedule' to free the slot between pokes.
• provide_context is obsolete in modern Airflow; context is injected automatically when the callable accepts **context or named context keys.
• Custom operators subclass BaseOperator and must call super().__init__(**kwargs) before using self.

Rule of thumb

One operator should do one thing. If a Python callable contains branching logic that decides which downstream work happens, that is a control-flow problem (use branching or dynamic task mapping), not an operator configuration problem.

Worked example

BashOperator(task_id='load', bash_command='python load.py --date {{ ds }} ') - note the trailing space after the template, which prevents the rendered command from being concatenated with stray arguments.