I85PREP
⌘ K
Airflow DAG Authoring study materials
Airflow DAG Authoring study resources

Airflow DAG Authoring: Data Passing with XComs and Templating

Airflow DAG Authoring · Work through your exam, one topic at a time.

All guide sections

Data Passing with XComs and Templating

XComs: small values, explicit keys

XComs move small serializable values between tasks. A task pushes with task_instance.xcom_push(key, value) or by returning a value (stored under key return_value); downstream tasks pull with xcom_pull(task_ids=..., key=...). Values are stored in the metadata database, so they must be JSON-serializable and small.

Templating uses Jinja. Fields marked template_fields on an operator are rendered at runtime with context variables: {{ ds }}, {{ ds_nodash }}, {{ ts }}, {{ data_interval_start }}, {{ data_interval_end }}, {{ params.my_param }}, {{ var.value.my_var }}, {{ ti.xcom_pull(task_ids='x') }}, and {{ dag.dag_id }}.

Traps candidates hit

  • Passing DataFrames or large payloads via XCom bloats the metadata DB; use object storage and pass a path.
  • Pulling before the producer runs returns None, not an error - a silent bug.
  • xcom_pull without task_ids in a mapped or multi-upstream context may pull the wrong value; be explicit.
  • Templating only works on template_fields. Putting {{ ds }} in a non-templated argument leaves it literal.
  • params must be supplied via params={...} on the DAG or task; {{ params.x }} fails if x is undefined.
  • Jinja renders to strings, so numeric arguments may need explicit casting.

Rule of thumb

XComs are for metadata and pointers, not data. If the value is bigger than a few kilobytes, write it to storage and XCom the URI.

Worked example

Producer: return {'path': 's3://bucket/2024-03-01.parquet'}. Consumer: path = ti.xcom_pull(task_ids='extract')['path']. In a templated field: bash_command='process.sh {{ ti.xcom_pull(task_ids="extract")["path"] }}'.