Basics of XCom
What XCom is
XCom (cross-communication) lets tasks exchange small pieces of data through the metadata database. A task pushes a value with ti.xcom_push(key=..., value=...) or by returning it (TaskFlow does this automatically); another task pulls it with ti.xcom_pull(task_ids=..., key=...). XComs are keyed by DAG, task, run, and key, so they are scoped to a specific DAG run - a task cannot pull another run's XCom without explicitly targeting it.
Limits and traps
The most tested trap: XComs are for small metadata, not dataframes or files. Values are serialized into the metadata database, so large payloads bloat the DB and slow everything down. Pass a path, URI, or identifier instead and let the next task read the real data from storage. Another trap is assuming XComs are available across DAGs by default - they are not; cross-DAG data should go through storage or explicit configuration. Also, a task can only pull from upstream tasks that have already run in the same run; pulling from a not-yet-run task yields None.
Worked example
Task A writes a file to object storage and returns its key. Task B pulls that key via xcom_pull(task_ids="extract") and reads the file. The XCom carries the pointer, not the payload - the correct pattern.
Rule of thumb
If the value is bigger than a few kilobytes, store it externally and pass a reference through XCom.
Connections, Variables & Sensors
Connections and Variables
Connections store credentials and endpoints (host, login, password, port, extra) under a conn_id, and operators reference them by ID rather than hardcoding secrets. They can be managed in the UI, CLI, environment variables, or a secrets backend. Variables are key-value configuration stored in the metadata database and read with Variable.get("key"). A crucial trap: calling Variable.get() at DAG top level hits the database on every scheduler parse; use templated {{ var.value.key }} or pass values into operators instead. Mark secrets as such so the UI masks them.
Sensors
A sensor is an operator that waits for a condition: FileSensor, S3KeySensor, ExternalTaskSensor, SqlSensor. Sensors occupy a worker slot while waiting. Two controls matter: poke_interval (how often to check) and timeout (when to fail). Deferrable sensors (deferrable=True) hand waiting to the triggerer, freeing the worker - the preferred modern pattern. mode="reschedule" releases the slot between pokes instead of holding it (mode="poke").
Traps to watch
Don't confuse Variables (general config) with Connections (credential bundles). Don't assume a sensor fails immediately when data is missing - it waits until timeout. Don't use long poke_interval values expecting sub-minute latency. And remember that Variable.get() in a top-level expression is a performance and correctness hazard.
Rule of thumb
Secrets → Connections; tunable config → Variables (templated); waiting → deferrable sensors with a timeout.