Airflow DAG Authoring: Connections, Hooks, and Variable Management

Connections and hooks

A Connection stores host, login, password, port, schema, and extra (JSON) under a conn_id. Hooks are the programmatic interface: BaseHook.get_connection('conn_id') retrieves it, and specific hooks (PostgresHook, S3Hook, HttpHook) wrap it with client logic. Operators accept conn_id and instantiate hooks internally.

Variables are key-value pairs stored in the metadata DB, accessed via Variable.get('key'), {{ var.value.key }}, or {{ var.json.key }} for JSON values.

Traps candidates hit

• Calling Variable.get() at DAG parse time hits the DB on every scheduler loop; use templated {{ var.value.x }} so lookup happens at runtime.
• Secrets belong in a secrets backend (Vault, AWS Secrets Manager), not in Variables or code. Connections support the same backends.
• extra is JSON, so conn.extra_dejson['region'] is the safe accessor.
• Environment variables (AIRFLOW_CONN_<CONN_ID>) can define connections and override the DB; the name is uppercased with underscores.
• Variable scope is global; naming collisions across teams are common, so prefix keys.

Rule of thumb

Connections for credentials and endpoints, Variables for small configuration, secrets backend for anything sensitive. Never hardcode either in a DAG file.

Worked example

Instead of bucket = Variable.get('bucket') at module level, write S3KeySensor(..., bucket_key='{{ var.value.bucket }}/{{ ds }}.csv'). The lookup happens per task run, and changing the Variable takes effect without a code deploy.