Connections and Variables That Leak or Drift

The password lives in two places, and rotation updated only one. The PostgresOperator read the Connection, which had the new password; the PythonOperator read the Variable, which still had the old one. When the warehouse invalidated the old password, the Variable-based task failed. This is the ordinary failure mode of duplicated secrets: two sources of truth drift, and the drift is invisible until an authentication error surfaces, often weeks later and in a different DAG than the one that was 'fixed'.

The underlying mistake is using a Variable for a credential at all. Airflow has a purpose-built abstraction, the Connection, which bundles host, schema, login, password, port and extra, is consumable by operators and hooks without custom code, and is the object the UI, CLI and secrets backends are designed around. A password in a Variable:

• is not treated as a secret by default (a plain Variable is visible in the UI and in the metadata database in clear text unless encryption is configured);
• is not covered by the secrets-backend lookup path in the same way;
• has no standard consumer, so every task re-implements retrieval and error handling.

The fix is to delete the Variable, have the PythonOperator obtain credentials via PostgresHook(postgres_conn_id=...), and rotate in exactly one place. If a non-connection secret is genuinely needed, it should be a secret Variable or, better, fetched from a secrets backend (Vault, AWS Secrets Manager) so rotation is centralized.

The objection is correct and the impact is concrete. The DAG file is parsed by the scheduler repeatedly - at least every min_file_process_interval (default 30 seconds) and on every file-change detection - and parsing executes top-level code. A Variable.get() at module level therefore issues a metadata-database query on every parse, for every DAG file, whether or not any task is running. With many DAG files this becomes a steady, pointless load on the metadata database and can slow the scheduler's parse loop, delaying task scheduling for everyone.

There is a second, subtler problem: the value is frozen at parse time in the serialized DAG. If the Variable changes, running tasks may still see the old value until the DAG is re-parsed and re-serialized, producing confusing 'I changed it but nothing happened' behaviour.

The fix is to move retrieval inside the task, where it executes once per task run and reads the current value:

def run(**context):
    pwd = Variable.get('warehouse_password')

or use a templated field, which is rendered at task execution: {{ var.value.warehouse_password }}. For values that change rarely and are needed at parse time (e.g. a config blob), the sanctioned pattern is a separate small DAG or a startup job that fetches and caches, not an inline call in every DAG file. And as noted above, credentials belong in Connections, not Variables, which removes this instance entirely.

Use environment-scoped Connections and Variables rather than branching in DAG code.

The mechanism: give each environment its own Airflow deployment (or at minimum its own metadata database), and name Connections with an environment-neutral id such as warehouse_default. Each deployment's warehouse_default points at that environment's host and credentials. The DAG code references warehouse_default and never knows which environment it is in. Non-secret settings (bucket names, table prefixes) go in Variables with the same neutral names, or in a config file loaded per deployment.

If a single deployment must serve both, prefix the ids (warehouse_staging, warehouse_prod) and select via an environment variable read once at DAG-parse time - but be aware this makes the DAG definition environment-dependent, so the UI shows only one environment's view and a misconfiguration can point production code at staging.

Risks that remain:

• Silent misconfiguration: a staging deployment with a production Connection id will happily write to production. Guard with a startup check that asserts the expected host, and with least-privilege credentials per environment.
• Secret sprawl: two deployments mean two places to rotate. A shared secrets backend with environment-scoped paths reduces this.
• Divergent Variables: a Variable added to production but not staging fails only at runtime; keep the set of required Variables in code and validate at DAG parse or in a startup task.
• Testing gap: staging data volumes differ, so a DAG that passes staging can still time out in production; do not treat a green staging run as proof.