Airflow Fundamentals case studies
Airflow Fundamentals · Put your knowledge to work in context.
The Level 1 Apache Airflow 3 Fundamentals credential is an entry-level certification offered by Astronomer, the company behind the Astro platform, through its Astronomer Academy. It is designed for practitioners with roughly three or more months of hands-on Airflow experience who want to demonstrate that they understand core Airflow concepts and can build, schedule, and monitor basic pipelines. The Airflow Fundamentals case studies are 5 cases and about 4,614 words.
1The DAG That Runs Twice a Day and Reads Yesterday's Files
A data team runs a DAG that ingests a partner's CSV drop. The partner uploads files to an S3 prefix once per day, but the exact time varies between 01:00 and 04:00 UTC, and occasionally a file arrives late at 07:00.
917 words, about 5 min2A PythonOperator That Works Alone and Fails in the DAG
An engineer writes a three-task DAG: extract returns a dict of row counts, transform is supposed to use those counts to decide how many partitions to build, and load writes the result. Each function is tested locally by calling it directly and all three pass.
893 words, about 4 min3The Sensor That Ate the Worker Pool
A DAG has twelve parallel sensor tasks, each waiting for a partition to appear in a Hive metastore. Each sensor uses poke interval=60 and the default mode='poke' , with no timeout.
950 words, about 5 min4Connections and Variables That Leak or Drift
A team stores its warehouse password in an Airflow Variable named warehouse password , marked as a normal (non-secret) variable, and reads it in a PythonOperator with Variable.get('warehouse password') . The same password also exists as a Connection of type Postgres used by a PostgresOperator .
921 words, about 5 min5The Backfill That Doubled the Revenue Table
A DAG computes daily revenue and appends rows to revenue daily with a plain INSERT, keyed by nothing. It has run successfully every day for 90 days.
933 words, about 5 min