XCom Payload Bloat
A pipeline extracts a 2 GB Parquet file from object storage, then passes the file path and a 40 MB summary dictionary to a downstream task via xcom_push. The DAG runs daily. After two weeks, the metadata database has grown by 30 GB and the scheduler UI is slow. The team uses the default SQLite-backed metadata database in a small deployment. The downstream task reads the summary dictionary from XCom and writes it to a report. The team wants to keep passing the summary but reduce database pressure. They also ask whether they can pass the 2 GB file itself through XCom to avoid re-downloading.
1. Diagnose the source of the metadata database growth and explain why passing the 2 GB file through XCom is a bad idea.
Read the worked answer
XCom values are serialized and stored in the metadata database (or an external backend if configured). A 40 MB dictionary pushed daily for 14 days is roughly 560 MB before serialization overhead, and depending on the serializer (pickle, JSON) the stored size can be larger. The 30 GB growth suggests either larger payloads than reported, repeated pushes per task attempt, or both. SQLite is especially sensitive because it does not handle large blobs or concurrent writes well, which explains the slow UI.
Passing the 2 GB file through XCom is worse: XCom is designed for small metadata, not bulk data. It would be serialized into the database, exceed typical column or row limits, slow every scheduler query, and risk corruption. The correct pattern is to pass a path or URI and let the downstream task read from object storage. If re-downloading is a concern, use a shared volume or a caching layer, not XCom.
2. Recommend a concrete design that keeps the summary available downstream while reducing database pressure.
Read the worked answer
Write the summary to durable storage (object storage, a database table, or a shared volume) and push only a small reference through XCom, such as a URI or a row key. The downstream task reads the reference and fetches the summary. This keeps XCom payloads in the kilobyte range.
Alternatively, if the summary must be in XCom, compress and trim it to only the fields the report needs, and ensure the task returns it rather than pushing multiple times per attempt. Set do_xcom_push=False on tasks that do not need it. Configure an external XCom backend (for example, one backed by object storage) if large values are unavoidable.
Operationally, migrate off SQLite to PostgreSQL for any production workload, enable XCom retention or cleanup, and monitor xcom table size. Add a test that asserts pushed payload size stays under a threshold.