The Workflow That Got Slower Every Week
A workflow that previously ran in 8 minutes now takes 35 minutes and the slowdown has grown gradually over two months. It reads a transactional table, filters to the last 90 days, joins to two dimensions, and writes a file. Nothing in the workflow logic has changed. Row counts in the source have grown about 15% per month. The team's first instinct is to add more compute to the environment. There is a filter step that runs after the join rather than on the source read.
1. Where would you look first, and why is 'add compute' a weak first move?
Read the worked answer
Look at where the work happens relative to the data volume, not at the cluster size. Two red flags:
- The 90-day filter runs after the join, so the join processes the full history rather than the filtered window. As history grows 15%/month, the join's input grows even though the relevant data (last 90 days) is roughly flat. This is the classic cause of gradual, logic-unchanged slowdowns.
- The source read pulls the full table, so I/O and memory scale with total history, not with the window you need.
'Add compute' is weak because it pays to process data you do not need. It may mask the problem briefly, then the 15% monthly growth erodes the gain, and it does nothing about the structural inefficiency. Push the filter to the source read (or as early as possible) so the join and downstream steps see only the 90-day window; confirm by comparing row counts entering the join before and after the change.
2. After pushing the filter early, how would you verify the improvement is real and durable, and what else would you check?
Read the worked answer
- Measure the right thing: compare rows entering the join and total runtime before/after, and confirm the output is identical (same row count and a checksum or spot-check of totals). A faster wrong answer is not an improvement.
- Durability: re-run a week later; if runtime is now roughly flat despite source growth, the fix addressed the structural cause. If it creeps again, the filter is not actually being pushed down or another step scales with history.
- Other suspects: check the two dimension joins for fan-out or missing indexes/keys, confirm the output write is not rewriting the full file each run, and look for any step that sorts or aggregates full history before filtering.
- Monitoring: add a runtime and row-count trend so the next gradual regression is visible before users notice.
The tested judgement: diagnose scaling behavior against data growth before reaching for more resources, and prove the fix with output equivalence plus a trend check.