A Mutable Default Argument That Only Bites on a Live Worker
A payments service exposes a Python function def add_fee(items, fee, ledger=[]) that appends a fee record to ledger and returns it. Under load, fee records from one customer's transaction appear on another customer's receipt, roughly 3 times per 10,000 requests, always on the same worker process, and only after that worker has handled a refund. The unit suite is green: 214 tests, 87% line coverage. Each test builds its inputs in setUp and asserts on the returned list, and CI runs all tests in a single process in a fixed order with a gc.collect() between modules. The team proposes three responses: (a) change the signature to ledger=None and default inside the function, (b) add a regression test that calls add_fee twice, and (c) add pytest-forked so each test runs in its own process. The release manager wants one change before the freeze and a follow-up ticket for the rest.
1. Identify the defect mechanism and explain why 214 passing tests at 87% line coverage did not detect it.
Read the worked answer
Mechanism. A default argument is evaluated once, at function-definition time, and the resulting object is stored on the function (add_fee.__defaults__). Every call that does not pass ledger appends to that same list, for the lifetime of the process. The list is therefore process-global mutable state wearing the costume of a local variable.
This explains the three observed symptoms precisely:
- Same worker process - the state lives in that interpreter, so only that replica is affected.
- After a refund - the refund path is the first code path that calls
add_feewithout supplying a ledger; it seeds the shared list, and subsequent calls inherit the contents. - Rare - most callers pass an explicit ledger, so only the no-argument path leaks.
Why the suite is silent. Coverage measures which lines executed, not which state interactions occurred. Each test asserts on the returned list; if that list already contains a previous test's records, assert fee in result and assert len(result) > 0 still pass. setUp rebuilds inputs but never resets the function object. gc.collect() cannot free an object still referenced by a live function. Fixed ordering makes the suite deterministic, which converts an order-dependent leak into a stable green run.
2. Evaluate the three proposals. Which is the actual fix, which is a real regression test, and which is mitigation?
Read the worked answer
(a) is the correct fix, but only if written carefully. def add_fee(items, fee, ledger=None): if ledger is None: ledger = [] is right. The common shorthand ledger = ledger or [] is a second bug: a caller who deliberately passes an empty list to collect into would have it silently replaced, so their container never receives the records. Use the is None test.
(b) is a genuine regression test only if it is designed to fail on the old code. Calling twice and asserting each result has one element fails correctly today. Asserting only that the fee is present does not. Stronger: assert the two returned lists are distinct objects, and add a test that passes an explicit empty list and asserts the caller's list was mutated.
(c) is mitigation, not a fix. Process isolation would hide this and every similar leak while slowing CI. It also removes the cross-test contamination signal that is currently the only thing that could have caught the bug. Legitimate as a short-term diagnostic, wrong as a permanent setting.
What to ship before the freeze: (a) plus (b) - small, isolated, reversible. Follow-up ticket: a lint rule for mutable defaults (flake8-bugbear B006 / ruff B006), randomized test ordering in CI, and an audit of other functions with container defaults.
3. Beyond the code fix, what single test-suite invariant would most reduce the chance of this class of bug recurring, and what does it cost?
Read the worked answer
Invariant: the suite must pass in random order and in isolation. Run the full suite with randomized ordering (and a recorded seed so failures reproduce) and also run each test module alone. Any test that depends on residue from another test then fails loudly instead of passing by luck.
Why this is the highest-leverage change. The fixed ordering was a necessary condition for this bug to survive 214 tests. State leaks - mutable defaults, module-level caches, class attributes, singletons, un-reset fixtures, environment variables - all present as order dependence. A random-order run converts them from latent production incidents into reproducible CI failures at the moment they are introduced.
Costs, stated honestly.
- Genuine order dependence must be fixed rather than suppressed, so expect a burst of red builds when the invariant is first switched on; budget a day for cleanup.
- Randomized order makes failures non-deterministic for developers unless the seed is printed and replayable - make that a hard requirement of the harness.
- Runtime is unchanged; the added cost is developer attention, not compute.
Complementary, not sufficient: a lint rule catches mutable defaults specifically but not the wider class; process isolation hides the class; higher coverage is irrelevant, since the bug lived inside lines that were already executed.