A Flaky Integration Neither Team Owns
Two teams share an integration between a platform API and a fulfillment service. Their contract test fails about 12% of runs across three sprints. Each team's dashboards show its own side as healthy: platform reports 99.95% availability with sub-100ms p99, fulfillment reports 99.9% with normal queue depth. The shared schema has one nullable field whose meaning is undocumented, and platform recently began sending it as empty string instead of null. Both teams agree the other side is responsible; the weekly release train keeps shipping, and each sprint the flake is deferred as "not reproducible in isolation." No one has run the test with both services instrumented end-to-end.
1. Both dashboards are green and the test fails 12% of the time. How do you localize the fault without relying on either team's metrics?
Read the worked answer
The reason both dashboards are green is that each measures inside its own boundary. A defect at the boundary - in serialization, schema interpretation, or timing between the two - is invisible to both by construction.
Localization approach:
- Instrument the seam, not the services. Log the exact payload at three points: platform egress, fulfillment ingress, and fulfillment deserialization. Compare them. The 12% of runs will show a divergence at exactly one of the three gaps, and that divergence identifies the layer.
- Correlate failures with payload shape. The nullable field is the obvious candidate: group the failing runs by whether the field arrived as null, empty string, or absent. A 12% failure rate with a sporadically empty field is a strong prior, and the grouping either confirms or eliminates it in an hour.
- Test the null-versus-empty hypothesis directly by replaying captured failing payloads against a stub. Replay removes the timing variable entirely.
- Check timing separately. If payload grouping shows no correlation, the failure is likely ordering or retry-related - rerun under artificial latency to reproduce it deliberately.
Why the current approach fails: "not reproducible in isolation" is an accurate finding reported as a dead end. It is evidence the defect is at the seam, not evidence the defect is imaginary. Treating it as the former is the error that has cost three sprints.
2. Who owns the fix? Reason about ownership rather than assigning blame.
Read the worked answer
Ownership should follow three things - who can change the behavior, who bears the consequence, and who can verify the fix - and those point in different directions here, which is why the dispute has not resolved itself.
Analysis:
- Platform owns the change in behavior. Sending empty string instead of null was a unilateral change to a shared contract. Even if it was technically permissible, it was not coordinated, and it correlates with the onset of flakiness.
- The contract owns the ambiguity. The field is undocumented, which means both interpretations were defensible. Fixing the sender without documenting the field leaves the next team free to make the same change again.
- Fulfillment owns the tolerance. A service that breaks on empty-versus-null for an undocumented field is brittle. It should not be the only defense, and today it is.
Resolution: platform reverts to null immediately as the short-term unblock and adds the field to the contract with explicit semantics. Fulfillment adds handling for both representations within the same release window. Ownership of the incident sits with platform, since its change is the proximate trigger; ownership of the defect class is shared.
What to avoid: assigning ownership by who has more capacity or who deferred last. Both produce a fix without a rule, and the same dispute recurs with a different field.
3. Design the prevention mechanism. What specifically stops the next three sprints from looking like these three?
Read the worked answer
The failure mode is not the nullable field; it is that ambiguity can persist unresolved while both sides stay green. Prevention must target that, not the field.
Mechanisms, in order of leverage:
- Make the contract executable and bidirectional. Both services run contract tests against shared fixtures; either side's change to serialization fails the other's build before merge. This alone would have caught the empty-string change on the day it shipped.
- A shared failure budget with a named owner. Define "same defect deferred twice" as an automatic escalation to a jointly named owner, not a standing agenda item. Deferral by default is a structural incentive, and it needs a structural counterweight.
- A seam dashboard. Emit the three-point payload comparison continuously in production, so boundary health is visible on someone's screen instead of being nobody's metric.
- Document every field's null semantics as part of the contract, with a rule that undocumented fields cannot be changed unilaterally.
Be honest about cost: mechanism 1 slows both teams' merges and requires ongoing fixture maintenance. It is worth it here because the seam has already consumed three sprints.
Success measure: time from first flake observation to a named owner, which should drop from three sprints to under a week. Track that, not the failure rate alone.