Payment API fails under load and the vendor submits a change order
A 14-person distributed team is building a payment API for six downstream systems. The vendor is in a timezone 7 hours behind you. At the sprint 6 demo, 3 of 8 endpoints fail under 200 concurrent requests. The vendor says non-functional requirements were never in the statement of work; your architect points out that the SOW requires 'industry-standard performance for the stated load'. The vendor's key developer has resigned with 4 weeks' notice. The vendor submits a change order for £90,000 and 5 weeks. Your business owner says to accept it because the release cannot slip. The contract has a 10-day dispute window, which closes in 6 days. Two downstream teams have already started integration against the unstable endpoints. The steering committee meets in 4 days. You have no spare test environment. What do you do?
1. What are your first actions in the next 48 hours? Distinguish containment from diagnosis.
Read the worked answer
Containment first, because the damage is spreading.
- Freeze new integration work against the three unstable endpoints. Issue a short technical notice naming the affected endpoints and the safe contract version or mock to use.
- Stop any planned load testing on shared environments until a controlled test window is agreed. Lack of a spare environment is a constraint, not a reason to test informally.
- Preserve evidence: test scripts, logs, environment configuration and the exact conditions that produced the failures.
- Serve written notice to the vendor within the 10-day dispute window, reserving rights. Do not let the window expire while the technical argument continues.
- Assign one owner for vendor communication so the vendor does not receive conflicting messages from architects, testers and business owners.
Diagnosis second. Reproduce the failure with the architect and vendor present, using the same 200-concurrent-user profile. Agree the test method before interpreting results. Separate three questions: is this a defect, a missing requirement, or an ambiguous requirement? Compare the SOW performance clause with the demo evidence. Do not accept the change order before this distinction is documented. The resignation is a vendor resourcing risk; it does not automatically make a defect chargeable.
2. How should you evaluate the vendor's £90,000 and 5-week change order? What evidence and contractual positions matter?
Read the worked answer
Evaluate it against four tests.
- Scope test. Does the SOW already require this performance? 'Industry-standard performance for the stated load' is ambiguous but not empty. Get the commercial or legal view and compare it with the load profile in the requirements. If the requirement is already in scope, the change order is a defect fix, not new work.
- Effort test. Ask for a breakdown by activity, rate and assumption. £90,000 and 5 weeks must be traceable to work, not a round number. Check whether the vendor is pricing the resignation and its own rework into your change.
- Alternatives test. Ask the architect for options: caching, load balancing, connection pooling, phased endpoint release or a temporary throttle. Compare their cost and risk with the change order. Do not accept the first price.
- Delay test. Quantify the business cost of 5 weeks against the change order. The business owner's 'cannot slip' statement is a constraint to test, not a blank cheque.
Negotiate from evidence: vendor fixes the defect under warranty, change order only for genuinely new scope, and acceptance withheld until the load criteria are met. Record a reservation of rights and keep the dispute process alive. Also require a named replacement for the resigning developer and a knowledge transfer plan.
3. Draft the core messages for the downstream teams and the steering committee. Include what you do not yet know.
Read the worked answer
To downstream teams: 'Endpoints A, B and C are not stable under 200 concurrent requests. Do not build production integrations against them. Use the published mock version 1.2 and join the daily 30-minute integration call. We will confirm whether this is a defect or a change in 5 working days. No change to your own release dates is agreed yet.'
To the steering committee: 'At sprint 6, three of eight endpoints failed under the contractual load profile. The vendor has submitted a change order for £90,000 and 5 weeks. We have frozen new integration against the affected endpoints and served notice within the dispute window. We do not yet know whether the SOW already covers this performance, whether the vendor's price is justified, or the final schedule impact. We need a decision on negotiation authority and a contingency position, not a decision to accept today. Options are: require vendor fix under warranty, negotiate a reduced change order, or accept a phased release. Recommendation: authorize technical and commercial evaluation, with a decision at the next checkpoint in 5 days.'
Be explicit about unknowns. A steering committee can act on known facts and a decision timetable; it cannot act on blame.