Supplier Selection: Escape Cost Versus a Hard Submission Window
You are selecting a supplier for an 85 KLOC embedded data-parsing module. Vendor A quotes $210,000 with a historical first-six-month escape rate of 0.9 defects per KLOC; it has been late on 3 of its last 4 deliveries by 2 to 6 weeks. Vendor B quotes $165,000 with an escape rate of 2.4 defects per KLOC; it has delivered on time 3 of 3 times but has no process appraisal, and its last delivery produced two severity-1 field defects. Your internal cost to diagnose and fix an escaped defect averages $3,200. The module sits on the critical path of a release candidate that must enter a regulatory submission in week 14; the program manager has one week of schedule float and estimates that missing the submission window costs about $1.9M in delayed revenue and rework. Vendor B's proposal claims 82% statement coverage from its unit tests.
1. Build the comparison. Which vendor do you select, and what does the arithmetic show?
Read the worked answer
Combine escape cost with schedule exposure, because the two are not on the same scale - the decision turns on the second, not the first.
Expected escape cost:
- Vendor A: 0.9 defects/KLOC × 85 KLOC = 76.5 defects × $3,200 = $244,800. Total ≈ $454,800.
- Vendor B: 2.4 defects/KLOC × 85 KLOC = 204 defects × $3,200 = $652,800. Total ≈ $817,800.
On escapes alone, A is ahead by about $363,000.
Schedule exposure. One week of float; A was late by 2-6 weeks on 3 of 4 deliveries. Any slip beyond one week triggers the step-function cost of a missed submission, roughly $1.9M. Using the historical record as a crude frequency estimate, P(slip beyond float) for A is around 0.75, so expected schedule cost ≈ $1.42M. For B, with 3 of 3 on time, a conservative estimate might be 0.10, or ≈ $190,000.
Totals: A ≈ $454,800 + $1,425,000 ≈ $1.88M; B ≈ $817,800 + $190,000 ≈ $1.01M. Select B, or select A only if the schedule exposure is contractually removed.
Note the sensitivity: the decision is not robust to the schedule assumptions. If A could guarantee delivery with two weeks of buffer, or if the submission cost were, say, $200,000 instead of $1.9M, A becomes the cheaper choice. State that explicitly rather than presenting a single number as destiny.
2. How would you treat the data quality behind these figures, and what contractual or technical actions would you take to reduce the risk in whichever vendor you choose?
Read the worked answer
Treat both escape rates as weak estimates and design the contract so the decision can be corrected rather than locked in.
Data quality first. A's 0.9 and B's 2.4 derive from a handful of deliveries - four and three - so the confidence intervals around them overlap substantially. The apparent 2.7× difference may not be statistically meaningful. Also interrogate comparability: were the escape rates measured over the same window, with the same severity inclusion rule, on similar domain complexity and similar KLOC? An escape rate that excludes severity 3 or that counts only customer-reported issues is not the same metric. Ask both vendors for defect data by severity, phase of injection and phase of removal, plus latent-defect estimates from their own testing.
Contract actions regardless of vendor:
- Incremental delivery: require a working build at weeks 6 and 10, not a single delivery at week 14. This converts schedule risk from a step function into an early warning.
- Acceptance gates tied to objective criteria, with defined remedies: quantitative escape thresholds for the first 90 days, warranty repair obligations, and a defined severity-1 response time.
- For Vendor A, a liquidated-damages clause or a schedule guarantee would be the only basis for choosing it despite its record.
- Require access to process evidence (review records, test results, defect data) and the right to audit, particularly important for B, which has no appraisal.
The general principle: never let a selection decision depend on a single point estimate from small samples when you can instead structure the relationship so that being wrong is recoverable.
3. The team proposes accepting the module on the strength of the 82% statement coverage claim. What does that figure tell you, and what incoming verification would you actually specify?
Read the worked answer
82% statement coverage tells you that 18% of statements were never executed. It says nothing about the quality of the tests that ran, and it is a self-reported figure.
What it does not establish:
- Control-flow adequacy. Statement coverage is the weakest structural criterion. Branch or decision coverage, and multiple-condition coverage for the parser's decision logic, can be far lower than 82% on the same code base.
- Requirement adequacy. Covered statements may be exercised only by the happy path; error handling, malformed input and boundary conditions are typically the uncovered remainder and the richest source of parser defects.
- Assertion quality. Executing a statement without checking the result produces coverage with no detection power.
- Truth of the claim. Self-reported coverage is unverified. Require the instrumented run, the coverage tool and version, the configuration used, and the raw report.
Specify instead:
- Re-run coverage independently on the delivered source with your own tool; set an acceptance threshold on branch coverage, not statement coverage, with the gap in high-risk parsing functions explicitly itemized and justified.
- Requirements traceability from every requirement to at least one test, including negative and malformed-input cases.
- Risk-based inspection rather than statistical sampling of code: inspect 100% of the input-validation and numeric-conversion functions, plus a random sample of modules, against a defined checklist. Sampling defect counts across 85 KLOC is not a reliable latent-defect estimator.
- If formal acceptance sampling is used, define the lot, the sample size, the acceptance number and the resulting operating characteristic curve so both parties understand the probability of accepting a bad lot.