Choosing Between Two Vendors With Incomplete Evidence
You must select a vendor for a system handling regulated data. Vendor X scored 82/100 on your evaluation rubric, Vendor Y scored 79. Vendor X's score includes a 15-point 'innovation roadmap' category where their evidence was a slide deck; Vendor Y's includes a 12-point 'existing integrations' category backed by three reference customers. Vendor X has no current customers in your region. Vendor Y had a security incident 14 months ago, disclosed within 36 hours, with a public post-mortem. Your rubric weights were set before either vendor was known. The procurement deadline is in 6 days and re-tendering would take 10 weeks.
1. Evaluate whether the 3-point gap is meaningful, and recommend a vendor.
Read the worked answer
The 3-point gap is not meaningful, and the rubric is doing work it wasn't designed for.
Three points on a 100-point scale is within the noise of subjective scoring, and the two scores rest on very different evidence quality:
- Vendor X's 15 innovation points came from a slide deck - unevidenced claims. Strip or discount that category and X's score falls by up to 15, to ~67.
- Vendor Y's 12 integration points are backed by three reference customers - verifiable. That evidence survives scrutiny.
So the ranking likely reverses once you weight by evidence strength. The raw totals hide this because the rubric treats a slide deck and a reference customer as equivalent inputs.
Recommendation: Vendor Y, with conditions.
On the incident. A 14-month-old incident with 36-hour disclosure and a public post-mortem is a positive signal about their security culture, not a negative one. Most incidents are discovered late and disclosed reluctantly. Ask for the remediation evidence and confirm the specific control that failed is now tested.
On Vendor X. No current customers in your region is the decisive fact for regulated data - you would be their first, with no local reference for compliance handling and no proven support path in your jurisdiction.
On the deadline. Six days is enough to select Y; it is not enough to re-tender. Do not let the deadline force a choice you'd reverse with more information.
2. The rubric was fixed before vendors were known. Does adjusting for evidence quality violate that commitment, and how should you document the decision?
Read the worked answer
No - but the adjustment must be transparent and principled, not ad hoc.
Pre-committing weights protects against tailoring the rubric to a favored vendor. That protection is real and should be honored. But a rubric also assumes each category is scored on comparable evidence, and here it wasn't: a slide deck and three reference customers are not the same kind of input. Correcting for evidence quality is applying the rubric as intended, not rewriting it after the fact.
The line to hold: you may discount scores for weak evidence, but you may not re-weight categories because you prefer the outcome. Discounting X's unevidenced roadmap is defensible. Quietly raising the weight on integrations because Y is strong there would not be.
Documentation:
- Record the original scores and weights unchanged.
- Add an evidence-quality assessment per category: source, verifiability, independence.
- Show the adjusted comparison with the adjustment rule stated in advance of applying it.
- Note that the ranking reverses under the adjustment, and state plainly that the reversal drove the recommendation.
- Record the dissenting view if any evaluator disagrees.
- State the conditions on Vendor Y: remediation evidence, contractual security commitments, and a review at 12 months.
A regulator or auditor reading this should be able to reconstruct exactly why Y was chosen and see that the process was consistent. That is the test - not whether the weights were touched, but whether the reasoning is visible and honest.