38 Tickets, 40 Deploys a Day
An engineering organization has 140 repositories and a CI pipeline that builds 60 services. A critical vulnerability is published in a widely used JSON parsing library; 38 services depend on it, 12 of them transitively. The scanner automatically opened 38 tickets. The platform team argues most services never exercise the vulnerable code path because they parse only trusted internal messages. Two services are internet-facing and do parse untrusted input. Deployment is continuous, averaging 40 production deploys a day. The security champion wants the build to fail on any critical finding; the platform lead says that would halt nearly all deploys within a week and the team would simply route around the gate.
1. How do you triage the 38 findings, and what evidence justifies recording a service as not affected?
Read the worked answer
Triage on reachability and exposure, not on the scanner's severity label.
Tier 0 - patch now (hours). The two internet-facing services that parse untrusted input. This is not a debate; it is the exploit path the vulnerability describes. Include the 12 transitive consumers of those two in the immediate scope, since a version bump upstream may need a coordinated release.
Tier 1 - patch this sprint. Services that parse untrusted input of any kind (external API, user upload, partner feed) even if not internet-facing, and services where the library is loaded in a shared runtime that also handles untrusted data elsewhere.
Tier 2 - not affected, documented. Services that demonstrably do not reach the vulnerable function on untrusted input.
What constitutes adequate evidence for Tier 2 - the bar must be defined in advance or it becomes 'the developer said so':
- A data-flow statement naming the input sources the service accepts and showing they are internal/authenticated, with a named owner.
- Reachability analysis - a call-graph or dependency-scanning report showing the vulnerable function is not invoked, not merely that the package is imported.
- A machine-readable VEX record (or equivalent) in the repository so the finding is suppressed with a rationale rather than silently ignored by the next scanner run.
- An expiry date and review owner, because input sources change when someone adds a public endpoint six months later.
The platform lead's argument is plausible for the majority - but 'most likely not exploitable' is a hypothesis that must be tested per service, and the two internet-facing services prove the hypothesis is not universally true. Ship the patch for Tier 0 today, then spend the sprint proving Tier 2.
2. Design a pipeline gate that the team will not route around, and explain each design choice.
Read the worked answer
A gate that blocks everything on day one will be disabled by day seven. The design goal is a gate whose failures are rare and believed.
- Block on exploitability, warn on severity. Fail the build only when a critical finding is (a) in a service classified internet-facing or untrusted-input, or (b) reachable per automated analysis. Everything else produces a warning plus a ticket. False positives are what teach engineers to bypass.
- Enforce the control that actually fixes it, not the one that reports it. Automated dependency-update pull requests, pinned versions, and a centralized version catalogue mean the median fix is a merged bot PR, not a project. If fixing is cheaper than arguing, the gate stops being a fight.
- A defined exception path with a cost. A documented exception - owner, business rationale, compensating control, expiry (e.g. 30 days, renewable once) - recorded in the same system as the finding. No exception means no merge; a silent exception means the gate has already failed. Expired exceptions turn the build red automatically, so the pressure lands on the exception holder, not on security.
- Break-glass with logging and alerting. A release-block override that requires a named approver and posts to a visible channel. You cannot stop routing around a gate; you can make it attributable.
- Measure the gate, not the finding count. Track median time-to-remediate by severity tier, exception volume, and override frequency. If overrides climb, the gate is mis-calibrated - that is a signal to fix the gate, not to lecture the team.
The one non-negotiable: Tier 0 findings from the current incident block. They are reachable, internet-facing, and being actively exploited; a two-day deploy pause is cheaper than the alternative, and framing it as a time-boxed pause rather than a permanent policy is what keeps the team on side.
3. What systemic change reduces the frequency and cost of the next occurrence, and how would you measure whether it worked?
Read the worked answer
The 38-ticket morning is a symptom of dependency sprawl and missing inventory, not of one bad library.
Systemic changes, in order of leverage:
- SBOM generated at build, per artifact, stored centrally. You cannot triage what you cannot enumerate. This turns the next advisory from a multi-day archaeology exercise into a query. It also answers the customer and regulator question in minutes.
- A curated dependency catalogue with version ranges managed centrally. Teams consume an internal, tested version, so patching the parse library becomes one change propagated to 38 services rather than 38 independent investigations. This is the single biggest reduction in recurring cost.
- SCA in the IDE and pre-commit, so a vulnerable package is visible before it is a ticket. Shift-left here is about feedback latency, not about moving blame.
- Threat model the data flows once per service and record 'trusted input only' as an architectural assertion. That assertion is what makes reachability triage reproducible instead of a judgement call made under deadline.
- Service ownership metadata so the ticket routes to a team rather than a queue.
Measurement:
- Median time-to-remediate critical findings by exposure tier (target: hours for internet-facing, days for the rest).
- Percentage of services with current, build-generated SBOMs - the leading indicator.
- Count of services pinned to the centrally managed version - adoption of the catalogue.
- Override and exception rates - a governance health check.
- Recurrence cost: person-hours spent on the next comparable advisory. This is the number that justifies the programme; the first incident's 38 tickets is the baseline.
Note what is deliberately not on the list: more scanning. Detection was never the constraint here - enumeration and remediation throughput were.