One hundred and twenty deploys a day versus a critical CVE

Assessment. Uniform critical-fail is a policy that will not survive contact with the estate. It would trigger roughly 900 rebuilds for a library that 41 running images actually load, and it treats a dev-only test dependency like an internet-facing API. The predictable outcome is engineers adding --ignore flags and skip annotations, which converts a loud scanner into a silent one - strictly worse than the current advisory state. A gate that is bypassed is not a control.

Design a risk-weighted gate. Fail the build only when all of these hold:

1. Severity is critical or high and the dependency is reachable at runtime (reachability analysis, or VEX statements from the vendor marking components not affected).
2. A fixed version exists.
3. The service is internet-facing or handles PHI - tier-1 classification.

Everything else gets an SLA rather than a block: high in 30 days, medium in 90, with automatic PRs for dependency bumps so remediation is cheaper than arguing.

Enforce at the right chokepoint. Gate the deployment, not only the build: an admission policy admitting only signed images from the trusted registry means a hotfix pushed straight to the cluster cannot bypass the check either.

Exceptions. A PR-based, time-boxed record naming an owner, a compensating control, and an expiry date, visible on a dashboard. If the same advisory exception is requested three times, the policy is wrong, not the team.

What build-time scanning misses. It sees the image at the moment CI ran. It does not see images deployed before the scanner existed, images built outside the pipeline, dynamically loaded code, base-image drift, or a CVE published after the build. Most importantly it cannot tell you, when a new CVE lands at 2 a.m., which running pods contain that library right now - and that is the question the incident always asks.

Controls that close the gap.

• Signing and admission: sign images with cosign/Sigstore and enforce with Kyverno or OPA Gatekeeper that only signed images from the approved registry run in production. This makes bypassing CI impossible rather than merely discouraged.
• SBOM per image digest: generate SPDX or CycloneDX at build time and store it keyed by digest, then query across the cluster to find every workload containing the affected package. This converts the 900-repository number, which is noise, into the 41-running-images number, which is the actual risk.
• Continuous rescanning plus rebuild automation: rescans of the registry on new vulnerability data, triggering automated base-image rebuild and redeploy. This is how a transitive dependency actually gets fixed at 60-service scale.
• Runtime detection: eBPF-based tooling (Falco or equivalent) for deserialization-like behaviour and unexpected egress, which catches exploitation of a flaw nobody had a signature for.

The reporting change matters as much as the tooling: track "running images containing an exploitable critical" rather than total findings across repositories.

Root cause. The root cause is not that a developer was careless - people are careless at 350-person scale, and this will recur. The root cause is that a static, long-lived credential existed that could be pasted into a file, committed, and then used from any network in the world with no additional signal. Everything else is a symptom.

Changes, in order of leverage.

1. Remove the credential class. GitHub Actions OIDC federation to an IAM role with conditions on repository, branch and environment. With no static secret to commit, this failure mode disappears.
2. Shift-left prevention. GitHub push protection and pre-receive secret scanning to block the commit, plus organisation-wide scanning with partner notification for anything that already escaped - a public repository means assume immediate compromise, not nine hours of grace.
3. Narrow the blast radius. Least-privilege scoped roles per repository and environment, so a leaked credential cannot reach production data. An SCP restricting where the CI role may be assumed adds a second barrier.
4. Shorten the window. Automated revocation triggered by the secret-scanning alert (Lambda or webhook) rather than a ticket queue. Nine hours is the metric that failed; target under fifteen minutes.
5. Detect misuse. CloudTrail alerting on use of CI roles from unexpected ASNs or from outside the runner's egress range.

Measure mean time from secret exposure to revocation, and the count of long-lived CI credentials remaining - not the number of alerts generated.