Going Green and Staying Green Without Slowing Delivery

Reasoning principle. A gate earns its place only if it is (a) accurate - low false-positive rate on this codebase, (b) actionable - the developer can fix it in the same change, and (c) consequential - the failure mode is one you would actually stop a release for. A gate that fails these tests gets bypassed, and a bypassed gate is worse than no gate: it teaches the organisation that red means "ask a release engineer to clear it". The evidence is already in the setup - 310 skipped tests and a disabled secret check are the organisation's verdict on the current gating.

Blocking:

• Secrets in code on the mainline. High signal, high consequence, trivially fixable by rotation plus removal. The 30 fixture hits are not a reason to disable the check - they are a reason to add a documented, expiring allowlist scoped to test paths, with an owner. Disabling on the test branch is the wrong fix because test branches merge.
• Critical or high vulnerabilities with a fix available and a reachable package. Narrow the rule to reachability plus fixability so the 380 unfixable CVEs are excluded by construction.
• Failing unit tests on changed code, with flaky tests removed from the blocking set (see below).
• Build reproducibility and artifact signing, because a failure here means you cannot trust what you shipped.

Advisory:

• Full-suite SAST on legacy repositories - blocking here blocks everything and fixes nothing.
• CVEs with no upstream fix - route to a risk-accepted list with review dates.
• Style and complexity linters.

Also blocking, but it is a process defect: the five-hour manual clear. Replace with a self-service, time-boxed, audited break-glass so a gate cannot become a human queue.

The trap. Total-findings-count as a gate is unworkable at 900 and, worse, it makes the number go down through suppression rather than remediation. The team is already demonstrating this with skips.

Step 1 - Baseline and ratchet. Snapshot the 900 as a known baseline per repository. From that day, the gate applies only to new findings introduced by the diff, plus a rule that the baseline may never grow. This immediately stops the bleeding without freezing delivery, and it changes the metric from "total open" (which nobody can move) to "new findings reaching main" (which must be zero).

Step 2 - Tune the ruleset. 900 findings across 60 repos and two legacy systems usually means the rule set was never adapted. Measure the false-positive rate per rule over 50 sampled findings; disable or downgrade rules that are systematically wrong for this stack. This is not reducing security, it is making the remaining signal believable.

Step 3 - Assign and burn down the legacy. The two unowned repositories are the actual problem. Ownership is a prerequisite: findings without an owner never close. Assign them, agree a burn-down target (say 25 findings per sprint each, prioritising injection, authentication, authorisation and deserialisation rules), and report the trajectory, not the absolute number.

Step 4 - Legitimate acceptance with expiry. Findings that are genuine false positives or in dead code get a documented suppression with an owner, a reason and an expiry date. Expired suppressions resurface automatically.

Metrics to report: new findings introduced per week (target 0), median time to remediate a new finding, and the baseline trajectory per repository. Do not report "findings closed", which rewards bulk-closing trivia.

Order of attack: flakiness, then the bottleneck, then duration.

1. Flakiness first, because it destroys the meaning of the whole pipeline. At 12%, roughly one in eight runs fails for reasons unrelated to the change. Every engineer who reruns a red build learns that failure is noise, and that lesson transfers to the SAST gate, the secret scanner and eventually to production alerts. Fix it with process before tooling: quarantine flaky tests into a separate non-blocking job, require each quarantined test to have an owner and a fix-or-delete deadline (10 days is a reasonable start), and delete tests nobody will fix. A smaller suite people trust beats 1,800 tests they rerun. Measure the flake rate per test, not overall, so the top offenders are visible.

2. The five-hour manual clear is a bigger delivery constraint than 47 minutes. Three blocks a month at five hours of engineer time is 15 hours, plus the queueing effect on everyone waiting. Self-service break-glass with an audit trail and a mandatory follow-up ticket converts a gate into a decision the team can make. If the gate must stay human, at least page the on-call reviewer rather than a single named release engineer.

3. Duration, last - and partly as a by-product. Shard the test suite, run independent stages in parallel, cache dependencies, and use test-impact analysis so a one-line change does not run 1,800 tests. But note that removing flaky tests alone often cuts wall-clock time, because reruns and manual triage are a hidden part of the 47 minutes.

Measurement to prove it worked: median and p95 pipeline duration, flake rate, skip count (should fall, not rise), percentage of builds passing first time, and deploy blocks per month. If skipped tests keep climbing, the gates are still wrong regardless of what the other numbers say.