Enforcing Artifact Signing Without Stopping Delivery
Your organisation runs 120 repositories on GitHub Actions, 14 product teams and about 60 production deploys per day. Build jobs assume a long-lived AWS access key stored as an organisation secret and print environment variables on failure; three leaked credentials have appeared in build logs in the past 90 days, the most recent used to read a staging S3 bucket containing masked test data. There is no artifact signing, SBOMs are generated for 30% of repositories and enforced for none, and an emergency hotfix path skips required review and uses a deploy role with administrator rights. Mean deploy time is 25 minutes, the security review queue is four days, and engineers route around friction. Leadership wants a policy blocking any deploy without a signed artifact within 30 days, and has asked you to make it stick without stopping delivery.
1. Given 30 days, 14 teams and a delivery commitment, sequence the controls. What goes first and what should not be attempted yet?
Read the worked answer
Resist the blanket 30-day block. Enforcing signing across 120 repositories in 30 days produces exceptions, shadow pipelines and a security team blamed for a release freeze - and the control still will not be real. Sequence by risk reduction per unit of friction.
Weeks 1-2, stop the bleeding without changing developer workflow:
- Replace the long-lived AWS key with GitHub OIDC federation to short-lived role credentials; revoke and rotate the existing key immediately. This alone removes the class of leak you have already had three times.
- Enable secret scanning with push protection, mask environment variables in logs, and scrub existing log history.
- Strip administrator rights from the deploy role and scope it per environment.
Weeks 2-6, build the capability once, centrally:
- Sign artifacts in a shared reusable workflow using keyless signing, and generate provenance and SBOM by default so teams inherit rather than implement.
- Enforce verification in the admission controller in audit mode first, then block in one non-critical environment.
Weeks 6-12 and beyond: pilot two willing teams, publish a migration guide, then ratchet enforcement repository by repository with published dates. Announce the direction in week 1 so nobody is surprised, and make the paved road faster than the bypass.
2. Redesign the emergency hotfix path so it stays fast but is no longer an unlogged administrator bypass.
Read the worked answer
Keep the speed, remove the invisible admin.
- Same pipeline, different gate: a hotfix still runs build, signing and SBOM steps; only the human review is compressed.
- Review: an on-call security reviewer with a 30-minute response SLA, reachable through the incident channel. If nobody responds in 30 minutes, an incident commander plus a second engineering approver can authorise - always two people, never one.
- Break-glass deploy role: time-bound (for example four hours), scoped, session-recorded, triggers a high-priority alert, and requires an existing incident ticket number.
- Mandatory post-hoc review within 24 hours owned by the team lead; unreviewed hotfixes are flagged automatically in the weekly report and the next deploy for that service is blocked until closed.
- Measure it: hotfix count, percentage reviewed within 24 hours, and merge-to-production time during incidents. If the hotfix path becomes the normal path, that is a signal to fix the main pipeline, not to tighten the rules.
The principle: a bypass that is invisible is a vulnerability; a bypass that is narrow, two-person, recorded and reviewed is a controlled exception. Removing the long-lived admin role also makes the hotfix path auditable, which the current design never was.
3. Which metrics would you put in front of engineering leadership, and which would you deliberately avoid?
Read the worked answer
Pick metrics that change behaviour and cannot be gamed by volume.
Leading, adoption-based:
- Percentage of production deploys with a verified signature, by team, trending to 100%.
- Percentage of repositories using OIDC, and count of long-lived cloud keys remaining (target: zero).
- SBOM coverage and percentage of deploys carrying provenance attestation.
Lagging, outcome-based:
- Credentials detected in logs or code, and median time from detection to revocation. The second matters more: a leak revoked in 10 minutes is a near miss.
- Percentage of hotfixes reviewed within 24 hours, and break-glass role uses per month.
- Security review queue time; if it exceeds a day, teams route around it and every other metric degrades.
Avoid 'vulnerabilities found' and 'scans run'. They reward activity rather than risk reduction and grow every time you add a scanner. Report a small dashboard monthly with team-level visibility, and do not punish self-reported near misses - the leak reports stop the moment reporting becomes career-limiting, and you lose the only early signal you have.