A 38-Minute Pipeline, a 24% Flaky Suite, and Coupled Rollback

Principle: cheapest and most discriminating checks first, artifact built once, risk increasing only as confidence increases.

1. Lint, static analysis, dependency and secret scan (2-3 min). Runs on every commit and would have caught the hard-coded access key before it ever reached a branch.
2. Unit tests (fast). Highest defect-detection-per-second in the pipeline; should not wait behind a 22-minute suite.
3. Build artifact once, tag immutably. Everything downstream consumes this exact artifact. This also fixes the coupling problem: rollback is a redeploy of a known digest, not a rebuild.
4. Provision ephemeral environment + config render and diff. The three-of-five incidents about a value present in staging but absent in production are a config parity failure. Render both from one source and diff them; this check is seconds, not minutes, and belongs early.
5. Integration suite, parallelized and split. Moved later than unit tests but made concurrent rather than sequential; 22 minutes of wall time should become a handful. It is the slowest stage, so it should not be the first thing that can fail.
6. Deploy to staging, run contract and smoke tests.
7. Progressive production rollout with automated verification between steps (canary, then a small percentage), replacing "release manager is free" as the gate.

Moved earlier: secrets scanning, config parity, unit tests. Moved later: integration and staging deploy, because they are slow and their failures are less diagnostic. Removed as a gate: manual release-manager availability.

The dangerous move is making the pipeline blocking while the suite still fails a quarter of the time. Engineers will retry, then add || true, then stop believing the pipeline - and you have traded a flaky gate for no gate.

Reasoning about the decision:

• Flakiness is not uniform. Measure per-test failure rate over the last ~50 runs on unchanged code. A test failing 30% of the time on identical commits is providing no signal and is actively harmful; a test failing 2% of the time is probably a real race or an environment issue worth fixing.
• Quarantine (move out of the blocking set, keep running, keep reporting, assign an owner and an expiry date) is correct for high-flake tests whose intent is still valuable. The expiry matters: quarantine without a deadline is deletion with extra steps.
• Fix first for tests that are slow, environment-dependent, or that share mutable state - those have structural causes, and quarantining them hides a real environment defect that will resurface as an incident.
• Delete tests that assert something already covered and cannot be made deterministic.

Effect on the gate: the blocking set must be trustworthy before it is authoritative. Build the gate out of the deterministic subset, publish the flake rate as a visible metric, and let the blocking set grow as tests are fixed. Until the suite is under a few percent flake, a red pipeline should block a merge but not necessarily a deploy - with the deploy still gated by canary verification, which catches real breakage independently of the flaky suite.

Root cause. Terraform state is a single mutable pointer to "current." The previous build's state was overwritten in place, so the previous environment definition no longer exists anywhere - there is nothing to roll back to. A rebuild from the old commit is not the same thing: it replays every intervening change and takes as long as a deploy.

Design, in order of impact:

1. Immutable artifacts and immutable state versions. Every apply writes a new state version to a versioned backend rather than overwriting the only copy. Now "the environment as it was at release N" is a retrievable object.
2. Rollback is a redeploy of a known artifact digest, not a pipeline re-run. The application is the artifact; the infrastructure is a versioned state plus a versioned config. Roll back the app by digest; roll back infrastructure only when it is actually the cause, and from a recorded state version.
3. Keep changes backward compatible for one release so application rollback does not require schema rollback. Expand-contract migrations: add new columns/fields first, migrate code, remove old fields a release later. This is what makes fast app rollback possible without touching infrastructure.
4. Feature flags for behavior, so the most common rollback - "the new behavior is bad" - is a flag flip taking seconds, not a deployment.
5. Decouple deliberately and document the order: flag flip → app rollback → infra rollback. Practicing this monthly and recording the actual time is what converts 50 minutes into a routine operation; an untested rollback path is a hypothesis.
6. Separately: rotate the copied access key now and replace it with short-lived, scoped credentials; treat the copy-paste as evidence that the pipeline has no approved pattern for credentials.