A Pipeline Nobody Trusts
A 40-engineer organisation merges about 60 pull requests a week into main. A single GitHub Actions workflow runs lint and 1,400 unit tests (22 minutes wall clock), builds an image tagged both latest and $GITHUB_SHA, then runs kubectl set image against the production Deployment. Fourteen tests hit a shared staging Postgres and flake roughly 8% of the time; engineers re-run failed jobs until green. Repository-level Actions secrets hold the cloud deploy key, and three contractors have write access to the repository. Last month a bad merge reached production; rollback took 45 minutes because latest had already been overwritten by a later build, so kubectl rollout undo restored a ReplicaSet that re-pulled the wrong image. The only deploy freeze is a Slack message. Leadership wants zero-downtime deploys and faster feedback with no new headcount.
1. Rank the first three changes by risk reduced per unit of effort, and justify the ordering.
Read the worked answer
Rank by blast radius and by how small the change is, not by how modern it sounds.
1. Immutable references and fast rollback. Stop deploying latest; deploy image@sha256:... (or a manifest pinned to $GITHUB_SHA) and record the digest in the pipeline summary. The 45-minute rollback becomes a re-apply of the previous recorded manifest, roughly two minutes. This is a few lines of workflow YAML and it directly addresses the incident that already happened, so it wins on both axes.
2. Quarantine the 14 flaky tests. With 14 flaky tests, the probability that any given run is red for no reason is high, which is exactly why re-running until green became normal - and a gate everyone re-runs is not a gate. Move the 14 into a separate non-blocking job with owners and expiry dates, gate merges on the remaining 1,386, and publish the flake rate. Immediate effect on trust, near-zero implementation cost.
3. Access and secrets. Three contractors with write access can push to main and read repository-level deploy secrets. Move deploy credentials to GitHub Environments with required reviewers and branch restrictions, prefer OIDC federation to the cloud so there is no long-lived key at all, and require review plus passing checks on main so the deploy freeze stops being a Slack convention.
Zero-downtime mechanics - readiness probes, maxUnavailable: 0, graceful shutdown, canary - matter, but they come after you can roll back reliably and trust the gate.
2. Design a policy for the 14 flaky tests that avoids both 'rerun until green' and 'delete them'.
Read the worked answer
Both extremes fail for the same reason: they discard information. Reruns hide genuine failures and inflate CI time; deletion removes coverage of whatever those tests were written for.
Step one is instrumentation, not triage. Parse the CI result JSON over the last 30 days and compute a per-test failure rate, so you are prioritising by measured flakiness rather than by whoever complained loudest. You will usually find that a handful of tests account for most failures.
Step two is fixing the harness before the tests. All 14 hit a shared staging Postgres, which points at cross-test interference rather than 14 independent bugs: parallel workers sharing a schema, fixtures using fixed ids or emails, no transaction rollback between tests, and assertions depending on wall-clock time or row order. Move to an ephemeral database per worker (a schema per worker, or Testcontainers in CI), seed deterministic fixtures, and make assertions order-independent. That change often fixes most of the 14 at once - far cheaper than debugging each individually.
Step three is quarantine with teeth. Anything still flaky moves to a non-blocking job with a named owner, a linked issue and a two-week expiry. On expiry: fix or delete, deliberately. Track flake rate as a metric with a budget (for example under 1%) and flag any new test that fails intermittently in its first week, so the backlog does not refill.
3. Why is `kubectl set image` on its own not sufficient for zero-downtime deploys, and what must be true for the rollout to be safe?
Read the worked answer
kubectl set image triggers a rolling update, but a rolling update only avoids dropped requests if several independent conditions hold.
The manifest. replicas must be greater than one, the strategy must be RollingUpdate with maxUnavailable: 0 and a maxSurge of at least one, and a PodDisruptionBudget must protect capacity during node drains.
Readiness. A pod must only receive traffic when it can actually serve. Without a readinessProbe, Kubernetes routes to pods whose event loop or database pool is not yet ready; with the current /health that also returns 503 on database errors, the opposite failure appears. Readiness buys the rollout correctness, which is why the probe redesign is a prerequisite, not a nice-to-have.
Draining. On SIGTERM the app must stop accepting new connections, finish in-flight requests, and close the pool, with a preStop sleep at least as long as the load balancer's deregistration delay. Otherwise the pod exits while the balancer still sends it traffic. Note that CMD node ... instead of npm start is what makes the signal arrive at the app at all.
Schema compatibility. If a migration breaks the previous code version, old and new pods cannot coexist, and no rollout strategy saves you. Use expand/contract migrations across two deploys.
Version identity. Pulling latest with imagePullPolicy: Always makes pod versions non-deterministic. Pin by digest.
A rolling update alone also will not stop a bad version from reaching 100% of traffic; add a canary stage with automated error-rate and latency checks if that is the goal.