Zero-Downtime OKE Release with a Fifteen-Minute Observation Window

Blue-green, not rolling. A rolling update interleaves old and new pods under one Service, so there is no clean abort point and no intact previous environment to fall back to; with 2,400 rps on a payment path, the abort has to be instantaneous. Blue-green gives you a complete green Deployment that receives no production traffic until you move the Service selector, and a complete blue Deployment still serving if you do not.

Pipeline shape:

1. Deploy stage (blue-green to OKE) - creates the green Deployment with the new image digest; production Service selector still points at blue, so no customer traffic reaches green.
2. Verification - a smoke test against green through a dedicated test hostname: readiness endpoint, a login, and an authorize call against a test merchant account. Fail the stage on any non-200.
3. Observation gate - a wait or approval stage holding for 15 minutes, with the option to send a small slice of real traffic to green through a second service or weighted routing during the window.
4. Cutover - patch the production Service selector to green.
5. Post-cutover gate - after error-rate and latency checks, scale blue down. If the console's blue-green strategy performs cutover automatically at the end of the deploy stage, split it into two deploy stages or gate the second with an approval stage so the observation window is a real gate rather than a comment.

Success meant the apply command exited 0, not that the workload was healthy. kubectl apply returns immediately; nothing in that stage asserted that the new pods became Ready or served a request. A stage that only applies manifests will call a crashlooping deployment a success every time.

Manifest changes:

• readinessProbe on a real dependency-aware endpoint (DB connection pool established, cache warmed), not /health returning 200 unconditionally. Pods that are not Ready must never be added to the Service endpoints.
• startupProbe for the JVM boot so a slow start is not mistaken for a hang, plus a livenessProbe that only fails on genuinely wedged processes - an aggressive liveness probe turns a slow start into a restart loop.
• progressDeadlineSeconds around 120, explicit maxSurge/maxUnavailable, and a PodDisruptionBudget with minAvailable so node drains and rollouts cannot take the service below capacity.

Pipeline changes:

• Replace bare apply with an explicit kubectl rollout status --timeout=180s in the deploy step if the stage does not already wait, and surface a non-zero exit.
• Add a verification step that asserts version, not just liveness: have /health return the running build SHA and compare it against the artifact version the pipeline just deployed. A pod can be Ready and still be the wrong build.

Expand, migrate, contract - across at least two releases. During blue-green both versions run simultaneously for the whole observation window, so the schema must serve old and new code at the same time.

Sequence:

• Release N: additive DDL only (new nullable column, backfill job, dual-write from the app). The migration runs as its own pipeline stage before the app deploy stage, with its own approval and its own verification query, never as a Job inside the application manifest where it shares the app's rollback fate.
• Release N+1: app reads the new column and keeps writing both.
• Release N+2, well after the observation periods: drop the old column.

Rollback semantics: flipping the Service selector back to blue restores the application but does not undo the schema. That is precisely why destructive DDL must never ship in the same pipeline run as an app deploy - the rollback path would then require restoring data. Migrations must be idempotent and versioned (Flyway or Liquibase) with a migration lock, so a retried stage does not re-apply a partially completed change, and the migration stage's own verification should query the schema version before the app stage is allowed to start.