Tailwind: Regional Rollout and a 45-Minute Rollback
Tailwind runs 40 microservices on AKS in three regions, deployed by one YAML pipeline. Build takes 22 minutes on Microsoft-hosted agents with no caching, and every service builds on every commit. Deployment is a single stage: a failure in region 2 blocks regions 1 and 3, and the on-call engineer rolls back by locating the previous build and redeploying it, with MTTR around 45 minutes. Some deployments run EF Core migrations at application startup. The audit team requires documented approval before production; leadership wants daily deploys. Builds fail intermittently on self-hosted agents with no space left on device; agents are reused indefinitely.
1. Redesign the pipeline structure and rollout strategy for zero-downtime deployment with database migrations, and specify the rollback story.
Read the worked answer
Split into a path-filtered build per service that produces immutable artifacts, and a deployment pipeline with independent per-region stages (or a matrix over regions) so region 2 cannot block regions 1 and 3. Use deployment jobs against Environments so each deployment is recorded with its artifact and commit.
Rollout: canary into one region with a slice of traffic, run synthetic and health checks, then expand region by region. Blue/green works too, but canary gives earlier signal with less duplicated infrastructure.
Rollback must be redeploy the previous known-good immutable artifact - the Environment deployment history gives you that ID. This only works if artifacts are immutable and configuration is externalized.
Migrations: run as a separate job before the app rollout using expand-and-contract (add nullable column, backfill, ship code that tolerates both, drop later). Never migrate at startup: N replicas race, and there is no rollback for a dropped column. A backward-compatible migration is what makes canary and rollback safe at all.
2. Where should the production approval sit so audit is satisfied while daily deploys remain possible?
Read the worked answer
Put the approval on the production Environment as a required check, not as a manual gate per region.
One approval covers the release, tied to the artifact and commit; the approval record itself is the audit evidence, so nobody reconstructs it later from email. If the auditor needs a change ticket, wire a ServiceNow check to the same stage rather than adding a second human step.
Issues three per-region approvals and daily deploys are incompatible. Instead set an approval SLA (for example four hours, on-call lead as approver) and an auto-reject timeout, then monitor rejected-by-timeout counts - if they climb, the approval is the bottleneck, not the pipeline.
Tradeoffs: auto-reject causes re-queues and can look like flakiness; you can also make the check bypassable for severity-1 fixes, but then log and review every bypass. Keep business-hours restrictions out unless leadership explicitly trades deploy frequency for them - an approval check enforces control in the pipeline, which is what an auditor is actually testing.
3. In what order would you fix the 22-minute build and the intermittent agent failures? Justify the priority.
Read the worked answer
Fix the agents first, because an infrastructure bug that presents as intermittent pipeline failure corrupts every other signal - you cannot trust build-time measurements while agents are dying. Diagnose with disk usage on the agent, prune Docker images and old workspaces, and move to ephemeral or regularly reimaged agents so state cannot accumulate.
Then cut build time with cheap wins in order of effort:
- Path-filtered triggers so only changed services build - with 40 services this is the single largest saving.
- Package restore and Docker layer caching.
- Matrix parallelism across independent services.
- A shared YAML template parameterized by service, so filters and caching stay maintainable.
Aim for well under ten minutes before buying bigger agents; hardware hides waste. Tradeoffs: path filters need accurate dependency mapping - a shared library change must trigger its consumers, or you ship stale services. That mapping belongs in the template, reviewed like code.