Cloud Posture Debt, Console Drift, and a Failed Pipeline Gate
Vantage Logistics runs 480 cloud accounts across AWS, Azure and GCP; 63% of production workloads are containerised on managed Kubernetes. Terraform is the declared deployment path, but the posture tool reports 1,100 high-severity findings and roughly 300 new console-made changes each week. Only 22% of production workloads have runtime protection. Application teams hold admin in their own accounts. Last quarter the platform team proposed making the Terraform pipeline a hard gate; it blocked a release for 11 hours over an unencrypted test bucket, and the change was reverted. The CISO still wants high-severity findings down 70% in one quarter without a repeat of that outage, and the application teams have said publicly they will not accept a gate that slows releases.
1. Sequence the remediation work for the quarter and justify why the pipeline gate is not first.
Read the worked answer
Order by risk reduction per unit of political capital, not by finding count.
First, guardrails that apply to every path. The 300 weekly console changes prove the pipeline is not the only deployment mechanism, so a Terraform-only gate controls maybe half the estate. Organisational policy that denies at the platform level - public storage, unencrypted volumes, wildcard IAM, open management ports - stops the creation of the worst findings regardless of whether the resource came from Terraform, the console or a notebook. That is the highest-leverage change and it does not slow a release.
Second, runtime coverage on the workloads that matter. Moving from 22% to full coverage is impossible in a quarter, so pick the crown-jewel workloads first - those with customer data or payment paths - and instrument them before widening. Runtime detection catches what posture scanning never will, including a compromised container running on a compliant host.
Third, the pipeline gate, narrowed. Its failure last quarter was scoping, not concept: an unencrypted test bucket should never block production. Redesign it per the next task before re-enabling it anywhere.
Fourth, the 1,100 findings. Triage by exposure - internet-facing plus privileged identity plus sensitive data store - so the 70% target is met on the findings that carry real risk rather than the easiest ones to close.
2. Design the pipeline gate so it reduces risk without a repeat of the 11-hour block.
Read the worked answer
The gate must be narrow, fast, explainable and escapable.
Narrow: block only a small, non-negotiable invariant set where a false positive is nearly impossible and the consequence is severe - public object storage, storage encryption disabled, wildcard or cross-account-trust IAM, a security group open to 0.0.0.0/0 on management ports. Everything else is advisory, reported in the pull request with a severity label. A 200-policy blocking gate will be reverted again; a 10-policy one will survive.
Fast: the check must complete inside the normal pipeline wait - target under five minutes - because a gate that queues behind a slow scan will be disabled for throughput reasons, not security reasons.
Explainable: every failure names the resource, the policy, the risk in one sentence, and the exact Terraform change that fixes it. The 11-hour outage happened because someone had to research the finding to argue with it.
Escapable: an exception path with a named owner, a business justification, and an expiry of 30 days or less, recorded as code so it appears in review. Unexpired exceptions are visible on a dashboard to the CISO.
Phased: run the gate in warn-only mode on one pilot team for two weeks, measure false-block rate, then enforce. Greenfield accounts enforce from day one; legacy accounts get a dated roadmap.
3. How would you detect and reconcile resources created outside the pipeline, given 300 console changes a week?
Read the worked answer
Detection alone generates 300 alerts a week that nobody will triage, so the design goal is reconciliation with a small human queue.
Telemetry: cloud control-plane audit logs (CloudTrail, Azure Activity, GCP Audit) streamed to the SIEM, tagged by principal, with the deployment method inferred from user agent and role. Resources lacking the pipeline's ownership tags are candidates.
Classification, not alerting: split the 300 into (a) expected operational changes - autoscaling, key rotation, break-glass - which need no action; (b) unmanaged but benign resources, which should be imported into Terraform state so the pipeline owns them going forward; (c) unmanaged changes to security-relevant configuration, such as a firewall rule or a public bucket policy, which page someone.
Automatic response for class (c): revert the configuration to the declared state and notify the owner, with an explicit break-glass path that requires a ticket number at the time of the change.
The metric that matters is not 'drift detected' but 'percentage of production resources under pipeline ownership'. Report that weekly and aim to move it from its current level toward 95%, because a resource in state cannot drift silently. Pair it with a monthly review of the class (b) backlog so import work does not stall.