Console Drift and a Stuck CloudFormation Stack
A platform team owns 40 CloudFormation stacks deployed through CodePipeline with a manual approval stage. During an incident an engineer added an inbound rule to a production security group in the console: TCP 5432 from 10.0.0.0/8. The template does not contain that rule. A routine update that adds a new subnet now rolls back, and two unrelated stacks sit in UPDATE_ROLLBACK_FAILED. Some stacks use nested stacks. The RDS instance has deletion protection enabled. Nobody wants to delete and recreate the production stack.
1. How do you add the subnet without silently reverting the console change or rebuilding the stack?
Read the worked answer
Sequence matters. CloudFormation will overwrite unmanaged properties when it updates the security group resource, so reconcile before changing anything else:
- Run drift detection on the stack and nested stacks.
DescribeStackResourceDriftsgives the exact property diff versus the template. - Decide on the console rule deliberately. If port 5432 from 10.0.0.0/8 is legitimate, add it to the template and update so the template matches reality. If it is not, plan a controlled removal rather than letting an unrelated update delete it by surprise.
- Use a change set for the subnet addition and inspect which resources will be modified. If the security group is not in the change set, it will not be touched, but do not rely on that as the control.
- Recover the failed stacks with
ContinueUpdateRollback, skipping only specific resources if a dependency is genuinely stuck. For nested stacks, continue from the innermost failed stack outward. - Never delete the stack. Deletion protection on RDS helps, but deleting a stack can still destroy non-protected resources.
The correct order is drift, reconcile, change set, then apply.
2. What controls stop console drift from reaching production again?
Read the worked answer
No single control is sufficient; layer prevention, detection, and process:
- Prevention: an SCP that denies mutating EC2 security-group actions for all roles except a break-glass role. Route normal changes through the pipeline.
- Detection: a scheduled drift-detection mechanism (EventBridge schedule invoking drift detection, with notifications on
DRIFTED) and the AWS Config rule that checks stack drift. Drift detection is not continuous by default; you must trigger it. - Standardization: Service Catalog products so teams provision approved templates rather than hand-editing stacks, and resource import for resources that legitimately predate CloudFormation.
- Process: a documented break-glass procedure that requires the change to be back-ported to the template within a defined window, with an alert if a stack remains drifted.
Stack policies protect resources from stack updates; they do not prevent console changes, so they are not a drift control.
3. How should the pipeline change so failures and drift surface before the manual approval?
Read the worked answer
Move checks left of the approval gate:
- Static validation: cfn-lint and cfn-nag or Guard rules in the build stage.
- Change set as an artifact: create the change set, render it to a readable diff, and require approval on that diff rather than on a generic deploy prompt.
- Drift gate: fail the pipeline if the target stack is drifted, forcing reconciliation before new changes. This prevents the next update from silently reverting console edits.
- Ephemeral test stack: deploy the same template to a disposable account or stack, run integration tests, then delete.
- Rollback triggers: attach CloudWatch alarms to the CloudFormation deploy action so a bad release rolls back automatically.
The goal is that the human approves a known diff against a known-clean stack, not that the human is the only check.