Terraform Drift After a 02:00 Security-Group Hotfix
A platform team manages production VPC and IBM Cloud Kubernetes Service with IBM Cloud Schematics workspaces; each environment has its own workspace and state. During a 02:00 incident, an engineer edited an existing Terraform-managed security-group rule from the partner's /32 to 0.0.0.0/0 so an external partner could reach a debug endpoint. Separately, the engineer increased a worker pool from 6 to 9 nodes because pods were pending. The next scheduled Schematics apply is in six hours and will reconcile both changes back. The engineer did not update Terraform, and a new hire has an admin API key that can run terraform apply locally. The security-group change was intended to be temporary, the pool increase is still needed for capacity, and the partner has not finished testing. Activity Tracker is enabled but not routed to the team SIEM.
1. What is the first safe action before the scheduled apply, and how do you classify each drift item?
Read the worked answer
First, stop the automated apply from making an unreviewed change: pause the Schematics workspace or remove auto-approve, then run a plan with refresh and inspect it. Do not run a local apply with the admin key; that would create a second state authority.
Classify each drift item by intent and risk:
- Security-group rule: this is an emergency exception, not an accepted design. Activity Tracker shows who changed it, when, and from where; VPC flow logs can show whether the partner is actually using port 8443. Treat it as high risk because 0.0.0.0/0 on a debug port is broader than the incident required.
- Worker-pool size: this is an operational capacity change that is still needed. Reverting it would recreate pending pods and could cause an outage.
The plan should be reviewed by both platform and security. If the plan would revert the pool to 6, block the apply until the Terraform variable is updated to 9. For the security group, decide whether to keep a narrowed rule (partner CIDR) or remove it after confirming the partner's test window. The principle is: do not let the tool decide an emergency exception; codify the intended end state first.
2. Design the safe remediation sequence and the guardrails that prevent recurrence.
Read the worked answer
Sequence: (1) update Terraform to declare the worker pool at 9 and apply that change in a controlled window so state and reality agree; (2) replace the /0 rule with the partner's current egress CIDR, or remove it and provide a private path; (3) apply the security-group change only after the partner confirms the new path works; (4) verify with a post-apply plan that is empty and with connectivity tests.
Guardrails: remove local admin API keys and run CI through a trusted profile or service ID with least privilege; require Schematics approvals for production and lock workspaces; enable scheduled drift detection using plan with detailed exit codes and route results to the team; use Context-based Restrictions so only the CI identity can change VPC and cluster resources; store the Terraform in reviewed merge requests; add policy-as-code checks in IBM Cloud Security and Compliance Center for public security-group rules and unapproved instance profiles. For temporary access, require an expiry field and an automated cleanup job. Activity Tracker should route to the SIEM so out-of-band CLI changes are visible in near real time, not discovered at the next plan.
3. How would you handle the partner's debug access without leaving a permanent 0.0.0.0/0 rule?
Read the worked answer
Use a time-bound, least-privilege path. Best option: put the debug endpoint behind a private connection such as a VPE or a bastion host with an IP allowlist, MFA, and session logging, so the partner never reaches the workload directly. If public reachability is unavoidable, allow only the partner's specific egress CIDR, restrict the source port range if the partner can fix it, and set an expiry that automation enforces by removing the rule. Do not rely on a calendar reminder. Add an alert when any security-group rule with 0.0.0.0/0 is created on a non-standard port. Record the exception, owner, and expiry in the change ticket. After the test, verify removal with a plan and a connectivity test, and keep the Terraform declaration as the source of truth. If the partner needs repeated access, build a reusable private-access pattern rather than reopening the same emergency rule.