Config drift, a plaintext API token, and 140 firewall VMs managed two different ways

Terraform should not be the source of truth for firewall policy - that is exactly what caused the outage. Terraform is a good provisioner (VMs, ENIs, licenses, bootstrap config, cloud networking) and a poor policy workflow: it has no concept of a change ticket, no install-to-device semantics, and a terraform apply that reverts anything it does not know about. FortiManager is the correct system of record for policy; Terraform's job ends where FortiManager's begins.

Target workflow

1. Engineer raises a ticket, then opens a PR against a policy repo containing the FortiManager change (a JSON-RPC script, a provisioning template, or a FortiManager Terraform provider resource scoped to objects and packages only).
2. CI runs a validation and a plan-equivalent against FortiManager, posting the diff to the PR. Peer review approves both the intent and the diff. The PR description must contain the ticket ID.
3. Merge triggers apply through FortiManager to a device group, not to individual devices, with a scheduled install window.
4. FortiManager revision history plus the PR plus the ticket form the audit trail - every installed revision is attributable to a person, a review, and a ticket.

Drift elimination. Remove local admin accounts on the FortiGates so the only path is FortiManager (break-glass accounts in a sealed vault, alerted on use). Schedule a nightly retrieve-and-compare; any device whose running config differs from the approved revision is auto-remediated and raises a ticket. The 38 out-of-sync devices are reconciled first, and each discrepancy is reviewed manually before reinstall - never bulk-reinstall blind.

Credentials. The token in the pipeline file must be treated as compromised: revoke it now, then read the pipeline logs and repository history to see what it could reach. Reissue a scoped FortiManager API account limited to the ADOM the pipeline manages, with a short validity, stored in a secrets manager (Vault or the cloud-native equivalent) and injected at runtime. Better: use OIDC federation from CI to the secrets manager so no long-lived static token exists in CI at all, and rotate on a schedule with a documented break-glass procedure. Add secret scanning with push protection to the repo so this cannot recur.

Upgrades. Never replace instances one at a time across the whole estate.

1. Pin the firmware version in the golden image and in the Terraform/ARM variables - the upgrade is a PR, reviewed like any other change.
2. Wave 0: one non-production FortiGate plus a lab pair, running 48 hours with the production policy package installed.
3. Wave 1: a canary of roughly 5% of production traffic in a single region, with a synthetic transaction probe and a defined abort threshold (for example error rate > 0.5% for 5 minutes).
4. Waves 2–n: roll forward region by region, never during a change freeze, one AZ at a time inside a region so capacity never drops below N-1.
5. Rollback is a tested artifact, not an intention: keep the previous image available, keep a FortiManager config revision from before the change, and have a scripted path to redeploy the prior version. Test the rollback on the canary before you need it. Autoscaling groups make this clean - build a new launch template, shift traffic, keep the old template for 7 days.

Start from the principle that TLS inspection is a per-destination policy decision, not a global switch. On the FortiGate, build an SSL inspection profile with a deep-inspection rule for categories you must see (uncategorised, newly registered domains, file-sharing, personal webmail) and a certificate-inspection-only rule - or no inspection - for the pinning workloads and for destinations that break under re-signing (Windows Update, some banking and SaaS APIs, OCSP/CRL endpoints, anything using client certificates).

For the 50 pinned VMs specifically, identify the exact destinations first from logs, then bypass inspection for those SNI values rather than disabling inspection for the whole VM, so the rest of the VM's egress stays visible. Deploy the enterprise CA to the VM trust stores, and remember that a pinned application ignores the machine trust store entirely - attempting deep inspection produces a hard fail, not a warning, so a wrong rule looks like an application outage.

Operational consequences of getting it wrong: silently broken business apps that the network team cannot see (the app team sees a TLS error, the firewall log shows an allowed session), TLS 1.3 encrypted ClientHello hiding the SNI so a destination-based bypass rule never matches, and a compliance problem if you exempt too much and lose visibility. Mitigate by logging ssl-exempt hits to a SIEM, reviewing exemptions monthly with expiry, and requiring the application owner to justify each bypass in the ticket.