Portal Cutover with Hardcoded Client IPs and Pinned Certificates
A subscription media company is cutting over its customer portal - 60 Linux VMs, two web tiers and a 3 TB PostgreSQL database - from a colocation cage to OCI in a single 9-hour Saturday window. Bulk data has been replicated with rsync every 6 hours and the final delta is expected to be under 40 GB. Constraints: 400 enterprise customers call the API from fixed source IPs whitelisted at their firewalls; 60 of those also pin TLS certificates; public DNS records currently have a TTL of 86400; a nightly settlement batch runs at 02:00 UTC and must not miss a day; the colocation contract ends in 30 days and the provider has refused an extension. The rollback plan states that 'the old environment stays intact', but the old load balancer's certificate expires in 45 days and the environment is billed monthly with 30 days' notice.
1. Sequence the cutover and place the decision gates. Justify why the window should not start at 22:00 UTC.
Read the worked answer
The window should start after the 02:00 UTC settlement batch completes on the old platform, not before. Starting at 22:00 puts a hard, non-negotiable batch inside the cutover - if the cutover runs long you either miss a settlement day or run the batch on a half-migrated system. Start at 03:00 UTC and use the 9 hours until 12:00.
- T-7 days: reduce public DNS TTL from 86400 to 60; reserve the OCI public IPs; issue and validate new certificates; run a full parallel test with synthetic traffic from a whitelisted customer IP.
- T-24h: final full rsync, verify object counts and checksums, confirm the old environment's certificate is not about to expire mid-window.
- T-0, gate 1: prerequisites verified - TTL propagated, IPs reserved, certificates installed on the new load balancer, rollback point agreed in writing.
- T-0: put the portal into maintenance mode, quiesce writes, run the final rsync delta, verify checksums (gate 2).
- T-0: start the application on OCI in read-only, run smoke tests including one whitelisted customer path and one pinned-certificate client (gate 3).
- Flip the DNS weighted steering to OCI, then flip fully once error rates and latency are nominal.
- T+0 to T+48h: monitor, verify the next settlement batch on OCI, keep the source read-only.
Gate discipline matters more than the step list: three cheap stop points are what keep a 9-hour window from becoming a 14-hour incident.
2. How do you handle the 400 fixed-IP customers and the 60 that pin certificates? Explain which of the two is the harder problem and why.
Read the worked answer
The fixed-IP customers are an infrastructure problem; the certificate-pinning customers are a coordination problem, and coordination is harder because you cannot engineer around a decision someone else's security team has already made.
Fixed IPs - in order of preference:
- Bring your own public IP prefix to OCI. If the current range can be transferred from the colocation provider and the RIR/location records move in time, the whitelisted addresses follow you and 400 customers change nothing. Check the transfer lead time immediately; it is measured in weeks and is the pragmatic long pole.
- If the prefix cannot move in 30 days, run a transition listener at the old address that proxies to OCI - but the cage is being returned, so this only works if you can keep a minimal footprint or place a proxy elsewhere at the same address.
- Otherwise it is a managed, dated customer programme: parallel whitelisting of the new OCI IPs before cutover, per-customer confirmation tracking, and a named account owner for each of the 400. Do not treat this as an email; treat it as a release dependency with a burn-down chart.
Certificate pinning: DNS changes do not help, because the client validates the key or the chain, not the address. Where customers pin the leaf or intermediate, reissue certificates from the same CA; where they pin the public key, issue the new certificate from the same key pair so the pin still validates after the move. Either way, get confirmation from the 60 in writing before the window, and be explicit about the rollback implication: rolling back to a different certificate may break exactly the clients you just fixed. Assume at least a few customers will not respond, and have a per-customer break-glass plan rather than discovering it live.
3. Evaluate whether the stated rollback plan is real. What would you rewrite, and what circumstances would make you abort the cutover before the DNS flip?
Read the worked answer
The rollback plan as written is a sentence, not a plan. 'The old environment stays intact' is contradicted by two facts: the cage is returned in 30 days, and the environment is billed monthly with 30 days' notice - so the practical preservation window is roughly 30 days, not indefinite. And the old load balancer's certificate expires in 45 days, which is outside the contract anyway, so any rollback path that depends on that certificate has a hard expiry earlier than people assume.
Rewrite it around the real rollback point. With rsync and no reverse replication, rollback is safe only up to the first write accepted on OCI. After that, rollback means reconciling writes made in OCI back into the old database by hand. State the moment explicitly, name who declares it, and cap the rollback window at 7 days rather than 30, because everything past that is a comfort blanket.
Add a reverse path if you can: a short-lived logical replication back to the source buys a real rollback for the soak period. It is cheap relative to a failed weekend.
Abort before the DNS flip if: the final delta checksum does not reconcile; any whitelisted customer path or pinned-certificate client fails the smoke test; the new load balancer's certificate chain is not the one the pinned clients expect; or fewer than a defined threshold of the 400 have confirmed the new IPs. Aborting a cutover early is cheap; aborting it after the DNS flip and a settlement batch is a customer-visible outage that costs more than the migration saved.