The Certificate That Expires on a Holiday
An internal API gateway terminates TLS for 60 backend services. Its public certificate is issued by an external CA with a 90-day validity and is renewed manually: an engineer requests a new certificate, downloads it, and uploads it to the gateway. The last renewal was performed by one engineer who has since left. The current certificate expires in 11 days, on a national holiday when change approval requires a director's sign-off. The gateway also pins its certificate to two mobile clients, which validate the pin. Monitoring checks that the gateway responds on port 443 but does not check certificate expiry or chain validity. A separate internal CA issues certificates for service-to-service traffic with a 2-year validity.
1. Identify the risks in this setup and rank them.
Read the worked answer
Ranked by likelihood times impact:
Holiday expiry with no automated renewal. The certificate expires in 11 days on a day when approval is hardest. If it lapses, all 60 services behind the gateway become unreachable to external clients, and the mobile apps fail. This is the highest-priority risk because it is nearly certain and catastrophic.
Pin breakage. The two mobile clients pin the gateway certificate. Even a successful renewal replaces the certificate, so unless the pin is updated or the clients support a backup pin, those clients will reject the new certificate. This risk is triggered by the fix itself, not just by expiry, and is easy to miss.
Single-person knowledge. The only engineer who knew the renewal process has left. There is no runbook, so the renewal may fail on procedure even if someone is available.
Monitoring blind spot. Port 443 responding does not mean the certificate is valid. The team would learn of expiry from users, not alerts.
Inconsistent lifetimes. The 90-day external certificate and 2-year internal certificate create two very different renewal cadences, and the shorter one is the manual one.
The internal CA is lower risk here only because its validity is long; it is still a manual process and should be treated as the same class of problem.
2. What should be done in the next 11 days, and what should be done afterward?
Read the worked answer
Next 11 days, in order:
- Today: export the current certificate, chain, and private key location; confirm who holds access to the gateway and the CA account. Write a one-page runbook.
- Request the renewal immediately rather than waiting for the holiday window. A 90-day certificate can be replaced early; there is no reason to run to expiry.
- Before deploying the new certificate, resolve the pin. Determine whether the mobile clients support multiple pins or a pin set. If they do, publish a pin set containing both old and new keys, ship the client update, then rotate. If they do not, coordinate a forced update or a maintenance window with the mobile teams. Do not rotate the gateway certificate without a pin plan.
- Add an expiry check to monitoring now: alert at 30, 14, and 7 days, and validate the full chain, not just port reachability.
- If a director is needed for the holiday change, secure approval in advance rather than on the day.
Afterward:
- Replace manual renewal with ACME or the CA's automated API, and store the private key in a managed secrets store.
- Remove single-person dependency: at least two engineers trained, runbook in the team wiki, renewal owner named.
- Align certificate lifetimes so short-lived certificates are the automated ones.
- Add a quarterly review of all certificate inventory, including internal CA certificates, so the 2-year ones do not surprise anyone either.