APM Login Loop After an LDAP Certificate Rotation
A BIG-IP APM access policy protects an internal HR portal. The policy performs LDAP authentication against Active Directory, then RADIUS one-time-password verification, then issues a Kerberos SSO credential to the backend. ASM runs in blocking mode with a default negative security policy. During a planned certificate rotation on the LDAP endpoint, about 20% of users begin looping between /my.policy and the portal login page and receive HTTP 401. The remaining 80% authenticate normally. APM logs for affected sessions show 'session variable not found' and 'access policy agent evaluation failed'. The new LDAP certificate is valid and the LDAP servers are reachable from the BIG-IP. Users with existing sessions continue to work until they log out. Security insists that authentication strength must not be reduced.
1. Form a hypothesis for why only some users fail and list the first checks you would run.
Read the worked answer
The 20% pattern points to a subset of APM blades or a specific policy branch, not a total LDAP outage. 'session variable not found' usually means the access policy reached a branch that expected an LDAP-derived variable (for example a group or last-attempted result) that was never set because the LDAP agent failed. A certificate rotation can fail only on blades that did not receive the updated CA bundle, or only for users whose policy branch contacts a different LDAP endpoint or health-check path.
First checks: compare APM logs across all blades at the same timestamp; run openssl s_client -connect <ldap-host>:636 -CAfile <bundle> from each blade; inspect the access policy branch for the affected user group and the LDAP agent error; verify the certificate SAN matches the LDAP hostname and that the intermediate chain is complete. Also confirm whether the affected users map to a specific LDAP server or domain controller.
2. Decide between temporary mitigation and rollback, and justify the choice without weakening authentication.
Read the worked answer
Because 80% of users succeed and existing sessions continue, a global rollback is disproportionate and would not fix a blade-local trust problem. Identify the failing scope. If one blade or one LDAP endpoint is affected, remove that blade from service or point the affected branch to a healthy LDAP endpoint that uses the new certificate, provided the same authentication strength is preserved. Update the CA bundle on the affected blade and restart APM services in a maintenance window; test with a real account from the affected group before returning the blade to service.
If the failure is caused by a policy branch that loops on LDAP error, add a terminal error branch that logs the failure and shows a clear message instead of returning to login. Do not disable RADIUS OTP, do not fall back to LDAP-only, and do not put ASM in transparent mode as a workaround.
3. What controls would prevent this from recurring at the next certificate change?
Read the worked answer
Treat certificate rotation as a staged change with a canary group. Inventory every trust store and CA bundle on every APM blade and compare them before the change. Include the full intermediate chain in the new bundle, not just the leaf. Test LDAP over TLS from each blade, then authenticate a real user from each policy branch and each domain controller. Monitor APM logs and ASM events during the canary window before rotating the remainder.
Add certificate expiry and chain-validation monitoring to the operations runbook, and record the LDAP hostname and SAN requirements. After a successful rotation, run a config sync and verify that all blades report the same bundle. Keep a tested rollback bundle available, but do not rely on rollback as the primary control.