LDAP Certificate Rotation Breaks SSSD and Local sudo

passwd: files sss means glibc asks files first, then SSSD. A local user present in /etc/passwd should resolve from files without contacting SSSD. But the ordering does not protect against a slow second source for other lookups. group: files sss is consulted for supplementary groups via initgroups, and sudoers: files sss forces sudo to query the SSSD sudo responder. If SSSD's LDAP provider is stuck connecting to the new load balancer, the NSS, PAM, and sudo responders can block waiting on that provider, so even local lookups and sudo feel slow.

The 25-second hang is consistent with LDAP connection timeouts plus retries under a TLS handshake that either never completes or fails certificate validation. SSH also calls PAM (pam_sss, pam_systemd), and pam_sss waits on the same responder. Directory users fail because SSSD has no valid provider and, if cache_credentials is false or the cache is cold, cannot authenticate offline. The engineer's console root session works because it came from the local console and files, not because the system is healthy.

Reject the proposal as stated. enumerate = true makes SSSD download the entire directory on every refresh, which is expensive for 400 hosts and an LDAP service already behind a broken TLS endpoint, and it does not address trust. Restarting SSSD before the certificate is fixed also clears the cache, removing any cached credentials that could carry users through the outage while the provider still cannot connect.

Safer sequence:

1. Keep the console root session. Confirm break-glass access: a local account in /etc/passwd with a password and a rule in /etc/sudoers.d/ that does not depend on sss.
2. Diagnose without restarting: sssctl domain-status <domain> -o, journalctl -u sssd -n 200 --no-pager, and openssl s_client -connect newlb.example.com:636 -servername ldap.example.com -CAfile /etc/openldap/certs/ca.crt. A hostname mismatch or untrusted chain will be explicit.
3. Fix the trust: point ldap_uri at the name in the certificate, install the issuing CA at ldap_tls_cacert, and keep ldap_tls_reqcert = demand. Do not set never. If service must be restored before a new certificate is issued, allow still validates the chain and only skips hostname checking; time-box it and open a ticket.
4. Run sssctl config-check before restarting. Restart SSSD on one canary host, then test getent passwd <user>, id <user>, sudo -l -U <user>, and a real SSH login.
5. Roll out through configuration management in waves. Do not run sss_cache -E until the provider is healthy; clearing the cache during an outage removes offline authentication.

Settings:

• cache_credentials = true with entry_cache_timeout = 600, so identity lookups survive a provider outage.
• Bounded network timeouts: ldap_network_timeout = 3, ldap_opt_timeout = 5, dns_resolver_timeout = 5. A broken VIP should fail in seconds, not 25.
• ldap_tls_reqcert = demand with a managed CA bundle, plus certificate-expiry alerts for the load balancer.
• Keep files before sss in every nsswitch database. If sudo rules do not truly need to be central, manage them in /etc/sudoers.d/; if they do, keep the bounded timeouts and test the sudo responder explicitly.
• Verify PAM ordering for pam_mkhomedir, pam_sss, and the TOTP module with a canary account.

Monitoring:

• Export sssctl domain-status <domain> online/offline state from the monitoring agent.
• Time getent passwd <directory-user> and id <user> every minute; alert above 2 seconds.
• Alert on journal messages containing TLS, certificate, Cannot contact, or Offline.
• Synthetic LDAP StartTLS check from each site, plus certificate expiry monitoring.
• An hourly canary SSH login with a dedicated directory account that does not require TOTP, exercising NSS and PAM end to end.