Intermittent SAML Failures After an IdP Certificate Renewal
A Workday SAML integration fails for roughly 30% of users after the Okta IdP signing certificate was renewed. Users who launch the app from the Okta dashboard always succeed; users who start from Workday's own login page fail with 'Invalid signature on SAML response' or a NotBefore timestamp error. The app sits behind a load balancer fronting three ACS nodes. Node 2's system clock is four minutes twenty seconds behind, and the trust configuration on node 2 was never updated after the renewal. Okta's System Log shows user.authentication.sso with outcome SUCCESS for every failing user. Two of the failures come from a partner IdP that posts into the same application, and the app still negotiates SHA-1. Leadership wants a root cause today.
1. Give a triage sequence and state what each data source rules in or out.
Read the worked answer
Start with the shape of the failure, because 30% is a number that points at infrastructure.
- Split the population by entry point. Dashboard-initiated logins always succeed, SP-initiated logins fail. That immediately rules out Okta's signing key being wrong or the IdP metadata being stale, and points downstream of Okta.
- Split by node. 30% against three roughly balanced nodes is very close to one node in three. Check load balancer logs to see which node served the failing requests. If failures cluster on node 2, the investigation is over: this is configuration drift, not an intermittent bug.
- Check trust material on node 2. A node that never received the renewed certificate will reject an assertion signed by the new key while nodes 1 and 3 accept it. That is the signature error.
- Check time. The four-minute-twenty-second clock skew explains the NotBefore error independently. SAML tolerance is small; a slow node will reject assertions that are valid elsewhere even with correct trust material.
- Separate the partner IdP failures. Two failures from a different issuer are a different problem, not evidence of an Okta-wide issue. Pull the failing assertions' Issuer and Audience values; the likely causes are a partner certificate mismatch or an audience restriction.
- Treat SHA-1 as a risk finding, not the cause. It will not produce an intermittent per-node failure, but it belongs in the remediation plan.
Data sources: load balancer access logs, node-level SP logs with full SAML responses, Okta System Log filtered to the affected users, and the partner's metadata document.
2. Why does Okta's System Log report success while the user sees an error?
Read the worked answer
Because Okta's event records what Okta did, not what the application did with it.
The user.authentication.sso event is written after Okta has authenticated the user, built the assertion, signed it, and posted it to the ACS URL. From Okta's perspective the flow completed. The service provider then validates the signature, the issuer, the audience, and the time window, and rejects the assertion. That rejection never travels back to Okta, so the log entry remains SUCCESS and the user sees an error page.
This is the most common misinterpretation in SAML troubleshooting: 'Okta says success' is not evidence that the login worked. The only trustworthy record of the failure is on the service provider side, or in a browser trace.
To correlate the two sides, capture the assertion ID from the SP's error detail and match it against the Okta event, or use the external session ID. That gives you a single join key across both logs and removes the ambiguity.
The general principle worth stating to leadership: Okta is authoritative for authentication, the SP is authoritative for authorisation and assertion acceptance. A 'successful' Okta event only proves the first half of the transaction.
3. List the fixes and the preventive control that stops this recurring at the next renewal.
Read the worked answer
Immediate fixes, in order:
- Update the trust configuration on node 2 with the current IdP signing certificate. Better, push the same configuration to all three nodes from your configuration management system so they cannot drift individually again.
- Correct the clock on node 2 with a working NTP source, then verify offset is under a second. Certificate trust and clock skew are independent faults that happened to coincide, which is why the failure rate looked erratic.
- Remove node 2 from the load balancer pool until both fixes are verified, then return it and test SP-initiated login specifically against each node.
- Resolve the partner IdP failures separately by confirming their current signing certificate and audience value against your configuration.
- Plan the SHA-1 to SHA-256 migration with the application owner, with a date. It is not the root cause here but it will become one.
Preventive controls:
- Publish the new signing certificate with an overlap period: distribute the new cert to all SPs before the old one is retired, rather than swapping atomically.
- Add a synthetic monitoring check that performs an SP-initiated login and asserts a successful landing page, running per node, not just against the load balancer. A per-node check would have caught this within minutes.
- Put SP trust configuration and certificate expiry dates into the same change-control and monitoring system as the Okta tenant.
- Alert on NTP drift across all SAML endpoints.