Service-to-Service API Calls Fail Audience and Secret Rotation
Forty internal microservices call a billing API through an API gateway. Each service has its own Okta OIDC client and uses the client credentials grant. Twelve services were registered against Okta's org authorization server; their tokens have aud=api://default, and the gateway rejects them because it requires aud=billing-api and scope billing.write or billing.read. The other 28 use a custom authorization server but share one client ID and secret across all services, so audit logs cannot identify the caller. Secrets are stored in Kubernetes ConfigMaps and rotated every 90 days. During the last rotation, 5% of calls failed with 401 for about ten minutes because pods kept using cached tokens and old secrets. Security wants per-service identity, no long-lived shared secrets, and a rotation process with no outage. You need to choose the token model and authentication method, and explain how to debug the 401s.
1. How should the authorization server, audience, and scopes be set up for these services?
Read the worked answer
The org authorization server is the wrong issuer for a custom billing API audience. Use a custom authorization server that defines an audience of billing-api and the scopes billing.read and billing.write. Give each microservice its own client so token issuance and audit logs identify the caller. Assign only the scopes that service needs; a read-only pricing service should not receive billing.write.
Each service requests the client credentials grant with its own client ID and the required scope. The gateway validates iss for that custom authorization server, aud=billing-api, exp, nbf, alg, and kid via JWKS, and then enforces scope for the route. Do not validate only the signature or the presence of a scope claim. A service calling another service should use its own token, not forward an end-user token, unless the API explicitly supports on-behalf-of semantics.
2. What client authentication and key rotation approach meets the no-shared-secret requirement?
Read the worked answer
Prefer private_key_jwt for each service. Each service generates a key pair, registers the public key with its Okta client, and signs a short-lived client assertion with a kid. This removes client secrets from Kubernetes entirely and gives per-service cryptographic identity. Keep private keys in a secret manager or KMS-backed volume, not in a ConfigMap.
Rotate keys with overlap: publish the new public key under a new kid, wait for Okta and the service to recognize it, switch signing to the new key, then retire the old key after the overlap window. If a shared secret is unavoidable, store it in a secret manager, use client_secret_basic over TLS, and rotate with a documented overlap if the authorization server supports multiple active secrets; otherwise plan a brief deploy-coordinated cutover. Cache access tokens only until exp minus a small skew, and never cache a token past its expiry during rotation.
3. During secret or key rotation, some services get 401. How would you debug it systematically?
Read the worked answer
First decode the rejected token and check iss, aud, exp, nbf, scope, and kid. A 401 before the API logic usually means the gateway rejected the token, so inspect gateway logs for the exact validation failure. Common causes are a token from the org authorization server instead of the custom one, a missing scope, a stale JWKS cache after key rotation, clock skew, or a pod still using the old client credentials.
If the kid is unknown, force a JWKS refresh and confirm the gateway can reach the JWKS URI. If the token is valid but expired, check token caching and pod clocks. If the token endpoint returned an error, inspect the service's client assertion audience, signing key, and system time. Correlate with Okta System Log token events and include the Okta request ID in service logs. Fix rotation by overlapping keys and by refreshing JWKS on unknown kid, not by disabling signature or audience validation.