Centralizing Secrets for a Multi-Cloud Microservice Fleet
A SaaS company runs 180 microservices across AWS and Azure, plus an on-prem Kubernetes cluster. Each service currently stores database passwords and API keys in environment variables injected at deploy time from CI/CD. The security team wants to move to CyberArk Secrets Manager (Conjur) as the single source of truth. Constraints: (a) services must authenticate to the secrets manager without a human and without a 'master' secret stored in code; (b) a subset of 20 services run in the on-prem cluster and cannot reach the cloud secrets endpoints directly; (c) the team wants to avoid a big-bang migration; (d) auditors want to prove that a given secret was only ever read by the services entitled to it. The platform lead proposes: 'Put one shared API key in each service's environment and have it call the secrets API at startup.'
1. Critique the proposed shared-API-key-in-environment approach against the constraints.
Read the worked answer
The proposal recreates the problem it's meant to solve: a long-lived shared API key in an environment variable is just a new static secret, now with access to all secrets. Specific failures against the constraints:
- (a) No human, no master secret in code: an API key in an env var is a static credential - if one service is compromised, the attacker gets a key that may read many secrets. It's a master secret in disguise.
- (d) Proof of least privilege: a shared key makes it impossible to prove which service read which secret - every service authenticates as the same identity. Audit would show 'the shared key read it,' not the entitled service.
- Rotation: rotating the shared key means touching all 180 services at once - the opposite of safe, incremental change.
The core error is using a bearer credential where the design needs workload identity: each service should authenticate as itself, with no stored long-lived secret, so access is both least-privilege and attributable.
2. Propose an authentication and migration design that satisfies (a)-(d), including the on-prem cluster.
Read the worked answer
Authentication (a): use platform-native workload identity so no static secret is stored.
- AWS: IAM roles for the workload (e.g., IRSA / instance or task roles) - the service authenticates via its cloud identity, no key in env.
- Azure: managed identities for the workload.
- Kubernetes (on-prem): Kubernetes service account identity (e.g., projected service-account tokens) so the pod authenticates as its service account. Each service maps to its own secrets-manager identity/host, so entitlements are per-service.
On-prem reachability (b): the 20 on-prem services can't reach cloud endpoints directly - deploy a local secrets-manager follower/agent in the on-prem cluster that authenticates outward and serves secrets locally, or front it with a controlled egress proxy. The services talk to the local endpoint; the follower handles the cloud path. This keeps the workload identity model intact.
Incremental migration (c): move in waves - start with a handful of low-risk services, prove the identity + retrieval path, then expand. Run dual-path (old env var + new retrieval) per service during cutover, then remove the env var. No big bang.
Auditability (d): because each service has a distinct identity and entitlement, the secrets manager's audit log shows which identity read which secret and when. Combine with deny-by-default entitlements so any read outside the entitled set is both impossible and visible. That's the evidence auditors want - attributable, least-privilege access, not a shared key.