CI runners on a shared AppRole secret_id, 30-day deadline

Fixable now - shrink the blast radius.

• secret_id_ttl (e.g. 10m) and secret_id_num_uses=1: a leaked secret_id becomes worthless almost immediately.
• secret_id_bound_cidrs to the runner network, so an exfiltrated value cannot be replayed from a laptop.
• Remove the secret from the image. Fetch a secret_id at job start via a response-wrapped write: vault write -wrap-ttl=60s auth/approle/role/ci-deploy/secret-id. The wrapping token is single-use, short-lived, and the payload is readable only from the holder's own cubbyhole/ - which the default policy already allows.
• Tighten token_max_ttl to match the longest job, and attach only the policies each job family needs instead of one shared policy.

Not fixable in 30 days - attribution. One role means one identity: every runner maps to the same entity, so audit logs and incident response still cannot name the offending runner. That is the argument for migrating, not just the leaked secret.

Watch the new weak point. Minting secret_ids requires a token with update on auth/approle/role/ci-deploy/secret-id. That broker token is now the crown jewel - keep it on the CI server, never on the runners.

Pods → Kubernetes auth. Enable auth/kubernetes, configure kubernetes_host and a reviewer JWT, then bind identity tightly:

vault write auth/kubernetes/role/ci-deploy \
  bound_service_account_names=ci-runner \
  bound_service_account_namespaces=ci \
  audience=vault token_ttl=20m token_max_ttl=8h

Each pod presents its projected service-account token; no static secret exists anywhere. Set alias_name_source=serviceaccount_name if you want stable alias names rather than UIDs.

EC2 → AWS IAM auth. Enable auth/aws and log in with the instance profile; Vault verifies the identity by calling sts:GetCallerIdentity, so there is no secret on disk. Bind with bound_iam_principal_arn (or bound_account_id plus bound_arn_prefix) so a role in another account cannot authenticate.

One policy, two doors. Policies attach to tokens, not mounts, so give both roles the same policy name - but the durable answer is an identity group: create an external group ci-deployers holding the policy, then add group aliases pointing at the Kubernetes and AWS mount accessors. New runners inherit policy by group membership instead of by editing two roles.

Fallback: if STS is blocked from the VPC endpoints the 40 hosts need, keep AppRole for them with CIDR binding - a smaller, well-understood exception is better than an untested workaround.

Test the effective capabilities, not the policy file.

1. Log in as a canary pod and a canary EC2 instance, then run vault token capabilities <token> kv/data/deploy/app1 and database/creds/ro. Compare against the expected list; a wildcard in a policy file hides gaps that only show up per path.
2. Check the token shape with vault token lookup: TTL, max TTL, whether default is attached, and the entity/alias it resolved to. If the alias mapping is wrong the group grants nothing and the job just gets 403s.
3. Run a real deploy job end to end, including one that exceeds the old 1h window.

Most likely silent breakage: TTL and renewal behaviour. AppRole gave a fixed 1h token with max_ttl=1h (unrenewable); a Kubernetes or IAM token with a different TTL may expire mid-job, and a client that never called renew-self will now fail late rather than early. Also confirm the audience and bound namespace are narrow enough that a different namespace's service account cannot borrow the role.

Keep the AppRole role mounted but stop issuing secret_ids, and destroy outstanding secret_id accessors (auth/approle/role/ci-deploy/secret-id-accessor/destroy) so old values cannot be replayed during the overlap.