Shared Cluster Secrets, Overbroad RBAC, and Unverified Images
A team deploys to a shared Kubernetes cluster. Application secrets are stored as base64-encoded values in a Secret manifest committed to Git and applied with kubectl apply. Pods mount the default service account token automatically, and the CI service account has cluster-admin because it needed to create namespaces once. Images are pulled from public Docker Hub by tag, and a recent scan found 42 HIGH CVEs in a base image. A penetration test found that a compromised frontend pod could query the Kubernetes API for other namespaces' secrets. The platform team wants to adopt GitOps and must satisfy an auditor who asks for least privilege, secret rotation, and evidence of image provenance.
1. Redesign secret handling so that secrets are not stored in Git and can be rotated, while still working with a GitOps workflow.
Read the worked answer
Base64 in a committed Secret is plaintext for practical purposes: anyone with repository read access can decode it, and Git history preserves every value. Remove secrets from Git and rotate every value that has been committed, because history and forks may retain them.
Use an external secret store such as Vault, a cloud secret manager, or SOPS-encrypted secrets with keys held outside the repository. For GitOps, commit only a reference: an ExternalSecret custom resource that syncs from the store into a Kubernetes Secret, or a SealedSecret whose decryption key is scoped to the cluster. The GitOps controller applies the reference, and a controller reconciles the actual Secret. This keeps the desired state in Git without the secret value.
For rotation, make the external store authoritative and use short-lived credentials where possible, such as dynamic database credentials or workload identity. Mount secrets as files rather than environment variables when the application can reload them, because environment variables are fixed at process start. If the application requires environment variables, trigger a rollout on secret change. Add audit logging on secret access, restrict which service accounts and namespaces can read each secret, and test rotation in staging. The auditor should be able to see that no plaintext secret exists in Git, that access is logged, and that rotation does not require a code change.
2. Address the RBAC and service-account token problems that allowed a compromised frontend pod to read other namespaces' secrets.
Read the worked answer
The frontend pod should not have credentials that allow reading secrets at all. First, set automountServiceAccountToken: false on pods that do not need the Kubernetes API, including the frontend. If a pod needs the API, create a dedicated service account with a narrowly scoped Role and RoleBinding in its own namespace, not a ClusterRole. Grant only the specific verbs and resources required, such as get on one ConfigMap, and avoid wildcards.
The CI service account with cluster-admin is the largest risk. Replace it with a namespace-scoped service account for each environment and a Role that allows only the resources CI creates, such as Deployments, Services, and ConfigMaps in the target namespace. If CI must create namespaces, use a separate, tightly controlled role or a provisioning pipeline, not persistent cluster-admin. Disable auto-mounted tokens on CI runner pods and use short-lived projected tokens with an audience and expiration. Audit system:authenticated and default bindings; remove any RoleBinding that grants secret access broadly.
Use kubectl auth can-i --list --as=system:serviceaccount:... to verify effective permissions. Enable audit logging for secret reads and API calls from service accounts. Network policies can also limit a compromised frontend from reaching the API server, but RBAC is the primary control.
3. Design an image supply-chain process that addresses the CVEs, mutable tags, and the auditor's provenance requirement.
Read the worked answer
Start by pinning images by digest, not by tag. A tag such as latest or v1 is mutable, so what runs today may differ from what was scanned. The deployment manifest should reference image@sha256:..., and the GitOps pipeline should update the digest only after the image passes policy.
Build images in CI from a controlled base image that is rebuilt on a schedule. Use a minimal runtime base, remove build tools and devDependencies, and scan the result with a scanner such as Trivy or Grype. Fail the build on policy violations: for example, block HIGH or CRITICAL vulnerabilities with a fix available, and require an exception with an owner and expiry for accepted risk. The 42 HIGH CVEs should be triaged: upgrade the base image and dependencies, rebuild, and rescan. Do not rely on a one-time scan of a mutable tag.
For provenance, generate signed attestations: sign the image with cosign, produce an SBOM in SPDX or CycloneDX, and record the commit, build pipeline, and base image digest. Enforce admission control with a policy engine such as Kyverno or Gatekeeper to reject unsigned images, images without an SBOM attestation, or images from unapproved registries. Mirror approved images to a private registry and pull only from there. The auditor can then trace each running digest to a signed build, a source commit, and a vulnerability scan result.