A CI service account with cluster-admin, and a kubelet that answers to anyone
A kubeadm v1.29 cluster, three control-plane and three worker nodes. The CI pipeline authenticates as ServiceAccount deployer in namespace ci, and a single ClusterRoleBinding grants that ServiceAccount cluster-admin; the pipeline uses it to apply Deployments and ConfigMaps into four namespaces. kubectl get pods -A -o yaml shows 38 of 41 pods mounting the default service account token. The kube-apiserver static pod manifest sets --authorization-mode=Node,RBAC but contains no --anonymous-auth flag. On every node, /var/lib/kubelet/config.yaml contains authentication.anonymous.enabled: true and authorization.mode: AlwaysAllow. A developer with SSH access to a control-plane node says they only ever use it to tail logs. Nothing is currently broken and no alert has fired.
1. You must remove the cluster-admin binding without stalling the pipeline. What do you need to establish first, and what exactly do you replace it with?
Read the worked answer
First establish what the pipeline actually calls, because guessing produces a least-privilege role that breaks the build at 2am. Audit server-side with kubectl auth can-i --list --as=system:serviceaccount:ci:deployer (needs impersonate on serviceaccounts) and, better, read the API audit log or run the pipeline against a cluster with a ClusterRole limited to get,list,watch plus dry-run and see what it 403s on. Also check whether anything else in ci reuses the token.
Replacement: a namespaced Role in each of the four target namespaces, bound by four RoleBindings to the same ServiceAccount, granting only get,list,watch,create,update,patch on deployments.apps, configmaps, services - and not on secrets unless the pipeline genuinely writes them. If four Roles is unmanageable, a single ClusterRole with those resource names and verbs is still a large improvement over cluster-admin, because it excludes RBAC objects, nodes, pods/exec, persistentvolumes and wildcards. Explicitly confirm the new role cannot escalate: no escalate, bind, impersonate, no create on roles/rolebindings.
RBAC is evaluated per request, so deleting the ClusterRoleBinding takes effect immediately even for pods still holding an unexpired bound token - you do not have to wait for token expiry. Do rotate the ServiceAccount (kubectl delete sa deployer -n ci after moving the binding, or rotate the projected token) only if you believe the token leaked, since that forces a pipeline credential refresh.
2. Write the remediation for the kubelet configuration, state how you apply it safely on three worker nodes, and say how you verify it.
Read the worked answer
Desired /var/lib/kubelet/config.yaml fragment:
authentication:
anonymous:
enabled: false
webhook:
enabled: true
x509:
clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
mode: Webhook
With AlwaysAllow, any process that can reach port 10250 gets full kubelet API access - including pods/exec and log retrieval - with no credential at all. Anonymous access must go, and authorization must delegate to the API server so that normal RBAC on nodes/proxy, nodes/log, nodes/metrics and nodes/stats governs who can reach the kubelet.
Apply node by node, not to the whole fleet at once: edit the file, systemctl daemon-reload && systemctl restart kubelet, then immediately confirm kubectl get nodes shows the node Ready and that a pod can still be scheduled and exec'd there. Rolling it one node at a time keeps the metrics-server and log pipelines available; a bad kubelet config takes the node out of service, and on a kubeadm control-plane node the kubelet restart also restarts the static pods.
Verify: curl -k https://<node-ip>:10250/pods must return 401 Unauthorized, not a pod list. Then confirm the legitimate consumers still work: kubectl top nodes (metrics-server uses the system:kubelet-api-admin ClusterRole or nodes/metrics), kubectl logs, and kubectl exec. In the same change, add --anonymous-auth=false to the kube-apiserver static pod command line, since it currently defaults to allowing anonymous requests; the kubelet restarts the apiserver pod automatically.
3. Before touching anything, how do you find every other place in this cluster where the same class of exposure exists?
Read the worked answer
Enumerate bindings, not roles, because the binding is what grants: kubectl get clusterrolebindings,rolebindings -A -o json and filter for roleRef.name values of cluster-admin, admin, edit and any custom role containing wildcards. For each subject, resolve whether it is a human, a group, or system:serviceaccount:<ns>:<name> - a system:masters group membership in a client certificate is equivalent to cluster-admin and is invisible to RBAC auditing, so also inspect the API server's client CA and kubectl config view users.
Look for risky verbs rather than just risky role names: escalate, bind, impersonate, create on pods/exec and pods/attach, get/list on secrets, and write access to mutatingwebhookconfigurations or validatingwebhookconfigurations (a webhook that intercepts everything is a persistence mechanism). Kyverno, Gatekeeper, Argo CD and the CNI typically hold broad permissions by design - verify they are namespaced or resource-scoped where possible and that their ServiceAccounts cannot be impersonated.
Cross-check the human path too: SSH access to control-plane nodes is equivalent to root on the cluster, because /etc/kubernetes/admin.conf and the static pod manifests live there. Restrict SSH to a bastion with session recording, or remove it in favour of kubectl debug node/... under RBAC.