Single admission replica causes cluster-wide webhook outage
Kyverno 1.12 is installed with Helm into the kyverno namespace on a 3-worker-node cluster. The chart was installed with admissionController.replicas=1 and no PodDisruptionBudget or anti-affinity. The admission controller's webhook failurePolicy is left at the default. During a node drain for a Kubernetes upgrade, the only Kyverno pod is evicted and does not become Ready for 4 minutes. During that window, 2,200 Pod CREATE requests across all namespaces fail with 'failed calling webhook validate.kyverno.svc-fail: context deadline exceeded', including kube-system DaemonSets. Existing Pods keep running. The platform team proposes setting the webhook failure policy to Ignore and lowering webhookTimeoutSeconds to 2 so drains never block the cluster. The security team opposes this because it would silently bypass all policies during an outage. You must make the installation resilient without weakening enforcement cluster-wide.
1. Explain why a single Kyverno pod caused a cluster-wide admission outage, including the role of failurePolicy and the webhook scope.
Read the worked answer
Kyverno registers cluster-wide ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects. The API server calls them for every matching create or update, not just for one namespace. With the default failurePolicy: Fail, if the webhook cannot be reached or does not answer before the timeout, the API server rejects the request. A single admission replica means there is no other endpoint during the eviction; the pod may be rescheduled, but image pull and readiness can take minutes. The 4-minute gap is enough for every matching Pod CREATE to fail, including kube-system, because the webhook is not namespace-scoped to application namespaces. Existing Pods are unaffected because no admission request is made for them.
The webhook timeout and failurePolicy are cluster-wide properties of Kyverno's webhook configuration, so this is an availability design problem, not a policy bug. Check with kubectl get validatingwebhookconfiguration and kubectl get mutatingwebhookconfiguration; inspect failurePolicy, timeoutSeconds, and the service backends.
2. What changes would you make to the Helm installation and cluster objects, and why not set failurePolicy: Ignore globally?
Read the worked answer
Make the admission controller highly available: set admissionController.replicas=3, spread replicas across nodes with pod anti-affinity or topology spread constraints, and create a PodDisruptionBudget that prevents voluntary disruption from taking more than one admission pod at a time, for example minAvailable: 2. Use a priorityClass so the admission pods are not the first evicted. Keep failurePolicy: Fail for enforcement. Do not set Ignore globally: that converts every webhook outage into a silent bypass of all Kyverno mutate and validate policies, including the controls that protect critical namespaces. If the organization cannot tolerate fail-closed behavior, the correct fix is more replicas and disruption controls, not a global Ignore. A narrower option, only if the policy set allows it, is to exclude namespaces that never need policy enforcement from the webhook configuration, but that must be a deliberate policy decision. Verify the chart's exact value names for the installed version before applying via Helm; the important objects are the Deployment replicas, PodDisruptionBudget, and webhook configuration.
3. How would you verify the new setup before the next drain?
Read the worked answer
Before the next drain, confirm three Ready admission pods on distinct nodes with kubectl get pods -n kyverno -o wide; confirm the PDB with kubectl get pdb -n kyverno; confirm the webhook configuration still has failurePolicy: Fail and the expected timeoutSeconds; then do a canary eviction. Delete one admission pod, or drain a node with --dry-run=server first, and while it is unavailable create a test Pod in a non-critical namespace; it should still be admitted by the remaining replicas. Repeat with a policy that should deny and confirm the deny still happens. Check kubectl get policyreport -A after the test to ensure reports are still produced. Finally, run the drain and watch the webhook failure metrics and logs; the drain should complete without a cluster-wide admission gap.