A Failing Admission Webhook and a Denied Egress Path

failurePolicy: Fail means the API server rejects the request when the webhook cannot be reached; that is deliberate and correct for a security control, but it converts the webhook into a hard dependency of every write it matches. Two design choices made the blast radius total: an unfiltered namespaceSelector matching 14 tenant namespaces, and a single replica with no disruption budget, so a routine drain removed the only backend.

A defensible configuration:

• Run at least 2-3 Gatekeeper replicas across zones, with a PodDisruptionBudget and priorityClass so they outrank tenant workloads during eviction.
• Narrow the namespaceSelector with a label (for example policy.example.com/enforce: "true") so unlabelled namespaces are not matched, and keep the system namespaces excluded.
• Set timeoutSeconds to 5-10, below the API server's own request budget, so a slow webhook fails fast rather than stalling controllers.
• Keep failurePolicy: Fail for provenance and privilege policies but add a labelled break-glass namespace that the selector excludes, with its use logged and reviewed.
• Alert on apiserver_admission_webhook_rejection_count and webhook latency, and on the Gatekeeper pod count dropping below quorum.

Setting failurePolicy: Ignore would restore availability by silently disabling the control exactly when the cluster is unstable -- the worst possible moment to stop enforcing image provenance.

The connection timeouts are a DNS failure wearing a disguise: with default-deny egress, the pods cannot resolve redis.data.svc.cluster.local, and depending on client behaviour that surfaces as a timeout to the resolved-name attempt rather than an obvious lookup error. Always confirm with a resolver test from inside the pod before touching L4 rules.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: payments-egress, namespace: payments}
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: {kubernetes.io/metadata.name: kube-system}
          podSelector:
            matchLabels: {k8s-app: kube-dns}
      ports:
        - {protocol: UDP, port: 53}
        - {protocol: TCP, port: 53}
    - to:
        - namespaceSelector:
            matchLabels: {kubernetes.io/metadata.name: data}
          podSelector:
            matchLabels: {app: redis}
      ports:
        - {protocol: TCP, port: 6379}

Note that both selectors must sit inside the same to element to be ANDed; split across two elements they are ORed and the rule silently widens. The ipBlock on 10.96.0.0/12 fails for a subtler reason: network policy is evaluated on the packet's pod-to-pod addresses after service translation, so a rule written against ClusterIPs does not match the backend pod IPs -- it either has no effect or, if it does match, it grants egress to every service in the cluster. Select pods, not service VIPs. The matching ingress policy in data must allow TCP 6379 from the payments namespace, or the connection still fails.

Enforcement without a migration path produces exactly the outcome observed: teams work around the platform, or the platform is pressured into failurePolicy: Ignore. A staged path is faster in practice.

1. Shadow first. Ship the constraint with enforcementAction: warn (or dryrun). Writes succeed, violations are recorded in the constraint status and in Gatekeeper audit metrics, and you can quantify how many manifests each tenant must change. Publish that number per tenant.
2. Publish the contract. Document the allowed registries and the exemption mechanism before enforcement, not after the first outage.
3. Grant narrow, expiring exceptions. A Constraint exception list with a justification annotation and an owner, reviewed on a fixed cadence. An exception with no expiry becomes permanent architecture.
4. Enforce per namespace as tenants go green, using the label selector rather than a cluster-wide flip, so a straggler cannot hold the whole fleet hostage or block everything when the webhook is unhealthy.
5. Test policy in CI. Run gator test or Conftest against tenant manifests in their pipelines so violations fail before merge, not at apply time during an incident.

Governance signal to report monthly: violation count by namespace and constraint, exception count and age, webhook latency, and any request rejected while the webhook was unavailable.