CLI unit test cannot resolve a cluster ConfigMap context
A platform team keeps Kyverno policies in Git and runs Kyverno CLI 1.11 in GitHub Actions with kyverno test. The cluster runs Kyverno 1.12. One policy, disallow-host-path, uses a context ConfigMap named platform-settings in the kyverno namespace to read allowedPaths, and a variable from that ConfigMap in a deny condition. The test manifest lists the policy, two Pod YAMLs, and expected results. Locally, a developer with CLI 1.12 and a kubeconfig runs kyverno test . and all cases pass. In CI, the same commit fails with 'variable not found' before any test result is evaluated. A teammate suggests adding --cluster to the CI command so the CLI can read the ConfigMap. The security team wants the CI check to remain deterministic and not require cluster credentials.
1. Diagnose the CI failure and evaluate the --cluster proposal.
Read the worked answer
The CLI is offline by default. A rule that resolves a context entry from a ConfigMap needs either cluster access or a local substitute. In CI, no kubeconfig is provided, so the ConfigMap lookup cannot resolve and the variable is empty or missing; the failure occurs before test evaluation. The local developer has a kubeconfig and CLI 1.12, so the lookup succeeds. The version difference is a second risk: policy schema and CLI context handling can differ between 1.11 and 1.12.
Adding --cluster would make the CI job read the live ConfigMap. That requires kubeconfig credentials and RBAC in CI, makes the test depend on mutable cluster state, and can pass or fail because someone edited the ConfigMap. It is an integration test, not a unit test. Do not use it as the primary gate.
2. Design a deterministic CI test strategy and align CLI versions with the cluster.
Read the worked answer
Pin the CLI image to the same minor version as the cluster, 1.12.x, and fail the job if kyverno version does not match the expected value. Remove the live ConfigMap dependency from the unit test: define the allowed paths as a policy variable and supply it in the test manifest's variables block or through a values file. Keep a separate integration job that runs against an ephemeral kind cluster with Kyverno 1.12 installed, creates the ConfigMap, and then runs the same test. In the test manifest, make expected results explicit for both allow and deny cases and reference the exact policy and rule names; remember that a resource that does not match any rule is skip, and an expected pass on a skipped resource will fail. If the policy must keep the ConfigMap context in production, split it into a pure deny rule that is unit-testable and a thin context-dependent rule that is tested in the integration job. This keeps the fast CI gate deterministic while still testing the real ConfigMap path.