Test Kitchen matrix explosion and a flaky artifact dependency
The app_deploy cookbook supports Ubuntu 18.04, Ubuntu 20.04, and CentOS 7, with Chef Infra Client 14 and 15. Test Kitchen currently defines 4 suites x 3 platforms x 2 Chef versions = 24 instances. The CI pipeline takes 55 minutes, and about 20% of runs fail intermittently because a recipe calls an internal artifact server that is reachable only from the office VPN, not from the CI runners. ChefSpec covers 38% of resources, mostly the default recipe. The team wants to add Windows Server 2019 support next quarter and reduce feedback time to under 20 minutes without lowering confidence in production changes. You have two CI runners and a shared artifact repository that can be seeded with test fixtures. The cookbook has 12 recipes, but only 4 are exercised in CI.
1. How would you restructure the test matrix and test levels to cut runtime while keeping coverage? Be specific about what to test with ChefSpec, Test Kitchen, and InSpec.
Read the worked answer
The current matrix multiplies every suite by every platform and Chef version, which spends most of its time re-testing the same resource logic. Move fast, deterministic checks into ChefSpec: recipe inclusion, attribute precedence, guard behavior, template variables, and notification wiring. That is where the 12 recipes should be covered cheaply, not only the default recipe. Keep Test Kitchen for convergence behavior that depends on a real platform or package manager, and keep InSpec for post-converge assertions about ports, services, file modes, and package state.
Reduce the full matrix by testing each platform on the newest supported Chef Infra Client in pull requests, and keep the older Chef version only in a nightly or release matrix. Split the four exercised suites so independent suites can run in parallel across the two runners. Do not test all 24 combinations for every change; use a coverage table that maps each recipe to the one or two platforms where its behavior is platform-specific. This should bring pull-request feedback under 20 minutes while preserving a broader release matrix.
2. What is the right way to handle the internal artifact server dependency and secrets in tests? What should be faked, stubbed, or tested against a fixture?
Read the worked answer
The flakiness comes from making CI depend on a VPN-only service. Separate the cookbook logic from the artifact location. In ChefSpec, stub the remote fetch resource and assert the URL, checksum, and notification behavior; do not perform network I/O. In Test Kitchen, point the cookbook at a fixture artifact server on the CI network or a seeded repository path that serves a small test artifact. That tests the download, checksum, and install path without depending on the office VPN. If the real artifact server must be tested, put that in a separate scheduled integration job, not in every pull request.
Secrets should never be committed as data bag items or attributes. Use environment variables or a CI secret store to inject a fake credential for tests, and assert that the recipe consumes it without logging it. If the cookbook uses chef-vault or an external secret store, test with a test-only key and a fixture secret. The rule is: stub external network and real secrets at the unit level; use a controlled fixture at the integration level; reserve the real service for a small, non-blocking verification job.
3. Which checks should gate a pull request versus a release, and how would you add Windows support without doubling the matrix?
Read the worked answer
A pull request should gate on fast, high-signal checks: cookstyle, ChefSpec for all changed recipes, a single Kitchen converge on the primary Linux platform with InSpec verification, and the fixture artifact test. That catches syntax, unit logic, and basic convergence without paying for the full platform matrix. A release should gate on the full supported platform matrix, both Chef versions, the fixture and real artifact checks, and a canary converge against a representative node.
For Windows, add one Windows Server 2019 platform and one focused suite that covers the Windows-specific resources: package installation, service management, directory creation, and ACL behavior. Do not add Windows to every existing suite. Use platform-specific attributes and guards so the same recipe can be tested without duplicating the entire Linux matrix. If a recipe has no Windows code path, mark it not applicable in the coverage table. This adds coverage where it matters and keeps PR feedback time bounded, while the release job can afford the broader combinations.