Monorepo CI: Cache Misses, Flaky Integration Tests, and Manual Release Tags
Your team runs a monorepo with 40 Go services and 12 Node.js frontends. The CI pipeline on every pull request takes 38 minutes: it builds all images, runs unit tests, then a single integration suite against a shared staging namespace. The Docker build step almost always misses its layer cache because the Dockerfile starts with COPY . . before go mod download. Integration tests fail about 15% of runs; rerunning the same commit usually passes. Developers have started batching unrelated changes into large pull requests to avoid waiting, and release tags are created manually from whatever commit happens to be on main on Friday. Last week a hotfix was tagged from a feature branch and shipped without the integration suite. The engineering manager wants the pipeline under 15 minutes and fewer flaky failures, without adding more CI runners this quarter.
1. Explain why the Docker layer cache is being missed and describe the Dockerfile and CI changes you would make to fix it without introducing stale dependencies.
Read the worked answer
The cache miss is caused by ordering. COPY . . copies the entire repository, including source files that change on nearly every commit, before go mod download. Docker invalidates a layer and every layer after it when the input to that layer changes, so the dependency download layer can never be reused. The fix is to copy only dependency manifests first, download dependencies, then copy source. For Go, copy go.mod and go.sum, run go mod download, then copy the rest. For Node, copy package.json and the lockfile, run npm ci, then copy source. This creates a stable dependency layer that is reused until the lockfile changes.
In CI, build with BuildKit and an external cache such as a registry-backed cache: docker buildx build --cache-to type=registry,ref=... --cache-from type=registry,ref=.... Scope the cache per service and per architecture so parallel jobs do not fight over one cache. Add .dockerignore entries for .git, node_modules, test fixtures, and build output so unrelated file changes do not invalidate context. To avoid stale dependencies, make lockfile changes an explicit cache boundary and fail the build if go mod tidy or npm ci would modify the lockfile. Finally, only build images for services affected by the change, determined from the diff, rather than rebuilding all 52 images on every pull request.
2. The integration suite is flaky and runs against a shared staging namespace. Propose a test strategy that shortens feedback time and reduces flakiness, including how you would classify and quarantine failures.
Read the worked answer
The shared staging namespace is a likely source of nondeterminism: tests from parallel pull requests mutate the same data and compete for the same services. First, split the suite by scope. Run fast unit tests and contract tests on every pull request. Run integration tests that need real dependencies only for affected services, using ephemeral namespaces or per-run Docker Compose environments created from a unique namespace name and torn down afterward. Seed deterministic data inside the test run and avoid depending on data left by other runs.
Classify failures by cause: infrastructure (timeouts, DNS, image pulls), test isolation (shared state, fixed ports, clock assumptions), and product defects. Record the classification and the failing test owner. Quarantine tests that fail intermittently without a product bug into a separate non-blocking job, with an owner and an expiry date; do not silently skip them. Rerunning a failed test once can distinguish a true intermittent failure, but do not use automatic reruns to hide a suite that is 15% flaky. Track flakiness as a metric per test and fix or delete the worst offenders.
To cut wall-clock time, parallelize independent test shards, cache dependencies, and run the integration suite only after unit tests pass. Move long end-to-end tests to a post-merge or nightly pipeline, and keep the pull request gate focused on fast, deterministic checks.
3. Recommend a branching and release-tagging approach that prevents a hotfix from being shipped without the integration suite, while keeping the team's current GitHub-based workflow.
Read the worked answer
The current process fails because tagging is manual and decoupled from verification: a tag can point at any commit, including one from a feature branch, and nothing enforces that the commit passed the required checks. Move release creation into an automated pipeline that only tags commits that have passed the full required check set on the main branch. Protect main with branch protection: require pull requests, required status checks including the integration suite, and no direct pushes. Use short-lived feature branches merged by squash or merge commit, which fits the existing GitHub workflow.
For releases, use semantic version tags created by a release workflow, not by hand. The workflow should verify that the target commit is an ancestor of main, that required checks are green, and that the build artifact was produced from that exact commit. Sign the tag or use a release attestation so the artifact can be traced. If a hotfix is needed, branch from the release tag, apply the fix, and run the same required checks before tagging a patch release; do not tag directly from a feature branch.
This gives the manager a simple rule: no tag without a green pipeline on a commit reachable from main or from a release branch. It also removes the Friday manual step by making releases reproducible and auditable.