Slow Monorepo Pipeline at Helios Logistics
Helios Logistics has a monorepo with 9 services and 1,400 tests. CI averages 52 minutes: 18 minutes building all Docker images even when only one service changed, 24 minutes running the full test suite serially, and 10 minutes of linting. About 12% of test runs fail for reasons unrelated to the change, and engineers habitually click "retry" twice before investigating. The team uses long-lived feature branches - 41 open - with median time from pull request to merge of 3.5 days; the worst merge took two days of conflict resolution. Production deploys happen manually from a release branch on Thursdays, and a recent incident was caused by a staging-only config value that differed from production. Secrets live in Jenkins credentials, but two services read database passwords from a committed .env file. Developers report they batch changes because "CI is too slow to run more than once a day."
1. Rank the highest-leverage improvements and justify the one you would do first.
Read the worked answer
The pipeline is slow because it is undifferentiated: it rebuilds and retests everything for every change, then compensates with long branches.
Ranking by leverage:
- Affected-only builds and test selection, plus parallelism. Path filtering and dependency graphs can cut the 18-minute image build to a few minutes for single-service changes. Sharding the 1,400 tests across runners and running the fast unit suite before integration tests can take 24 minutes to roughly 6-8. Expected total: under 15 minutes.
- Flaky test quarantine. 12% flakiness is the reason 52 minutes is paid repeatedly; it also erodes trust in red builds.
- Trunk-based development with short-lived branches and feature flags, collapsing the 3.5-day merge queue and the two-day conflict.
- Config parity and secrets hygiene, which caused an actual incident.
Do (1) first, because it is the enabling constraint. Everything else depends on fast feedback: trunk-based development is only safe with a quick, trustworthy pipeline; smaller PRs are only practical if CI finishes in minutes. Fix the flaky suite second, since a faster pipeline that fails 12% of the time still trains people to retry.
Be explicit that this is feedback-loop economics: 41 branches and a 3.5-day merge time are symptoms of a pipeline people cannot afford to run.
2. How should flaky tests be handled so that retries do not mask real failures?
Read the worked answer
Treat flakiness as a defect with an owner and a clock, and make retries visible rather than silent.
Process:
- Detect flakiness empirically: run new or changed tests multiple times, or track pass-after-fail per test ID. Do not rely on developer reports alone.
- Quarantine confirmed flaky tests into a separate suite that does not gate merges. Quarantine is time-boxed: an owner must fix it or delete it within a set window (for example 10 working days). Deleting a test nobody will fix is a legitimate outcome; a permanently quarantined suite is just a slower pipeline.
- Require the quarantine list to shrink and report its size as a team metric.
- Never enable blanket auto-retry on the gating suite. If retries exist, record the retry count per build and alert when it rises; a retry should be an incident signal, not a normal path.
Root causes to attack: shared mutable test data, time and timezone dependence, test ordering, real network calls to external services, and fixed ports. Use hermetic tests with per-run isolated data, injected clocks, and contract stubs in place of live dependencies.
Why the current behavior is dangerous: clicking retry twice means the team cannot distinguish a genuine intermittent product bug from test infrastructure. In a payments-adjacent system, a real race condition can hide behind a retry habit for months. The goal is not zero flakiness overnight; it is that a red build means something again.
3. What branching and deployment model should replace the current one, and how do you reduce risk given Thursday-only manual deploys?
Read the worked answer
Move to trunk-based development with short-lived branches, continuous deployment to staging, and progressive delivery to production - decoupling deploy from release with feature flags.
Branching:
- Branches live under a day or two; merges to main are small and frequent, ending the two-day conflict scenario and the 41-branch queue.
- Main is always releasable, protected by a fast required pipeline.
- Incomplete work ships dark behind flags instead of living on a branch.
Deployment risk, replacing the Thursday window:
- Same image across environments. Build once, promote the artifact; configuration is injected per environment as code. This directly prevents the staging-only config incident.
- Progressive delivery: canary or blue/green with automated rollback on error-rate and latency thresholds. Small, frequent deploys with fast rollback are lower risk than a large weekly batch.
- Deploy any day, release when ready via flags. The fear of Friday deploys is really a fear of unrehearsed rollback.
- Game days that practice rollback give the team evidence that recovery works.
Secrets: remove the committed .env from the repository, rotate the exposed credentials immediately (assume compromise), move to a secrets manager with short-lived credentials injected at runtime, and add secret scanning to the pipeline. This is urgent and independent of the branching change - the committed passwords should be treated as an active incident, not a cleanup item.