CDOS study materials
CDOS study resources

The 6.2 GB Monorepo and a Leaked Production Token

CDOS · Put your knowledge to work in context.

All case studies

The 6.2 GB Monorepo and a Leaked Production Token

Acme Logistics has 85 engineers working in one Git monorepo of 6.2 GB and 140,000 commits. A full clone takes 9 minutes per CI job, and the team runs 1,800 jobs per day. Peak queue time is 25 minutes. The platform team proposes splitting into 12 repositories, but three product teams share protobuf definitions and frequently make atomic cross-service changes. CODEOWNERS covers only 60% of paths. Separately, a production API token was found in deploy/legacy/credentials.env in a commit from 14 months ago; the token is still valid and appears in at least two forks. The company is under a six-week audit freeze that blocks major migrations, and monthly release branches are cut with hotfixes cherry-picked back to main. Developers report avoiding large refactors because CI feedback is slow.

1. Should Acme split the monorepo? Recommend a decision and a phased plan, including criteria for which components move first.

Read the worked answer

Do not do a big-bang split into 12 repositories. The clone cost is real: 1,800 jobs per day times 9 minutes is about 270 job-hours per day, plus queueing, but a 12-way split creates cross-repo versioning, duplicated CI, and breaks atomic protobuf changes. The audit freeze also makes a risky migration poorly timed. Instead, reduce clone cost first with blobless partial clone, sparse checkout, and a shared dependency cache; measure clone and queue time. Then extract only components that meet all criteria: independent release cadence, low change coupling to the rest of the monorepo, clear ownership with CODEOWNERS at 100%, and a stable API contract. Start with two low-risk services, not the shared protobuf library. Keep the protobuf definitions in a versioned package published from the monorepo. Phase the work: two weeks for clone optimization, four weeks for the first extraction behind the audit freeze, then reassess. Success criteria are P95 clone under two minutes and no increase in cross-team coordination cost. If those are not met, stop.

2. How should Acme remove the leaked token from the repository and prevent recurrence, given 85 engineers and at least two forks?

Read the worked answer

Rotate first, purge second. The token is valid and in forks, so assume compromise: revoke it, issue a new one, and search provider audit logs for use from unexpected IPs or times. Do not start with a history rewrite, because it changes every commit SHA and will disrupt 85 engineers and CI. After rotation, freeze pushes, inventory all clones and forks, and use git filter-repo or BFG to remove the file from all refs and tags. Force-push, expire reflogs, invalidate CI caches and build artifacts, and require every developer to re-clone. Forks you do not control must be deleted or archived; otherwise the secret remains. Add preventive controls: pre-commit secret scanning, provider-side push protection, and short-lived credentials. Because the audit freeze blocks major migrations, schedule the rewrite as a coordinated four-hour window with a rollback mirror. Verify with a full-history scan and a test clone before announcing completion.

3. Without splitting the repository, how would you improve branching and CI clone performance?

Read the worked answer

Keep trunk-based development with short-lived branches and use the monthly release branches only for stabilization. Cut hotfixes from the release tag, then cherry-pick or merge back to main so the fix is not lost. For CI, stop cloning full history for every job. Use a blobless partial clone and sparse checkout limited to the affected service, cache dependencies outside the workspace, and use a remote build cache. Add test impact analysis so a change to one service does not rebuild everything. Shard the remaining jobs and run independent checks in parallel. Measure P50 and P95 clone time, queue time, and PR lead time before and after. Set a target such as clone under two minutes and queue under five minutes. This addresses the developer pain without the coordination cost of 12 repositories, and it preserves atomic protobuf changes.