Environment Branches, a Prod Hotfix, and 30 Tenant Overlays

Structural diagnosis. The hotfix conflict is not bad luck; it is the designed behaviour of long-lived branches. Promotion by merge makes every environment a divergent line of history, so any change that must exist in two environments has to be merged in both directions. Back-merging is where the 47 conflicted files come from: the same file was edited independently in two branches, and the tool has to guess intent. Problem (b) is the same cause: a revert applied in one branch is a change the other branch never sees, and the gap grows monotonically. Problem (c) is worse - the bot writes to a protected branch, so the review gate that justifies the whole layout is bypassed, and the canary is validating commits no human approved.

Recommendation: single trunk with directory-per-environment. One branch holds envs/dev, envs/staging, envs/prod plus a shared base/. Promotion is a pull request that bumps the image digest in the next environment directory. This eliminates back-merges entirely (there is one line of history), makes the diff between environments a literal directory diff, and gives one review path. The usual objection - "a bad commit can touch all three environments at once" - is handled by CODEOWNERS per directory plus required reviews, so the prod directory needs the prod approver.

Alternatives:

• Repo-per-environment: strongest isolation and access control (prod repo can require a different team), and it is defensible for regulated separation of duties. Cost: three promotion pipelines, cross-repo automation, and a third place for configuration to drift; with 30 tenants it multiplies again.
• Branch-per-environment: keep it only if you must, and then ban back-merges by making promotion one-directional and hotfixes cherry-picked with a mandatory forward-port PR - but you will keep paying the divergence tax.
• Whatever the layout, move the image bump from a direct commit into a PR (or a Git-writing automation that opens PRs), otherwise (c) recurs.

Preferred path: a narrow, fast PR. Open a PR that changes only the prod overlay (ideally one line: the image digest). Because overlays and base are separate directories, the change cannot silently alter staging or dev. Give it an expedited path: one required approver plus an incident ticket reference in the description, with the usual two-approver rule waived by policy rather than by bypassing branch protection. Merge, let the agent reconcile, and the canary validates the digest the same way it validates any other release. Then open a second PR that carries the same digest change into base/staging so production does not run code the lower environments never saw - that forward-port is a small, reviewable diff precisely because there was no back-merge.

Why the digest matters. If the hotfix is expressed as an immutable digest rather than a mutable tag, the emergency change is auditable, reproducible and reversible with a revert commit. If it must be a tag that the registry can repoint, you have lost the property that a commit identifies exactly what ran.

If the cluster must be patched before a PR can merge (true break-glass): write the change to Git first or simultaneously - a PR that can be merged in minutes is usually achievable, and if it is not, that is the real incident finding. If someone does use kubectl, the agent with self-heal enabled will revert it at the next reconciliation, so the sequence must be: commit → let the agent apply, not patch → commit. Record the exception, alert on the drift if the patch is observed, and review the break-glass in the postmortem.

Explicitly reject: committing to a prod branch directly, disabling the agent, or suspending reconciliation "temporarily" with no expiry - that is how the nine-day divergences and three-month quota surprises start.

Choose parameter-shaped configuration, not per-tenant overlay copies. Thirty tenants needing different resources.limits and a handful of flags is data, not structure. Model one base plus a Helm chart (or Kustomize component) parameterized by a small values file per tenant: tenants/acme.yaml, tenants/globex.yaml. Each file is five to ten lines. Generating 30 full overlay trees means 30 copies of every shared object, which is exactly the drift problem you just removed - a base change becomes 30 edits.

Wiring it up. A single ApplicationSet using a git files generator over tenants/*.yaml, with a matrix generator if configuration is per-tenant × per-cluster, materializes one Application per tenant. Adding a tenant becomes a one-file PR; removing one is a file deletion. Keep the tenant file schema validated in CI so a typo fails before it reaches a cluster.

Scaling limits to plan for:

• Application count. 220 workloads × environments plus 30 tenant apps is manageable, but every Application costs reconciliation work; watch the repo-server and controller, and shard or shard-by-repo when render latency grows.
• Rendering cost. Helm rendering is more expensive than plain manifests; caching and avoiding per-app chart fetches matter at this size. Consider pre-rendering or Kustomize for the high-churn apps.
• Repo contention and blast radius. A single config repo means one bad commit can affect many tenants at once; mitigate with CODEOWNERS on tenants/, validation, and progressive rollout (canary a tenant group before the rest). If tenants ever need different access control (a tenant team that can edit only their own file), split to a repo per tenant group rather than trying to express tenancy in RBAC over one repo.
• Cross-tenant isolation at runtime. Directory separation is a Git-side pattern; enforce namespace, quota and network-policy isolation in the cluster so a per-tenant config error cannot affect neighbours.