Ninety Services, Sixty-Two Values Files, One Stripe Key

Start by treating the Stripe key as already compromised

The findings are one problem with three faces: the same production key in 11 repositories, embedded in an image layer, with no audit trail. Anyone with repository or registry pull rights has had it, and you cannot know whether it was used. Day-one action is to revoke and reissue, then hunt the remaining copies. Do not start with tooling selection.

Target design

• One secret store (Vault, or a cloud secret manager backed by KMS) as the only source of truth. Nothing in git, nothing in Helm values, nothing baked into images.
• Workload identity per service per environment: the pod authenticates as itself via its Kubernetes service account, so there is no shared static credential and every read is attributable.
• Prefer dynamic, short-lived credentials (database leases, cloud STS tokens) over static keys; a credential that expires in an hour cannot be leaked for a year.
• Prefer file-mounted or sidecar-delivered secrets over process environment variables. Environment variables are read once at start, which is exactly why the last rotation forced a restart.
• One credential per service per environment. A key shared across 11 repositories makes rotation a coordinated outage and makes blast radius unbounded.

Rotation without dropping work

• Change the application to re-read the secret on a signal (file watch or SIGHUP) or to fetch on use with a cached client, and to finish in-flight jobs before exit - longer grace period plus job acknowledgement.
• Rotate on a 30-90 day schedule so it is routine. The three-week pain is a symptom of annual rotation: every rotation is a novel event nobody has rehearsed.
• Use overlapping validity where the provider supports two active keys, retiring the old one only after the new one is observed in use.
• Add secret scanning in CI and image scanning on push, and gate merges on findings, so the next occurrence is caught in a pull request.

Why per-repo values files fail

Sixty-two files across 380 repositories means configuration has no owner and no single view. Production's true state is an emergent property of which file was applied last. You cannot answer 'what is the database pool size in production?' without reading 90 repositories; there is no drift detection; and any change is a 90-repository pull request that will be applied partially. Secrecy aside, this is a correctness problem: configuration has become a copy rather than a reference, and copies diverge silently.

Layering

1. Safe defaults live in the image or chart, versioned with the code.
2. Shared, non-secret configuration lives in one versioned source of truth - a config repository or config service - keyed by service and environment.
3. Environment-specific overrides are small, explicit, schema-validated and reviewed.
4. Secrets come from the secret store at runtime and never appear in layers 1-3. Sealed secrets are acceptable as a transition but solve neither audit nor rotation.
5. Precedence is declared and enforced: chart defaults < shared config < environment override < runtime secret. Anything violating the schema fails at admission or startup, not at 03:00.

Operating it

• Config changes are versioned, reviewed and promoted through the same pipeline as artifacts, and are themselves deployable events with a rollback.
• Continuous drift detection compares declared and deployed state.
• Every pull request renders the final manifest diff, so reviewers see the actual change rather than a values file whose effect they must simulate mentally.

The migration path: freeze new values files immediately, move the shared non-secret keys first (they are low-risk and prove the mechanism), rotate every credential found in git as part of the move, and only then delete the old files.

The two hazards

A four-hour backfill over 90 million rows cannot live inside a deploy: it will hold locks, time out, and cannot be rolled back with the application. And rows written during the backfill must land in the new structure - in a logistics system those are real shipments, not test data.

Sequence

1. Expand. Create shipment_events with additive DDL that does not block writes. Deploy code that dual-writes: every event goes to both the old and new structure, ideally in the same transaction, or through a transactional outbox if the write paths differ. From this point the new structure receives all live traffic.
2. Backfill history. Run as a separate batch job with keyset pagination by primary key (not OFFSET), idempotent upserts, checkpointing and throttling. Because dual-write is already live, the backfill only covers data older than the dual-write start, so it cannot lose events and can be restarted freely. Verify with per-bucket row counts and checksums against the source.
3. Shadow-read. Read from the new structure behind a flag for a small traffic slice and compare results with the old read path in the logs. Mismatch rate, not the backfill's exit code, is what authorises cutover.
4. Switch reads. Ramp the flag 1% → 100% using the same canary mechanics the service already has. Dual-write continues.
5. Stop dual-write. Only after a full business cycle - including the weekly and monthly reconciliation jobs - with reads on the new path and zero discrepancies.
6. Contract. Drop the old column or table in a separate deploy, after a verified backup and a freeze window.

Reversibility

Steps 1-4 reverse by flipping the flag; step 5 reverses by re-enabling dual-write; step 6 is a one-way door and is deliberately scheduled last and alone.