Decomposing a Locked-Down Order Monolith Before Peak Season
Northwind Supply runs a 1.4M-line order-management monolith against one PostgreSQL 14 cluster (6 TB) used by 11 product teams. Deploys happen twice weekly inside a 5-hour Saturday window; a failed deploy in March left the platform read-only for 90 minutes. The nightly catalog ETL holds locks on four core tables from 19:45 to 20:30, during which checkout p99 climbs from 700 ms to 4.2 s. The codebase carries 340 foreign keys that cross module boundaries, and last quarter one team's missing index took down the entire checkout path. Leadership has committed to microservices within two quarters. The November-December peak (4.3x normal traffic) permits only low-risk change, and Finance requires a seven-year auditable record of every order state transition, including which human or system made it. The stated end goal is that any single team can deploy independently.
1. The CTO asks for a plan that reaches independent deploys in two quarters. What are your first three moves, in what order, and why is that order defensible against the obvious alternative of starting with service extraction?
Read the worked answer
Move 1 (weeks 1-2): remove the shared failure and locking hazards first. An ETL that locks catalog tables during checkout peak is not an architecture problem, it is an operations problem with a cheap fix: move the ETL to a read replica fed by logical replication, or convert it to incremental CDC writes that never touch the hot tables. Pair it with index review in CI so a missing index cannot repeat. Rationale: these changes are reversible, need no org change, and remove two of the three symptoms stakeholders actually feel.
Move 2 (weeks 2-8): draw seams inside the process. Use the 340 foreign keys as evidence, not as structure. Cluster tables by the team that writes them - you will typically find 8-10 owner clusters, not 11. Publish an explicit module boundary: separate packages, dependency rules enforced by a build-time check (ArchUnit, dependency-cruiser), and events emitted through an in-process interface rather than direct calls. This is the step that makes later extraction a deployment activity instead of a redesign.
Move 3 (weeks 8-20): strangle one read path end to end. Choose the catalog read API: high call volume, low write coupling, and it is the module with the worst failure history. Put a facade in front, route a percentage of reads to the new service, and shadow-compare responses against the monolith before cutting over. Keep the monolith write path authoritative.
Why this order beats extraction-first: extracting services before data ownership is mapped produces a distributed monolith - the same coupling, now over the network, with worse failure modes. It also front-loads the riskiest work into the release freeze window. Moves 1 and 2 are complete and revertible before the freeze; move 3 can be paused mid-cutover at any traffic percentage, which is exactly the property you need going into a 4.3x peak.
2. Once order, billing, and fulfilment data are split across services, how do you satisfy Finance's seven-year audit requirement and cross-module reporting without distributed transactions?
Read the worked answer
Use a transactional outbox per service plus an append-only audit log as the system of record for history.
- Write path: each service, in the same local transaction that changes order state, inserts a row into its own
audit_eventtable containing event id, aggregate id, monotonically increasing per-aggregate sequence, actor (user id or service principal), correlation id, before/after hash, and timestamp. Because it is the same local transaction, there is no lost-audit window and no need for two-phase commit. The business row and the audit row either both commit or both roll back. - Propagation: a CDC connector or an outbox relay publishes these events to a bus. Consumers are idempotent on
(aggregate_id, sequence)so duplicate delivery is harmless - this matters because at-least-once is the only guarantee you can get without heavy machinery. - Reporting: land events in an append-only warehouse table (insert-only, no updates) and build Finance's read models there. Store the raw log in object storage with immutability/retention controls to satisfy the seven-year requirement cheaply; keep the queryable window shorter.
- Gaps that will bite you: split-brain ordering when two services write about the same order - resolve by making exactly one service the owner of each state machine and having others emit their own domain events, never re-emit someone else's state.
- Auditor-facing consistency: eventual consistency means a report may lag by seconds to minutes. Publish a closed-period marker per day so Finance reconciles only after the watermark passes, and run a nightly reconciliation that compares current service state against the replayed audit log and pages on divergence.
- Corrections: never update or delete audit rows; append a compensating event with the reason code.
If you instead use a shared audit table written by all services, you have recreated the shared database you were trying to escape - expect lock contention and an ownership vacuum within a quarter.
3. Twelve months later, three services each hold a partial copy of customer credit status and they disagree. What governance mechanism prevents this, and what do you do about the disagreement that already exists?
Read the worked answer
Prevention: the ownership question must have exactly one answer per entity, written down. Maintain a data ownership registry that names, for every business entity, the single service authorised to write it and the events it publishes. Any other service may hold a local read-model copy only if it is derived from those events and is rebuildable from scratch - that is the test: can you drop this table and reconstruct it? If not, it is a second source of truth and must be flagged.
Enforce with a schema-level check: read-model tables live in a separate schema, are written only by the projector, and carry a source_event_seq column. Add an architecture test that fails if a service issues a write against another service's schema.
Remediation of the existing divergence:
- Declare the owning service - usually the one that owns the credit decision, not the one that consumes it most.
- Freeze writes to the two copies and put a read-through facade in front so consumers see one answer while you work.
- Reconcile. Pull the owner's current values plus its event history, replay it to rebuild the copies, then diff against what the copies held. Classify each mismatched record: staleness (copy missed events), logic drift (copy applied its own rules), or genuine owner error.
- For genuinely wrong owner records, fix them forward in the owner and let the correction event propagate; never patch the copies.
- For drift, delete the duplicated business logic. Credit rules belong in one place; consumers should receive a decision (approved/declined/limit) rather than raw inputs they re-interpret.
Add a standing reconciliation job that samples and reports divergence rate between owner and rebuilds, with a target near zero. Divergence discovered by an auditor or a customer is a much more expensive discovery than divergence discovered by a nightly job.