The Two-Quarter Microservices Mandate
Cartwheel Retail runs order management on a 900,000-line Java monolith deployed to four application servers. Twenty-two engineers in six teams ship once a week through a four-hour Saturday window; the median change lead time is 19 days, and two releases were rolled back last quarter. The CTO has announced the system will be decomposed into microservices 'within two quarters' and has asked for a target architecture. The checkout path writes orders, order_lines, inventory_reservations, and payments inside a single database transaction; peak load is 40 checkouts per second, and the database is the acknowledged bottleneck during flash sales. Two teams own reasonably clean modules behind internal interfaces, but a shared common library of 60,000 lines is imported by everything, so touching it triggers a full regression cycle. No engineer on the team has operated a distributed system in production. No additional headcount has been approved, and flash-sale season starts in five months.
1. Evaluate the CTO's two-quarter microservices mandate and state what you would recommend instead, with the evidence that supports your position.
Read the worked answer
The mandate mistakes an organisational problem for a topology problem. The symptoms quoted are change lead time of 19 days, a four-hour Saturday deployment window, and rollbacks. None of those are caused by the code being deployed as one artefact; all three are caused by a pipeline, test, and ownership problem. A team that cannot deploy safely on a Tuesday will not deploy safely on a Tuesday after splitting into nine services - it will deploy nine artefacts badly, and the checkout path will additionally acquire network partitions and partial failures.
The constraint that actually binds is operational capability, not architecture. There is no headcount increase, no production distributed-systems experience, and the season starts in five months. Microservices presuppose independent deployability, automated provisioning, per-service observability, and clear team ownership of a bounded context. This team has none of the four yet. Conway's law also applies in reverse: six teams over a 900,000-line monolith with a 60,000-line shared library do not have six separable bounded contexts today, so the service boundaries would be drawn along the shared library, which is the worst possible seam.
Recommendation: modular monolith first, then strangler-fig extraction of low-risk contexts.
- Break the common library into three or four versioned internal modules with enforced boundaries (build-time module checks, no cyclic dependencies, an architecture fitness test in CI). This is the prerequisite for any later seam; without it every extraction drags the library along.
- Fix the pipeline. Target on-demand deployment to production with a pipeline under 30 minutes, blue-green or canary, automated rollback. Measure change lead time weekly. This is achievable in two quarters and it directly addresses the stated symptoms.
- Extract one or two low-risk contexts - catalogue, notifications - to build operational muscle: service templates, distributed tracing, on-call rotation, runbooks. Learn on the path that is not the revenue path.
- Keep checkout in one ACID transaction for now. It is the highest-value, hardest-to-get-right path and the worst place for a first distributed transaction.
What to tell the CTO: 'Two quarters of decomposition is achievable for the library split, the pipeline, and two extracted services. Decomposing checkout in two quarters with no new headcount and no distributed production experience would convert a scalability bottleneck into a correctness risk on the revenue path.' That is a credible counter-proposal with measurable outcomes rather than a refusal.
2. If orders and inventory do become separate services, what design problem appears in the checkout path and how would you handle it? Address the failure semantics, not just the pattern names.
Read the worked answer
The problem. Checkout currently relies on a single ACID transaction: reserve inventory, insert the order, insert payment intent - all or nothing, with the database providing atomicity and isolation. Split the tables across services and you have a distributed transaction. Two-phase commit across services is technically possible but couples availability: the coordinator's availability is the product of all participants' availability, and a hung participant holds locks. At 40 checkouts per second during a flash sale, a stalled inventory service becomes a stalled checkout.
What replaces it: a saga with explicit intermediate states and compensations, plus idempotency.
- Reserve, then confirm. Inventory holds a reservation with a TTL (say 10 minutes) rather than deleting stock. Order is created in PENDING, payment authorised, then the reservation is confirmed. On any failure, the compensating action is release-reservation, and the TTL is the backstop for a crashed orchestrator.
- Idempotency keys. Every command carries a client-generated key so retried authorisations do not double-charge. This is non-negotiable in payments.
- Transactional outbox plus change data capture. The order service writes its state change and the event in the same local transaction, and a relay publishes. This removes the classic dual-write bug where the database commits and the event publish fails. Consumers must be idempotent, because delivery is at-least-once; exactly-once delivery is not on offer.
- Business decision, not a technical one. The team must choose the oversell tolerance: strict no-oversell means holding stock for the TTL and losing sales on abandoned carts; loose means accepting rare oversells and compensating customers. That number belongs to the business, and the design follows from it.
- Observability for the saga. A correlation ID across all steps, a saga state table, and an alert on sagas stuck in PENDING. Without this, the failure mode is silent: customers charged with no reservation and no order.
Cheaper alternative if the driver is only database load: read replicas, connection pooling, and moving the read-heavy catalogue off the transaction path may relieve the flash-sale bottleneck without splitting the write path at all. Confirm which problem is being solved before paying the distributed-systems tax.
3. Replace 'microservices in two quarters' with a sequenced plan and success metrics you would commit to.
Read the worked answer
Sequencing principle: prerequisites first, then extraction one context at a time, ordered by risk (lowest first) and by whether the boundary is genuinely clean. Each phase has an exit criterion; a phase that fails its criterion does not proceed, which is what stops the programme from becoming a schedule-driven big-bang.
Phase 1 (weeks 1-8): make the monolith modular. Split the 60,000-line common library into versioned modules with enforced dependency rules and an architecture fitness test in CI. Exit criterion: the build fails on a new cyclic dependency; the three largest modules build and release independently within the monolith.
Phase 2 (weeks 4-16, overlapping): fix delivery. On-demand production deployment, pipeline under 30 minutes, canary plus automated rollback, distributed tracing across the monolith. Exit criterion: change lead time under 3 days median; deployment window eliminated; rollback exercised in a game day.
Phase 3 (weeks 12-26): extract two low-risk services (catalogue reads, notifications) using the strangler fig pattern, behind the existing interfaces. Build the service template, on-call runbook, and per-service SLOs here. Exit criterion: each service deployed independently by its owning team at least daily; one real incident handled by the owning team using the new observability.
Phase 4 (post-season): checkout decomposition only after Phase 3 proves the operational capability, with the saga design and an oversell decision signed off by the business.
Metrics to commit to, in place of a service count: median change lead time (19 days → under 3 days), deployment frequency (weekly → daily), change failure rate (2 rollbacks last quarter → under 15%), mean time to restore, and percentage of production changes deployed without a maintenance window. Anti-metric: number of services. Service count is an output of boundary quality, not a goal, and optimising for it produces a distributed monolith - all the coupling of the monolith with none of the transactional safety.
Governance: a monthly architecture review with the CTO where the team reports the metrics and the phase exit criteria, so the two-quarter headline is replaced by a credible, evidence-based trajectory rather than quietly ignored.