TS-C (NCCT): Sizing a Four-Node Expansion When the Averages Are Lying

Arithmetic

• Current fleet capacity at 100% CPU = 120 rps ÷ 0.58 ≈ 207 rps, i.e. ~41 rps per node at 100%.
• Design ceiling of 70% → ~29 rps per usable node.
• Organic growth: 120 × 1.25^(14/12) ≈ 156 rps.
• Plus the new reporting line: ~216 rps peak at month 14.
• Nine nodes × 29 = 261 rps ceiling → 45 rps (17%) headroom.
• N+1 (one node down): 8 × 29 = 232 rps → 16 rps (7%) headroom.

On paper: yes, barely, and only if you accept a 7% margin with one node down.

Assumptions to state explicitly

1. CPU scales linearly with rps and every request costs roughly the same - false here, because reporting queries are far heavier than ticket reads. Averaging the two workloads is the main weakness.
2. The 60 rps forecast is a peak, not an average, and lands in the same window as the existing peak.
3. The skew is resolved. At 78% the two hot nodes are already past the 70% standard today. No number of extra nodes fixes skew - new nodes inherit the same long-lived connections only if the balancer is rebalanced. Weighted balancing plus connection draining is cheaper than nodes and must ship first.
4. Month 14 is not the horizon; growth continues past it.

Verdict: conditional yes, and the condition is rebalancing, not hardware.

The at-risk requirement is 99.9% monthly availability under single-node failure - it is implied by an engineering standard rather than written as a business requirement, so nobody owns it during the capacity decision.

• 99.9% monthly = a 43m 12s downtime budget.
• With the current skew, one hot node at 78% failing moves its share to the remaining node already at 78%. Latency degrades for the whole fleet, not just for the failed node's users. A single 45-minute incident consumes the entire month's budget.
• At month 14 with N+1 the margin is 7%. Any forecast error above 7% silently converts a capacity decision into an availability breach.

Reframing

1. Convert the engineering standard into a business statement: "we tolerate no more than 43 minutes of degraded service per month, including one node failure at peak."
2. Show the approver the N+1 number, not the 9-node number - the 9-node figure is the one that looks comfortable and is the wrong one.
3. Offer the cheapest way to buy margin: rebalance (engineering hours, no capex), then a fifth node if the rebalanced load still exceeds 70%, and only then re-forecast.
4. Ask for a written acceptance if four nodes is a hard cap, naming who accepts the residual risk.

1. Workload cost separation. Reporting and ticketing are not interchangeable units of load. Without a measured CPU-seconds-per-request figure for each, the 60 rps forecast cannot be converted into node count at all.
2. Definition of peak. Is 120 rps sustained, or a ten-minute spike? A ten-minute spike and a two-hour plateau have different headroom implications.
3. Concurrency/timeout behaviour under saturation. At 70% CPU a queue forms; the question is whether requests queue or fail fast. This determines whether an overrun is a latency event or an error event.
4. Failure-domain layout. Nodes in one rack or one zone are not an N+1 design. The standard says "single node failure" but not "single rack failure."
5. Maintenance window accounting. Does planned maintenance count against the 43m 12s budget? This changes the design more than any forecast.
6. Forecast accuracy and review cadence. Who re-checks the 60 rps number, when, and what triggers another procurement cycle?

I would sign off on the node count only against items 1, 2, and 4. The rest can be documented as open risks, but the workload-cost ratio has to be measured before any number is defensible.