TS-C (NCCT): An Intermittent Morning Slowdown That No Alert Sees

1. Connection-pool exhaustion with queueing (most likely). 6 × 20 = 120 pooled connections plus 15 reserved = 135 against a 150 limit. The extract adds 10–15 connections, which pushes the total to the ceiling; once a pool is full, requests wait before the handler runs. That wait is invisible to handler-scoped spans and consumes almost no CPU, which is exactly why the CPU metric stays flat and no alert fires. Evidence: pool wait time and queue depth per node, active sessions by application/user per minute, and whether timeouts cluster at the start of the window (queue) rather than spread through it.

2. Database contention from the extract (strong). Long read queries spilling to temp storage and holding locks or saturating I/O. Evidence: wait-event breakdown over the window, blocking-session chains, temp-file and buffer-cache spill counters, and whether p95 recovers the moment the extract ends.

3. Host I/O from the backup upload (weakest). The timing does not fit - the backup finishes at 09:05 and symptoms persist to 09:40 - and CPU/IOPS would move. It is a correlation trap created by the 5-minute averaging, which smears events across buckets.

Rank order: 1, 2, then 3. The discriminator is simple: if pool queue depth spikes at 09:10 while database wait events stay normal, it is 1. If waits and blocking rise together, it is 2.

Why it fails

• Five-minute averages turn a 20-minute event into roughly four data points and blur the start time, which is the single most diagnostic fact. The backup and the extract look equally guilty.
• Host CPU is an aggregation across all threads. A pool-blocked request is a parked thread: it consumes memory and a connection slot but no cycles, so the most likely cause is the one metric that will not move.
• Head-sampling at 1% plus handler-scoped spans means the sampling decision is made on arrival and the queue wait happens outside the span. You measure the time the request spent working, not the time it spent waiting.

Minimum additions, cheapest first

1. Pool wait time and queue depth as a gauge, scraped at 10 s resolution, emitted per node.
2. Database wait events attributed by application user, so batch and interactive load are separable.
3. A distinct database user and pool for the extract - this is both a fix and an instrument.
4. Client-side timing at the load balancer, so queueing before the app is measurable.
5. A slow-query log sample during 08:45–10:00 only, to keep volume down.

Re-baseline before changing anything, or you cannot prove the fix worked.

Against

• It raises the ceiling without touching the cause. Six nodes × 50 = 300 connections means more backend memory per connection, more context switching, and worse lock and I/O contention - the second-ranked hypothesis gets amplified by the fix for the first.
• It removes the signal. Once nothing queues, the symptom disappears while the batch job's real cost stays hidden, and the next incident arrives with no forensics.
• 300 connections against a database sized for 150 is a capacity decision smuggled in as an operational fix, with no change record and no testing.

First action

Separate the batch workload: give the extract its own database user and connection pool, and move its start time out of the interactive peak (or cap its concurrency). This is low-risk, reversible, shippable before the next morning window, and it simultaneously tests hypothesis 1 - if separating batch load restores p95, the diagnosis is confirmed, not assumed.

Sequence after that: measure for two weeks at 10 s resolution, then decide on connection limits with data. If pool sizing does need to change, change it deliberately, during a change window, with a before/after comparison.