The Queue That Outgrew the Engines

The queue with idle engines points away from CPU and toward I/O and orchestration. Four engines at two concurrent scans should run eight scans; if engines show idle gaps while 220 scans wait, check what the engines are waiting on: SMB writes to a 92%-full results share, SQL Server I/O during a backup that runs until 09:30, or Manager dispatch and database contention (the portal timeouts support this). The switch from Default to All increases the number of queries and the volume of results written per scan, which multiplies share and database load; that fits the jump from 18 to 47 minutes.

Before buying engines, collect scan phase timings from engine logs, per-engine CPU and disk latency, SMB latency and free space, SQL wait statistics and backup duration, and queue wait time per project. Check whether All is even required: many queries in it are low-value for these applications. Adding engines would add concurrent writers to the same saturated share and database, likely making the queue worse rather than better. The decision should be based on measured headroom, not queue length.

Sequence:

1. Restore headroom on the results share: archive or purge scans older than the retention policy and expand the volume. At 92% full, write latency and failed scans are expected.
2. Move the backup window so it does not overlap the 02:00-09:30 scan window, or use a storage snapshot.
3. Replace All with a tuned preset: Checkmarx Default plus specific queries the business needs, and use query tuning per project to disable slow, noisy queries with documented approval. Re-measure on ten representative projects.
4. Stagger the 900 nightly scans into waves by team or criticality instead of one 02:00 start, and consider reducing per-engine concurrency to one until share latency is healthy.
5. Add engines only after CPU utilization and database wait times show headroom.

Expected impact: share expansion and backup separation remove the I/O cliff; preset tuning cuts scan duration and result volume; staggering smooths the queue. This ordering is cheaper than four engines and addresses the actual bottleneck first.

Use a controlled benchmark. Pick ten projects representing sizes and languages, freeze the same commit, and scan before and after each change; record duration, queue wait, result counts by severity, and failures. Track p50 and p95, not averages, because the queue is tail-heavy. Monitor engine CPU, disk latency, SMB throughput, SQL waits, share free space, and portal response time for two weeks.

Confirm coverage did not silently drop: run the old and new presets on the benchmark set and diff findings, and keep a known-vulnerable sample project that must still produce its expected results. Compare query-level timing to identify which queries in All were expensive relative to their value.

Define success before starting, for example: p95 queue wait under 30 minutes, scan duration back under 25 minutes, no increase in failed scans, and no loss of High or Critical findings on the benchmark. If those hold, the engine purchase can be deferred or sized from measured CPU headroom.