Diagnosing Slow Gallery Response Under Load
A Gallery host (8 vCPU, 32 GB RAM) serving 300 users becomes sluggish every weekday between 09:00 and 11:00. Page loads take 8-12 seconds, and users report timeouts when opening large workflow results. The Controller and Workers are separate hosts and are not slow. MongoDB is on the same host as the Gallery. During the slow window, CPU on the Gallery host sits at 85-95%, RAM at 30 GB, and disk I/O is high. The team increased the Worker count last month, which did not help Gallery responsiveness.
1. Why did adding Workers not fix the Gallery slowness, and where is the bottleneck?
Read the worked answer
Workers execute workflows; they do not serve Gallery pages. The slowness is on the Gallery host itself, which is also running MongoDB. High CPU, near-saturated RAM, and high disk I/O on that host point to resource contention between the Gallery web/application tier and the database tier. Adding Workers increased execution capacity but also increased load on the Gallery (more results to store, more status polling), which can make the Gallery host worse, not better. The bottleneck is the co-located Gallery + MongoDB host, not execution capacity. The fix is to address the Gallery host: separate MongoDB onto its own host with adequate RAM and fast disk, and/or scale the Gallery host vertically, rather than adding execution nodes.
2. What specific checks would confirm the diagnosis, and what remediation would you sequence first?
Read the worked answer
Checks: (1) per-process CPU and memory on the Gallery host during the slow window to see whether MongoDB or the Gallery web process dominates; (2) MongoDB slow-query log and index usage, since result storage and retrieval can be disk-heavy; (3) disk queue length and IOPS to confirm I/O saturation; (4) Gallery application logs for slow requests and timeouts; (5) whether the 09:00-11:00 window correlates with a scheduled report or result-cleanup job. Remediation sequence: first move MongoDB to a dedicated host (or at least give it dedicated fast storage and adequate RAM) so the Gallery tier is not competing with the database; second, review and fix MongoDB indexes and result retention/cleanup so the database is not bloated; third, if the Gallery tier is still saturated, scale it vertically or add a load-balanced Gallery tier if the deployment supports it. Re-measure page load times after each step rather than changing everything at once.