Dashboard Slowdown During a Flash Sale

Increasing heap treats a symptom. If heap rises to 95% and timeouts continue, the JVM may be retaining too many result sets or waiting on slow database calls, or garbage collection may be consuming CPU. A larger heap can delay garbage collection but does not reduce query concurrency, connection pool exhaustion, or inefficient SQL. The fact that the problem returned suggests that the workload, not the heap setting, is the driver.

Diagnose in layers. Capture JVM garbage collection logs, thread dumps during the slowdown, and a heap dump if memory pressure persists. Review dispatcher and query service logs around the timeouts, and use the audit database to identify the top reports, users, and packages by execution time and frequency. Check the data warehouse connection pool metrics and wait times. Compare dashboard query plans and data volumes before and during the sale. Determine whether the 40 large extracts overlap with interactive demand, and whether the dashboard cache is actually being hit. Only after identifying the dominant wait condition should you change heap, pool size, or query concurrency.

Separate interactive and batch workloads. Move the 40 large extracts out of the 7:00 AM window or run them after the flash-sale peak, and stagger the remaining schedules so they do not all start together. Use job dependencies where one extract feeds another, and limit concurrent batch report service activity so batch jobs do not consume all query service capacity. If extracts can run against a snapshot or read replica, do that rather than competing with dashboards on the primary warehouse.

For caching, confirm what is cached and whether the 15-minute expiration matches the business freshness requirement. A dashboard that tolerates 15-minute-old data should not hit the warehouse on every request; pre-warm important dashboard caches before the sale if the platform supports it. Avoid caching highly personalized or security-filtered results in a way that could cross users. Review data source connection pooling and query service settings so that cache misses do not exhaust the warehouse pool. Measure cache hit ratio and dashboard response time before and after changes. The goal is to reduce warehouse calls during peak, not merely to increase heap or pool size.

Use targeted, temporary logging. Raise dispatcher, query service, and JDBC logging to a more detailed level for a short diagnostic window, capture the slow period, then revert to INFO. Verbose logging itself adds overhead and can fill disks, so define the start and end time and the exact loggers to enable. Preserve the relevant log files and correlate them with audit records and database metrics.

For monitoring, track JVM heap and garbage collection, dispatcher queue depth, active requests, query service timeouts, database connection pool usage and wait time, and schedule completion. Set alerts on connection pool exhaustion and dashboard response-time thresholds so the team learns before users report 90-second waits. Build an audit report for the top long-running reports and users. After changes, compare the same metrics over an equivalent peak period. Keep audit logging enabled for accountability, but ensure the audit database is sized for the load and is not competing with the content store. The final monitoring set should be sustainable at normal log levels.