Kubernetes Engine Restarts, Exhausted DB Pool, and Disk at 91 Percent
After upgrading to XSOAR 8 on Kubernetes, one engine pod restarts every two to four hours. Symptoms cluster together: war room entries show "Failed to execute" for reputation commands such as !ip, several tasks sit in progress for more than 30 minutes, and during peak hours the application server logs "database connection pool exhausted." The node disk sits at 91 percent, dominated by container image layers and by demisto.log plus the accumulated war room artifacts. CPU on the affected node is around 40 percent; application server memory is at 95 percent. The splunk-prod integration instance reports repeated connection timeouts in the integration health view, but the Splunk team confirms the API is healthy and other tools connect fine.
1. You have four symptoms and limited time. State your triage order and what evidence you collect before changing anything.
Read the worked answer
Order by blast radius and reversibility, and by which failure is upstream of the others.
First, disk at 91 percent. This is the most likely upstream cause: disk pressure causes kubelet eviction, pods get killed and restarted, and Mongo writes can fail. A pod restarting every two to four hours is consistent with eviction under disk pressure, and CPU at 40 percent argues against CPU throttling as the trigger. Collect: pod events showing Evicted or disk pressure, kubelet node conditions, and a breakdown of what is consuming space - image layers, logs, or persistent volume.
Second, database connection pool exhaustion. Collect Mongo connection counts, slow query logs, and whether any custom field or playbook is running unindexed queries. Application server memory at 95 percent suggests GC pressure that can also manifest as apparent DB timeouts.
Third, the engine restarts. Collect the container's previous-state logs and termination reason: OOMKilled, evicted, or failed liveness probe. These map to different fixes.
Fourth, the Splunk timeouts. Collect which engine hosts the instance. If the instance is pinned to the restarting engine, the timeouts are a symptom, not a separate incident. Do not open a ticket with the Splunk team until that is ruled out.
Record a baseline before any change so you can prove which symptom moved first.
2. What actually causes the "database connection pool exhausted" messages, and what are the candidate fixes?
Read the worked answer
The pool is shared, so exhaustion means connections are being held too long or demanded too aggressively. Three plausible drivers here, and you should distinguish them rather than tuning blindly.
Load: each engine worker can open connections while executing tasks. Four workers is modest, but if other engines were recently scaled up or a playbook fans out parallel tasks, the aggregate connection demand rises. Check total connections per engine and total across the deployment.
Duration: slow queries hold connections. The classic culprits are searches over custom fields that lack an index, unindexed regex filters in SearchIncidents, and recurring playbooks that query all open incidents on a timer. If a query that used to take 200 ms now takes several seconds, every concurrent task holds a connection longer and the pool drains. Look at Mongo slow-query output and compare against a pre-upgrade baseline.
Resource pressure: application server memory at 95 percent means the process spends time in GC; during a pause, connections stay checked out and requests queue, which surfaces to users as pool exhaustion even if the pool size is adequate.
Fixes in order of preference: eliminate the slow or repeated query, add the missing index, reduce parallel fan-out or worker count, raise memory limits on the app server, and only then increase pool size - enlarging the pool against a slow-query problem just moves the bottleneck to Mongo. Confirm the fix by watching connection count and query latency for a full peak cycle.
3. Write the remediation and prevention plan for the disk and log growth, and explain how you confirm the reboot loop has actually stopped.
Read the worked answer
Remediation, in sequence. Free space first so eviction stops: expand the persistent volume if it is small, delete unused container images, and rotate or truncate the oversized logs after copying the relevant portion for the investigation. Do not delete war room artifacts blindly - capture the incident IDs and window first, because you may need them as evidence for the DB and timeout issues. Set a retention policy on war room entries and artifacts, and confirm log rotation is actually configured for demisto.log; a log that grows unbounded is usually a missing rotation policy or debug-level logging left on in production.
Then reduce what is generated. Turn off verbose playbook logging for high-volume playbooks, prune debug output, and check for a playbook stuck in a loop writing war room entries - 30-minute in-progress tasks often correlate with a tight retry loop. Cap artifact size for integrations that store full API responses.
Confirmation requires more than "it has not restarted yet." Watch for at least three times the previous restart interval - so a full 12 hours - and verify three signals together: pod restart count is stable, no Evicted or OOMKilled events appear, and disk usage trends flat or downward rather than climbing back toward 91 percent. Also confirm the tasks that were stuck in progress have drained and that !ip succeeds from the engine you suspected.
Prevention: alert at 80 percent disk and on restart count above zero in a rolling window, and treat log growth rate as a monitored metric, not something discovered when a pod dies.