Shift-Change Meltdown at a 911 Center

There are at least three independent faults here, and treating them as one will produce a change that fixes nothing.

Fault 1 - portal JVM exhaustion. Java heap errors plus three to four minutes of unresponsiveness is classic GC thrash: the JVM spends all its time collecting and none serving. Concurrent logins and dashboard token traffic spike at shift change. Evidence: portal logs, a heap histogram (jmap/jstack) during a peak, and the configured heap size versus the machine's RAM. Fix direction: right-size the heap (the default is small relative to what a portal does), reduce cost per request by disabling unused features and clearing stale statistics, and consider a second portal node behind the load balancer.

Fault 2 - CPU saturation on one GIS machine. The 15-minute geoprocessing job at 7 minutes per run means the job overlaps itself and the shift change. By default a GP service shares the SOC pool with map services, so a heavy synchronous job starves rendering. That only one machine pins at 100% also indicates the requests are not being evenly distributed. Evidence: per-machine service instance counts, the GP service's instance settings, and the timing of the script relative to 07:00 and 15:00.

Fault 3 - storage and observability debt. DEBUG logging at 85% disk is a latent outage; a full log volume takes down the portal or server. And with no monitoring, the team cannot establish a baseline.

Triage order: stop the bleeding on disk first (it is the cheapest action and the most likely to cause a hard failure), then portal heap, then service instance settings, then monitoring.

Geoprocessing service:

• Set isolation to dedicated so the GP service gets its own SOC processes instead of competing with map services in the shared pool. This is the single highest-impact change: it stops the outage from being a site-wide event.
• Set max instances low (often 1–2 for a 7-minute job) to prevent the scheduled call from stacking copies of itself when a run overruns.
• Set a maximum execution time on the service so a runaway job cannot occupy a slot for hours.

The scheduled job:

• Move it off the shift-change windows, or reduce frequency; every 15 minutes around the clock is unnecessary for a 7-minute analysis. Consider precomputing results and serving them as a cached layer for the dashboard, which is the real fix: dashboards at shift change do not need live drive-time computation.

The site:

• Rebalance machine assignment. Watch whether one machine receives disproportionate requests; the load balancer, not the server, usually causes this.
• Review max instance counts and timeouts per service for map services too; an unbounded max instance count turns a traffic spike into a memory event.
• Determine whether the dashboard consumes cached tiles or dynamic layers. If dynamic, convert suitable layers to cached tiles; that alone removes most of the CPU load at 4,000 viewers.

Justification to give stakeholders: dedicated isolation plus a cap converts a site-wide outage into a degraded single service; moving the job and caching the dashboard addresses the load itself; both are reversible and require no new hardware.

Logging. Set server log level back to WARNING and cap log retention. A vendor who wants DEBUG should get it scoped: raise the level on a single service (or for a defined troubleshooting window), capture what is needed, and revert. Persisting DEBUG indefinitely is not observability - the volume destroys signal, fills disks, and degrades performance. Add central log collection with retention that is enforced automatically, and alert on disk usage above 75%.

Monitoring. Without a baseline, every incident is argued from anecdote. Deploy ArcGIS Monitor (or an equivalent) and alert on:

• per-service average and 95th-percentile response time, with thresholds set from a known-good baseline rather than guesses;
• service instance saturation (busy instances versus max instances) per service and per machine;
• error rates in server and portal logs, particularly 498 and timeout patterns;
• portal JVM heap usage and GC time;
• disk, CPU, and memory on every machine, plus data store health and backup status;
• an external uptime check that performs a real transaction (portal sign-in to service draw), not a port test.

Maintenance rhythm. Weekly: review dashboards, prune stale items, verify backups restorable. Monthly: patch cycle, review service instance settings against actual usage, audit log levels. Quarterly: failover drill, capacity review against known peaks such as the shift change, and a restore test of at least one data store backup.

Capacity note. A single portal machine serving 4,000 concurrent viewers at shift change is the structural risk. The heap tuning buys headroom; a second portal node behind the load balancer is the durable fix and should be in the next budget cycle.