Paging Space Pressure on a 64 GB Database LPAR
A production LPAR running AIX 7.2 with 64 GB of memory has logged SIGDANGER twice in the past week. Two paging spaces exist: hd6 (16 GB) in rootvg and paging00 (16 GB) in datavg. During both incidents vmstat -w showed avm around 78 GB, fre between 200 and 400 MB, and po sustained in the low thousands, while lsps -a reported both paging spaces above 90 percent used. CPU utilization never exceeded 25 percent. The Java and Db2 teams say their heap and buffer pool settings have not changed in a year, and neither team can explain the growth. datavg sits on SAN LUNs and has 500 GB of free space across four disks; rootvg has 40 GB free. Change control requires every change in the window to be reversible without a reboot, and the LPAR cannot be restarted during business hours.
1. Before adding any paging space, how do you decide whether the reported demand is genuine or partly an accounting artifact, and which commands do you use?
Read the worked answer
The first job is to separate 'memory is short' from 'AIX is counting address space that is not backed by page space'. The avm counter in vmstat is total active virtual memory across processes, so a process that maps a large file, attaches a big shared-memory segment, or reserves address space it never touches inflates avm without creating real paging demand. The authoritative figure is the pg space line of svmon -G, which should agree with lsps -s; if paging space really is over 90 percent consumed, the demand is genuine and the SIGDANGER events are consistent.
Next, attribute it. svmon -P -t 5 ranks processes by inuse and pgspace; ps -eo pid,vsz,rss,comm | sort -k2 -nr | head -20 shows which process has a VSZ far larger than its RSS. A process whose VSZ grew while RSS stayed flat, and whose pgspace grew in step, points to a leak or an unannounced configuration change. vmstat -v (numperm, numclient) tells you how much of the pressure is reclaimable file cache rather than anonymous memory; file cache is not a reason to add paging space.
Only after this triage do you size anything. If svmon -G shows pg space mostly consumed by two or three processes, you are dealing with a real consumer that will also need an owner and a fix, not just more swap.
2. Assume the pressure is genuine and you must act inside the constraints. What exactly do you create, where, and why not simply enlarge hd6?
Read the worked answer
Record a baseline first (lsps -a, lsvg datavg, svmon -G), because the change must be reversible and comparable. Note the PP size from lsvg datavg (assume 256 MB) so the arithmetic is explicit: if the target is 96 GB total, you need 64 GB more, which is 256 PPs.
Do not enlarge hd6. hd6 is the dump device and cannot be deactivated while the system is running, so any mistake there is not reversible online; it also concentrates paging I/O on rootvg. Instead create additional paging logical volumes in datavg, several smaller ones rather than a single huge one, and pin each to a different hdisk so the I/O is spread and one LUN loss does not remove all new space. For example, mklv -y paging01 -t paging datavg 64 hdisk3, then add /dev/paging01 to /etc/swapspaces and activate it with swapon /dev/paging01. The mkps -s <PPs> -a -n datavg route does the same thing and the -a flag is what registers the space for the next boot; if you build LVs by hand you own the /etc/swapspaces edit.
Reversibility comes from the new LVs: after the leak is fixed, swapoff and rmps remove them without a reboot. Finally confirm with lsps -a that all spaces are active and compare vmstat -w po before and after.
3. What permanent monitoring would you put in place, and what actually happens to the system if paging space reaches 100 percent?
Read the worked answer
Monitoring should alert on the trend, not the event. A cron job or monitoring agent sampling lsps -s every few minutes with a warning at 70 percent and a critical at 85 percent gives you hours of warning before SIGDANGER. Pair it with a weekly svmon -P snapshot so growth can be attributed to a process, and keep vmstat -w history so page-out rates can be correlated with the usage curve.
The failure mode matters when you argue for the change. AIX has no per-process out-of-memory killer: when page space is exhausted, forks and allocations fail (ENOMEM), applications typically abort or hang in unpredictable places, and a system with no page space can become unresponsive enough that even a login is difficult. SIGDANGER is advisory; a process that does not handle it simply dies later. If the application team ever sets PSALLOC=early, the situation gets worse, because every forked process reserves its whole virtual size in page space at fork time even if it touches little of it. That is also why blindly matching paging space to physical memory is the wrong reflex: the number to size against is peak observed pg space consumption plus headroom, not RAM.