nginx and PHP-FPM at 18,000 Connections

These are three separate limits, not one bug.

worker_connections are not enough is nginx's per-worker event slot limit. Each proxied HTTP connection consumes two slots (downstream client plus upstream). At 18,000 clients, a node needs roughly 36,000 slots. The default 512 per worker times 4 workers is far below that.

Too many open files is RLIMIT_NOFILE on the worker. The root shell's 65536 comes from PAM/login limits; it is irrelevant to the service. systemd gave the unit the default soft limit of 1024, and nginx inherited it. Once the worker reaches 1024 file descriptors it cannot accept a client or open an upstream socket. Setting worker_connections above the fd limit is meaningless.

connect() ... Resource temporarily unavailable (EAGAIN) to the PHP-FPM unix socket means the listen backlog is full. With pm=static and 120 children, throughput is roughly 120 divided by per-request service time. If each request takes 50 ms, that is about 2,400 req/s, which covers 1,000 req/s - until a database lock or storage stall makes requests slow. Then the 511-entry backlog fills and nginx gets EAGAIN instead of a queued connection.

Doubling pm.max_children to 240 treats the third symptom by adding workers that all contend for the same database and storage. If each child is 60-80 MB RSS, 240 children is 14-19 GB on a 16 GB node, which invites the OOM killer and makes 502s worse. It also does nothing for the nginx fd ceiling or connection slots.

Work from measured budgets.

1. Measure child memory: ps -o rss= -C php-fpm8.2 | awk '{s+=$1;n++} END {print s/n/1024, n}'. Check socket counts with ss -s and cat /proc/net/sockstat.

2. nginx connection and fd budget. For 18k clients plus 18k upstreams plus headroom:

worker_processes auto;
worker_rlimit_nofile 65536;
events { worker_connections 16384; multi_accept on; }

With 8 workers that is 131,072 total slots. Per worker, 2 x 16,384 plus overhead fits within 65,536 fds.

1. systemd must grant the limit at process creation:

# /etc/systemd/system/nginx.service.d/limits.conf
[Service]
LimitNOFILE=131072

Then systemctl daemon-reload && systemctl restart nginx. A reload is not enough; rlimits are inherited when the process is created.

1. FPM backlog: set listen.backlog = 65535 and verify sysctl net.core.somaxconn is at least that. Keep pm.max_children at 120 if RSS times 120 leaves memory headroom; otherwise add nodes instead of oversubscribing RAM. If the DB can only handle 120 concurrent connections, 240 children just moves the queue.

2. Kernel: net.core.somaxconn=65535, net.ipv4.tcp_max_syn_backlog=65535; if the L4 balancer does NAT, check conntrack as well. fs.file-max is not the constraint.

3. Validate with nginx -t and php-fpm8.2 -t, then restart during a canary. Confirm with grep 'open files' /proc/$(pgrep -o nginx)/limits and nginx -T | grep -E 'worker_connections|worker_rlimit_nofile'.

Canary one node, not the fleet.

1. Drain one node from the L4 pool (stop new sessions, let existing ones finish), then apply the systemd drop-in and nginx/FPM configuration and restart.
2. Confirm the limits are actually in force: grep 'open files' /proc/$(pgrep -o nginx)/limits, the same for a worker PID, and nginx -T for worker_connections and worker_rlimit_nofile.
3. Generate load above production on the canary: hey -c 20000 -z 3m https://canary/health, or replay production traffic at 120% for 10 minutes.
4. Watch ss -s, nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPAbortOnMemory, nginx stub_status active/reading/writing, and the FPM status page's listen queue and max listen queue. Check error-log deltas for EMFILE and EAGAIN, and dmesg for OOM or conntrack messages.
5. Let it run 30 minutes clean, return it to the pool, then roll the remaining nodes in waves with one canary always out of rotation. Keep the old unit file and config in version control so rollback is a single revert plus restart.