Nginx behind SELinux and firewalld on a nonstandard port
On a RHEL 9 host with SELinux enforcing, nginx 1.20 reverse-proxies an internal API at 127.0.0.1:8080 and listens on 8443 with a certificate under /etc/pki/app/tls. curl -k https://127.0.0.1:8443/health returns 200, but remote clients time out. firewall-cmd --list-all shows the public zone permits only 443/tcp. ausearch -m avc -ts today shows denied { name_connect } for nginx to 8080. The private key is mode 0644 owned root:root; security requires it readable only by root and nginx, and the certificate must stay outside /etc/nginx. Firewall changes need a 15-minute approval; SELinux must remain enforcing. The load balancer health check expects HTTP 200 from /health within 3 seconds and has already marked all nodes down. The nginx config was copied from a lab and has not been changed since deployment. The team wants the smallest change that restores service and passes the next security scan.
1. Explain why local health checks succeed while remote clients time out, and give the firewall-first fix.
Read the worked answer
Local curl connects to 127.0.0.1, which does not traverse the public zone, so it proves nginx is listening and TLS is terminating. Remote clients fail earlier: their SYN reaches the host but firewalld drops 8443/tcp because the public zone only permits 443. That is why the symptom is a timeout rather than an nginx error page. Check ss -ltnp | grep 8443 and firewall-cmd --get-active-zones before changing anything.
After the change approval, run firewall-cmd --permanent --zone=public --add-port=8443/tcp and firewall-cmd --reload. Verify from an external host with curl -k https://<host>:8443/health and capture with tcpdump -ni any port 8443 if it still fails. Do not disable firewalld or move nginx back to 443 just to avoid the approval; the port is part of the deployed service contract.
2. How do you fix the SELinux denial and secure the private key without moving it under /etc/nginx?
Read the worked answer
The AVC is a connect denial, not a bind denial: nginx is allowed to listen, but SELinux blocks its outbound connection to 127.0.0.1:8080. Confirm with ausearch -m avc -ts recent | audit2why. The smallest policy change is setsebool -P httpd_can_network_connect on, which allows the nginx/httpd domain to make network connections. Avoid setenforce 0 and avoid writing a custom policy before checking whether the boolean is sufficient.
For the key, set chown root:nginx /etc/pki/app/tls/server.key and chmod 0640. Label the directory with semanage fcontext -a -t cert_t '/etc/pki/app/tls(/.*)?' and apply with restorecon -Rv /etc/pki/app/tls. cert_t is appropriate for certificate material and is readable by the web server domain; do not label the key as generic content or make it world-readable. Verify with ls -lZ and matchpathcon. Reload nginx only after nginx -t passes, then clear the AVC by testing a real upstream request.
3. What end-to-end verification and rollback checks would you perform before telling the load balancer team the node is ready?
Read the worked answer
Verify in layers. Locally: nginx -t, systemctl status nginx, ss -ltnp, and curl -k https://127.0.0.1:8443/health. Through the actual network path: from another host, curl -vk https://<node>:8443/health and openssl s_client -connect <node>:8443 -servername <name> to confirm the certificate chain and SNI. Check ausearch -m avc -ts recent is clean for the test request and that no new denial appears in the nginx error log. Confirm the upstream API responds directly on 127.0.0.1:8080 so a 502 is not misdiagnosed as firewall.
For the security scan, verify namei -l /etc/pki/app/tls/server.key, ls -lZ, and that the key is not world-readable. Rollback should be the inverse of the smallest changes: remove the 8443 firewalld rule if the node must be isolated, revert the boolean only if the upstream connection is no longer needed, and restore the previous file context if the certificate path changes. Do not roll back by disabling SELinux; that would hide the next failure and fail the security requirement.