HR Portal Slowness and a Silent Cron Failure
At 08:10 on a Monday, the monitoring dashboard shows the internal HR portal (Java, Tomcat 9, CentOS 7) on srv-hr-01 is slow. Load average is 14 on a 4-vCPU host; %wa is 38%; the /var partition is 93% full. The Tomcat logs directory holds 41 GB of catalina.out files, the oldest from 14 months ago. On Friday a junior administrator ran chmod -R 750 /opt/hr to tighten security; since then the nightly export job, run by user hrexport via cron at 02:00, has produced no files. hrexport belongs to group hrdata, but /opt/hr/export is owned by hradmin:hradmin with mode 755. The HR manager also reports that two staff who moved departments last week can still open payroll PDFs on the shared drive, while a new hire cannot log in to the portal at all. The change freeze for month-end starts in three days.
1. Diagnose why the portal is slow and why the export job stopped. Give the ordered checks and commands you would run, and state what evidence would confirm or eliminate each cause.
Read the worked answer
Start with read-only evidence so the state is preserved.
1. Separate CPU pressure from I/O wait. Run uptime, vmstat 1 5, iostat -x 1 5 and pidstat -d 1. A load of 14 on 4 vCPU with %wa at 38% points to blocked I/O or a run queue, not a pure CPU shortage. Check df -h /var, df -i /var and du -sh /var/log/* | sort -h. Confirm whether the 41 GB catalina.out is still open with lsof +L1 and inspect /etc/logrotate.d/tomcat. If logrotate is missing or the file is held open, Tomcat can block on writes and the disk can approach full.
2. Prove the export failure. Run namei -l /opt/hr/export, getfacl /opt/hr /opt/hr/export, sudo -u hrexport ls -l /opt/hr/export, then execute the job manually as that user. Because chmod -R 750 /opt/hr removed the other execute bit, hrexport cannot traverse /opt/hr unless it is in group hradmin; even with traversal, mode 755 on the export directory gives it read/execute but not write. Check /var/log/cron and journalctl -u crond for the job entry and exit status. A manual run returning Permission denied confirms the cause.
3. Check the login and access reports separately. For the new hire, inspect passwd -S, chage -l, /var/log/secure, group membership and the portal role mapping. For the two transferred staff, check whether access comes from an old department group or a stale share ACL; use id, smbstatus and file ACLs rather than guessing. Record findings before changing anything, and raise an emergency change if the freeze applies.
2. Resolve the permission problems without weakening security: the export job, the two transferred staff, and the new hire. Describe the exact changes and how you would verify them.
Read the worked answer
Use least privilege rather than broad chmod 777 changes.
Export job. The job needs execute permission on every directory in the path and write permission on the output directory. Prefer an ACL: setfacl -m g:hrdata:r-x /opt/hr for traversal, and setfacl -m u:hrexport:rwx /opt/hr/export or change ownership of the export directory to hrexport:hrdata. Add a default ACL so newly created files remain accessible to the intended readers. If the design allows, create a dedicated service account and directory instead of reusing a human account. Verify with sudo -u hrexport test -x /opt/hr && sudo -u hrexport test -w /opt/hr/export, then run the script manually, confirm the output file, and wait for the next scheduled run. Record the ACL baseline so later recursive changes can be compared.
Transferred staff. Their continued access is probably membership of the old department group or an inherited share ACL. Remove them from the payroll-read group, add them to the new department group, and force a new logon session so group tokens are refreshed. Check id, smbstatus and effective permissions on a sample file. Do not delete the accounts, because mail and history may still be needed.
New hire. The account may be locked, expired, missing a group, or not mapped to a portal role. Check passwd -S, chage -l, /var/log/secure, directory-service logs and the application role assignment. Test with an actual logon and a portal transaction. Because the change freeze is close, raise a documented emergency change with a rollback step, and notify the HR manager before making the changes.
3. What monitoring and housekeeping would you add to catch these two classes of failure before users do? List metrics, thresholds, and owners.
Read the worked answer
The failures were both predictable and silent, so monitor leading indicators rather than only user complaints.
Disk and logs. Alert at 80% used and again at 90% for each mount, and monitor inode usage separately. Track log growth rate and the age of the newest log segment. Configure logrotate with maxsize, compression, a 14-day retention and a postrotate action that reopens Tomcat's file handle. A 41 GB single file is a design smell; it should have been rotated long before 93% fullness. Owner: OS/server team.
Job success. Wrap the cron job so it writes a heartbeat file and exits non-zero on failure, then alert if the heartbeat is older than 26 hours. Better still, check the freshness and row count of the exported file, because a job can exit 0 while producing nothing. Owner: application operations, with the HR data owner accepting the output.
Permission drift. Keep a baseline of getfacl output for /opt/hr and compare it daily. Alert on changes outside a change window. This catches recursive chmod accidents. Owner: security operations.
Service health. Monitor Tomcat response time, error rate, JVM heap and a synthetic login. A synthetic login would have caught the new hire's failure before the manager called. Set thresholds from a one-week baseline, route alerts to the on-call rota, and review alert noise weekly so the team does not ignore them. Document owners and escalation in the operations manual.