Group Membership That Never Reaches the Process
An internal team ships reporting.service on Ubuntu 22.04 with User=svc-report, Group=svc-report, Restart=on-failure, and ExecStart=/opt/reporting/bin/run --out /srv/data. The data directory is drwxrws--- root data-readers. The admin runs usermod -aG data-readers svc-report, confirms the change with id svc-report, and restarts the unit; the service still dies with Permission denied: /srv/data/q3.csv. A colleague then edits the unit to add SupplementaryGroups=svc-report, runs systemctl daemon-reload, and now systemctl restart reporting fails immediately with start request repeated too quickly. The unit has been in failed state for an hour and the 06:00 report did not run.
1. The admin argues the fix must be wrong because 'systemd caches the group database'. Is that true, and what is actually happening?
Read the worked answer
systemd does not cache group membership. When it spawns a service with User=, it resolves supplementary groups from NSS at that moment, so a usermod -aG takes effect on the next start without a daemon-reload. The colleague's edit is what broke it:
SupplementaryGroups= replaces the group set that would otherwise be derived from the user database. Setting it to svc-report means the process now has only svc-report as a supplementary group, and data-readers is gone.
Diagnose in this order:
systemctl cat reporting # is SupplementaryGroups= set at all?
systemctl show -p SupplementaryGroups reporting
ps -o pid,supgrp -p $(pgrep -f 'reporting/bin/run')
namei -l /srv/data/q3.csv # then confirm on-disk perms and group owner
Note the process's real group set, not id svc-report - id describes the account, not the running process.
Fix: either delete the line, or set SupplementaryGroups=data-readers. If the user database is remote (SSSD), a stale NSS cache is the one legitimate caching concern - sss_cache -E before restarting.
Also check whether the unit sets ProtectSystem=strict without ReadWritePaths=/srv/data, which would produce the same EACCES for a different reason.
2. The unit will not restart at all now. Explain the message, how to clear it, and whether raising the limits is the right response.
Read the worked answer
start request repeated too quickly is the start-rate limiter, not an application error. The defaults are 5 start attempts within 10 seconds; Restart=on-failure spent those attempts during the crash loop, so systemd refuses further starts until the interval expires.
Clear it explicitly rather than waiting:
systemctl reset-failed reporting
journalctl -u reporting -n 50 --no-pager # confirm the real error first
Then start only after the permission problem is fixed, otherwise you re-enter the same state.
On raising the limits: not as a first response. Loosening the limiter on a crash loop just hides a broken service and fills the journal. The correct tuning is a backoff, not a bigger burst:
[Unit]
StartLimitIntervalSec=300
StartLimitBurst=5
[Service]
Restart=on-failure
RestartSec=30
Point out the version trap: since systemd 230 the directives belong in [Unit]. The old [Service] spellings StartLimitInterval=/StartLimitBurst= are deprecated and are ignored in current versions, so a 'fix' placed there looks correct and does nothing. Verify with systemctl show -p StartLimitIntervalUSec,StartLimitBurst reporting.
3. The team proposes `DynamicUser=yes` to sidestep the account management. Evaluate that proposal.
Read the worked answer
It solves the wrong problem and creates a new one.
DynamicUser=yes allocates an ephemeral UID/GID for the lifetime of the service. That UID is not a persistent member of data-readers, so static group-based permissions on /srv/data will not apply. You would have to grant access at every start (ExecStartPre=+setfacl -m ...) or rely on StateDirectory= ownership - both more fragile than a static system account for a shared data directory.
What DynamicUser is good for: a service that needs no persistence across restarts and owns only its own state directory. That is not this service - it writes to a directory owned by another group.
Better options, in order:
- Keep the static user and delete or correct
SupplementaryGroups=. - Make ownership explicit in the unit rather than implicit in the filesystem:
[Service]
User=svc-report
Group=data-readers
StateDirectory=data
StateDirectoryMode=2770
UMask=0007
- If several accounts must share the directory, use a POSIX ACL:
setfacl -m g:data-readers:rwx /srv/dataplus a default ACL, and verify withgetfacl.
In all cases, prove the fix by restarting and re-reading the process's actual group set, not by reading id output.