Rootless Podman UID Mapping and Low-Port Bind Failure
A developer on a RHEL 9 workstation runs rootless Podman as user builder (UID 1000). /etc/subuid and /etc/subgid contain builder:100000:65536. The application image declares USER 1000. The developer runs podman run --rm -v /home/builder/app/data:/data:Z -p 8080:8080 app:dev. The container starts but immediately logs PermissionError: [Errno 13] Permission denied: '/data/out.db'. On the host, /home/builder/app/data is owned by builder:builder with mode 0775, and files previously created inside the container appear as 100999:100999. SELinux is enforcing and container-selinux is installed. podman info reports cgroupVersion: 2 and cgroupManager: systemd. A second issue appears when the container tries to bind port 80: it fails with Permission denied, although the same image works on the developer's laptop with Docker Desktop.
1. Explain precisely why the write to /data fails, including the host UID that the container process uses.
Read the worked answer
Rootless Podman builds a user namespace from the user's subuid allocation. UID 0 in the container maps to host UID 1000; container UIDs 1-65536 map to host UIDs 100000-165535. Because the image declares USER 1000, the process runs as container UID 1000, which is host UID 100999 (100000 + 999).
The host directory is owned by 1000:1000 with mode 0775. Host UID 100999 is not the owner and not in group 1000, so it falls into the other class and gets read and execute but not write. Creating /data/out.db therefore returns EACCES. The :Z option only relabels the directory for SELinux, changing the context to container_file_t; it does not change owner or mode. The files appearing as 100999:100999 are the expected result of the same mapping and confirm the diagnosis.
Prove it before changing anything: ls -ln /home/builder/app/data shows numeric ownership without name lookup, and podman unshare cat /proc/self/uid_map prints the mapping in the form 0 1000 1 followed by 1 100000 65536. podman run --rm app:dev id shows the container-side UID.
2. Give two fixes for the volume permission problem and one fix for the port 80 problem, with trade-offs.
Read the worked answer
Volume fix A: use --userns=keep-id.
podman run --rm --userns=keep-id \
-v /home/builder/app/data:/data:Z -p 8080:8080 app:dev
This maps host UID 1000 to container UID 1000, so the process writes as the host owner. It fits this image because it already runs as UID 1000. The trade-off is that the container's root is no longer host root and other UIDs are remapped differently; images that expect to write as root need adjustment.
Volume fix B: let Podman chown the volume to the container UID.
podman run --rm -v /home/builder/app/data:/data:U -p 8080:8080 app:dev
:U recursively chowns the host directory to the container process UID (host 100999 here), or you can do it explicitly with podman unshare chown 1000:1000 /data. This works for any image user, but it changes host ownership, can be slow on large trees, and may lock the host user out of direct writes.
Port 80 fix: the process runs as UID 1000 without CAP_NET_BIND_SERVICE in the container network namespace. The clean fix is to keep the app on 8080 and publish 8080, placing a reverse proxy in front if port 80 is required. Alternatively add --sysctl net.ipv4.ip_unprivileged_port_start=80 for the container namespace. Changing the host sysctl affects every user, and --cap-add NET_BIND_SERVICE does not solve publishing a privileged host port.
3. Which commands would you use to prove the mapping and make the setup repeatable for other developers?
Read the worked answer
Prove the mapping with podman unshare cat /proc/self/uid_map, podman run --rm app:dev id, podman run --rm -v /home/builder/app/data:/data app:dev ls -ln /data, and ls -Zd /home/builder/app/data to separate SELinux context from ownership. podman info --format '{{.Host.IDMappings}}' gives the full UID and GID map.
Repeatability is mostly configuration management. Put --userns=keep-id or the :U mount option in a checked-in wrapper script or Makefile rather than relying on developers to remember it. Document that /etc/subuid and /etc/subgid must contain an allocation, and use usermod --add-subuids 100000-165535 builder plus usermod --add-subgids 100000-165535 builder when onboarding. If containers must survive logout, enable lingering with loginctl enable-linger builder; otherwise the systemd user manager and cgroup delegation disappear and rootless containers stop. For long-running services, generate a user unit with podman generate systemd --new --user and start it with systemctl --user enable --now. Finally, record the decision in the repository README so the next developer does not rediscover the 100999 mapping.