I85PREP
⌘ K
CS AO study materials
CS AO study resources

Departed contractor, a shared deploy account, and a two-hour Sunday window

CS AO · Put your knowledge to work in context.

All case studies

Departed contractor, a shared deploy account, and a two-hour Sunday window

Acme's platform team runs 180 Ubuntu 22.04 virtual machines. Authentication is centralised in FreeIPA, but sudo rules are still local files in /etc/sudoers.d and engineers' SSH public keys live in each user's ~/.ssh/authorized_keys. During a quarterly access review you find: contractor jsmith, whose engagement ended 19 days ago, still has an enabled IPA account, is in the wheel group, and is covered by /etc/sudoers.d/90-contractors containing wheel ALL=(ALL) NOPASSWD: ALL; a shared account 'deploy' used by six engineers, whose password is stored in a Jenkins credential and whose passphrase-free private key sits on build01; and 'deploy' is also in the docker group on every application server. Production deploys run as 'deploy' every weekday at 15:00 and must not be interrupted. An SSO/OIDC migration is planned for next quarter. The next change window is Sunday 01:00 to 03:00.

1. Sequence the containment of jsmith for the first hour. What do you do first, what do you deliberately not do, and how do you handle credentials that are already issued?

Read the worked answer

Order matters because each step trades speed against losing evidence or leaving a live path.

  1. Capture before changing. ipa user-show jsmith --all, group memberships, ipa sudorule-find, and the contents of /etc/sudoers.d/90-contractors, all attached to the ticket. Preserve auth logs (journalctl -u sssd, /var/log/auth.log across hosts) before anything is deleted.
  2. Disable, do not delete. ipa user-disable jsmith and expire the password. Deleting removes the audit trail and the uid mapping; disabling stops authentication and is reversible if the review is wrong.
  3. Kill live sessions. loginctl list-sessions, then loginctl terminate-user jsmith, plus pkill -u jsmith for anything outside a systemd session, and check ss -tnp for established SSH connections that survived.
  4. Remove the keys. Strip jsmith from every ~/.ssh/authorized_keys and authorized_keys2 through configuration management across all 180 hosts and record the run id. Local keys are the real access path; the IPA disable alone does not close them.
  5. Remove privilege. Delete jsmith from wheel and from any sudoers.d entry, then clear the cached sudo timestamp (/run/sudo/ts/jsmith) and the sssd cache (sss_cache -u jsmith). The default timestamp_timeout is short but not zero.
  6. Kerberos. Disabling the principal stops new TGTs, but tickets already issued remain valid until expiry - check the realm's maximum ticket life and note the residual window in the ticket rather than claiming instant revocation.

Do not delete the account, do not rename the home directory, and do not run a broad password reset as a first move. Then hunt: cron and at jobs, systemd user units, keys on other systems, CI tokens, VPN, cloud IAM, and database accounts.

2. The 'deploy' account cannot survive the SSO migration unchanged. Propose a target design and an interim step that works before SSO lands, with trade-offs.

Read the worked answer

The core problem is attribution: six people share one credential, so no log can tell you who deployed, and the passphrase-free key on build01 is a secret that anyone with build access can copy. Removing the docker group membership from 'deploy' is urgent regardless of design, because docker group is effectively root on those hosts.

Target design. Humans authenticate as themselves (IPA now, OIDC later) and never log in as 'deploy'. The deploy identity becomes a non-login service account: shell set to /usr/sbin/nologin, no authorized_keys, no password. Deployments are executed by the CI runner, which obtains short-lived credentials from Vault or via OIDC workload identity, so nothing long-lived sits on build01. Where an engineer genuinely needs to run the deploy as the service account, a sudoers rule permits the named user to run exactly that command as deploy, with log_output enabled and iolog captured. The Jenkins credential is deleted after the runner is migrated.

Interim design (before SSO). Move to per-engineer SSH keys distributed by configuration management, with a sudoers drop-in that allows each named engineer only sudo -u deploy /opt/deploy/bin/release *. Rotate the build01 key now and scope it to the runner host only. This is more work than keeping the shared key, but it is the only step that creates an audit trail, and it is achievable in one Sunday window.

Trade-offs to state. Per-user keys multiply the revocation surface but make leavers a single delete. The sudo-to-service-account pattern is weaker than workload identity because it still lets a human act as the service identity - acceptable as a bridge, not as the destination. SSH certificates with 15-minute lifetimes are the natural middle step if IPA already runs a CA.

3. The auditor asks you to prove the remediation without breaking the 15:00 deploy. What evidence do you produce, and how do you roll it out safely?

Read the worked answer

Rollout. Apply to one canary host and one non-production group first, run the 15:00 deploy through the new path on the canary, and confirm the job completes and the logs attribute the action to a named engineer. Only then widen to the rest of the fleet. Never make an access change and a deploy-tooling change in the same window - if the 15:00 job fails you need to know which change did it. Keep the old sudoers file archived in configuration management, not live, so rollback is a revert of a commit.

Evidence pack.

  • Before and after ipa user-show jsmith and a group membership export showing wheel removal, with timestamps.
  • The configuration-management run id and a grep -r jsmith /home/*/.ssh /etc/sudoers.d sweep across all 180 hosts returning nothing.
  • Session termination records from loginctl and auth log entries showing no jsmith authentication after the disable time.
  • The new sudoers drop-ins, with visudo -c output proving syntax validity.
  • A sample of deploy logs showing a named user and the command, plus iolog references.
  • A full access-review export listing every account, its owner, last login from lastlog and ipa user-status, and disposition, signed by the application owner.

Ongoing control. Automate the review instead of doing it quarterly by hand: alert when any account has no login for 90 days, when any host has an authorized_keys entry not present in configuration management, and when anyone not in the deploy group successfully runs a release. The auditor's real question is whether the control will still hold in three months.