I85PREP
⌘ K
Vault Associate study materials
Vault Associate study resources

A six-hour batch job on a one-hour token

Vault Associate · Put your knowledge to work in context.

All case studies

A six-hour batch job on a one-hour token

A data-platform team runs a Python worker that drains a nightly backlog. Median run is 35 minutes, but month-end runs reach 6 hours 20 minutes. The worker authenticates with AppRole and then reads static KV credentials. The role is configured with token_ttl=1h, token_max_ttl=1h, secret_id_ttl=24h, secret_id_num_uses=0. Two incidents: at month-end a run died one hour in with permission denied, and its retry loop kept re-reading the same dead token and burned 40 minutes before alerting; separately, an on-call engineer needed to cut off one runaway worker at 02:00 and could not - she had no copy of that worker's token, and the only lever she had was disabling the whole AppRole mount, which would have stopped every other job. The security team also requires that any single job's credential be revocable at any moment.

1. Choose between raising `token_max_ttl`, switching to batch tokens, and switching to periodic tokens. Defend the choice against the revocation requirement.

Read the worked answer

Batch tokens are disqualified by the requirement, not by the duration. They are lightweight and cheap precisely because they are not persisted: no accessor, not individually revocable, not renewable. They would fix the six-hour run and break the 02:00 story completely. Rule them out first.

Periodic tokens allow indefinite renewal as long as the worker renews inside each period, which is attractive if runs can legitimately exceed any ceiling. But setting a period is a privileged act the workload cannot grant itself, and an indefinitely renewable credential is the opposite of what a batch job needs.

Recommended: a plain service token with a ceiling.

vault write auth/approle/role/batch-worker \
  token_ttl=30m token_max_ttl=8h token_policies=batch-worker
  • 8h covers the 6h20 p99 with real headroom, and the token dies on its own if the worker is killed and never cleans up.
  • 30m TTL with renewal means a leaked token has a short life even while the job runs.
  • It is a persisted service token, so it has an accessor and is instantly revocable - which is what the 02:00 requirement actually needs.

Revisit periodicity only if the business says runs may exceed 8 hours; that is a policy question, not a technical one.

2. The 02:00 problem: how do you make any single worker revocable without ever storing its token?

Read the worked answer

Use the token accessor. The accessor is a pointer that can look up and revoke a token but cannot authenticate as it - so it is safe to write into the job's run record, structured logs, or a job registry.

At login, the auth response includes the accessor; have the worker log it with the run ID. Then:

vault token lookup -accessor <accessor>     # confirm it's the runaway job
vault token revoke -accessor <accessor>     # cut it off

This is the reason the batch-token shortcut fails: with no accessor, the only remaining levers are revoking every secret_id for the role or mounting-down the auth method - both collateral damage across all jobs.

Two supporting details:

  • Revoking the token also revokes its child tokens and every lease it created, so downstream database credentials die with it.
  • Keep the AppRole role per job family rather than one global role, so -prefix style bulk revocation is a scoped last resort instead of a fleet-wide outage.

Do not log the token itself as a shortcut. Anyone reading logs would then hold a live credential, and the accessor gives you the same operational power for none of that risk.

3. Why did the retry loop make the month-end incident worse, and what should the client do instead?

Read the worked answer

With token_ttl=1h and token_max_ttl=1h the token is at its ceiling from the moment it is issued: it cannot be renewed even once. The worker's retry loop re-issued the same doomed request against an expired token, so every retry was guaranteed to fail, and the loop hid the cause behind retry noise.

The client contract should be:

  1. Read the TTL at login and renew proactively at roughly two-thirds of it via renew-self (allowed by the default policy, which every token carries).
  2. Treat a renewal failure distinctly from a request failure. max_ttl exhaustion is permanent - the correct response is to re-authenticate with AppRole and get a fresh token, not to retry.
  3. Distinguish 403 permission-denied (do not retry; re-authenticate once, then fail loudly) from 5xx or standby redirection (retry with backoff).
  4. Bound the retries and surface the expiry in the job's exit reason, so month-end failures say "token expired" instead of "40 minutes of 403".

The configuration fix makes max_ttl comfortable; the client fix makes the next surprise diagnosable in one minute instead of forty.