Scoped Role Breaks the Nightly Pipeline

Two independent timing effects are superimposed, which is why it looks like flakiness.

1. Creator-derived permissions on new resources. When the pipeline creates a resource, the rights it has over that resource are frequently derived from the creating identity's role at creation time, not from an explicit policy on the object. Where the scoped role lacks an action on that derived set, creation still succeeds - the failure surfaces later, when a subsequent step acts on the new resource. That is exactly why the failures land on objects the run itself created, and never on pre-existing ones.
2. Token caching for 60 minutes. A token minted before the role change still carries the old, broad permissions until it expires. Any run that starts inside that window passes. So the same step succeeds or fails depending on whether it straddled a token refresh.

The 3% is not randomness. It is roughly (the subset of resource types whose creator-derived policy is missing an action) × (the fraction of runs using a post-refresh token). The role is deterministically under-scoped; the timing merely decides which nights reveal it.

Confirm it, don't assume it: pull the authorization failures from audit logs with resource type plus action, and diff against the scoped role's action list. If the missing actions cluster on child resources rather than the documented 14, the derivation theory is confirmed.

Enumerate from evidence, then grant narrowly.

1. Extract the exact missing actions from the three nights of audit data - resource type, action, and resource name. Do not add speculative permissions; that is how a scoped role quietly becomes a broad one.
2. Add grants scoped by resource type and by tag or resource-group boundary, so the pipeline can only act on objects it tagged as its own. The boundary, not the action list, is what keeps the role small over time.
3. Remove the dependence on creator-derived rights. Either set an explicit policy at creation time or transfer ownership to a stable identity, so the pipeline's authority does not depend on who happened to create the object.
4. Respect the token cache when testing. Force a refresh or wait a full 60-minute TTL before concluding the fix worked, otherwise you will read stale-token successes and fresh-token failures simultaneously and learn nothing. Test in a staging account with a short-lived token first.
5. Keep the broad role disabled but not deleted for 30 days as a documented break-glass path, with an alert on any use. Deleting it on day one trades an availability risk for a permission risk you have not yet finished measuring.

Make permissions a tested artifact rather than a document.

• Run the pipeline under the scoped role in staging on every merge. A role gap then surfaces in CI in minutes, instead of surfacing as 3% of production steps failing at 02:00 three nights running.
• Weekly effective-permission diff against the documented resource types. When the pipeline grows a new resource type, the diff flags it for review before it fails.
• Alert on authorization-error rate per pipeline, not per step. The signal that mattered was the 3% pattern; individual step failures were noise that got retried and forgotten.
• Expiry dates on every exception, including the break-glass role, so temporary grants cannot become the new baseline.

The general principle: an over-privileged identity hides missing design work. The scoped role did not create the failure - it revealed a pipeline that depended on implicit authority nobody had written down.