GitHub Certified: Agentic AI Developer study materials
GitHub Certified: Agentic AI Developer study resources

Deploy rights, three incidents, and an unreviewed prompt change

GitHub Certified: Agentic AI Developer · Put your knowledge to work in context.

All case studies

Deploy rights, three incidents, and an unreviewed prompt change

An agent holds deploy rights to staging and production. Over a quarter it performs 640 deployments and is involved in three incidents. Incident 1: it rolled back production at 02:00 during an on-call handoff because a health check flapped twice. Incident 2: it deployed a commit whose passing CI run belonged to a different SHA. Incident 3: an engineer edited the agent's system prompt in a PR merged with a single approval and no platform-team review. Audit logs record each deployment but capture neither the prompt version nor the model version in use. Company policy requires named-human accountability for every production change.

1. Rank the three incidents by the risk they represent and identify the single control that mitigates the most risk across all three. Justify the ranking.

Read the worked answer

Ranking, by expected loss rather than by how alarming the story sounds:

  1. Incident 2 - wrong artifact deployed. Highest: it is silent, it recurs at 640 deployments/quarter, and it breaks the core guarantee that what shipped is what was tested. A CI result not bound to a SHA is a false assurance that will pass every check you have.
  2. Incident 3 - unreviewed prompt change. Second: the prompt is the agent's behavior, so this is a production change with no code review, no eval, and - per the audit gap - no record of which behavior was live. It is also a governance failure against the stated policy, which matters independently of whether it caused harm this quarter.
  3. Incident 1 - auto-rollback at handoff. Third: it caused visible, recoverable disruption, and rollback is generally the safe direction. The defect is narrow (flap sensitivity and timing), not systemic.

The single highest-leverage control: bind every deployment to an immutable artifact identity. Require the deploy to name an exact digest/SHA, require the CI attestation to be for that same digest (signed provenance), and have the deployer refuse if they differ. This eliminates Incident 2 outright, makes Incident 1 auditable (you can see exactly what was rolled back and from what), and creates the identity substrate that Incident 3 needs - a prompt bundle tagged with the same release identity.

On accountability. An agent cannot be the named accountable party. Every deployment record needs a human accountable owner attached to the artifact digest. "The agent decided" is not an accountability answer, and policy compliance will not accept it.

2. Design the governance model for the agent's configuration - prompt, model version, tool scopes - and specify what an auditor must be able to reconstruct from logs.

Read the worked answer

Treat agent configuration as a deployable artifact, because that is what it is. A prompt edit changes production behavior more directly than most code changes; the current single-approval merge path is the root of Incident 3.

Change control:

  • Prompts, tool definitions, and model pinning live in version control and change only via PR, with CODEOWNERS requiring platform-team review. A single approval from the author's teammate is not review.
  • Pin the model version explicitly. An unpinned model means the vendor can change your agent's behavior without a commit - a silent, unreviewed production change. Upgrades go through the same PR and eval gate.
  • Eval gate on config changes. Run the offline eval suite from the quality gate on any prompt/model/tool-scope diff; block merge on regression. Config changes are the highest-risk, lowest-friction changes in the system, so they need the most friction, not the least.
  • Tool scopes in code, not in a console. The permission grant is part of the reviewed artifact; a scope escalation is a diff an auditor can see.

Reconstructability - an auditor should be able to answer, for any deployment:

  1. Which artifact digest was deployed, and who approved it by name.
  2. Which prompt bundle version and model version were live at that moment - recorded in the deployment event, not inferred from git history later.
  3. Which tool scopes the agent held at that moment.
  4. What the agent observed and decided: the triggering event, the checks it evaluated, and the decision record (for merges/deploys, the specific review or approval IDs and SHA it acted on).
  5. Which human was accountable, bound to the digest.

Also fix Incident 1's specific gap: suppress autonomous rollback during on-call handoff windows and raise the flap threshold (two flaps in quick succession is not a stable signal), routing to a paging human instead. Rollback stays enabled otherwise - removing it would trade a recoverable annoyance for a longer outage.