Automating Toil Without Automating a Bad Runbook
Helio SaaS runs 9,000 incidents a year across 60 services. Thirty percent are disk-pressure alerts on Kubernetes nodes, 25% are service-restart requests, and 15% are certificate-expiry warnings. Engineers keep 12 runbooks in Confluence; median execution time is 22 minutes because the responder must find the runbook, connect to the VPN, request just-in-time access, and then run commands manually. The change advisory board meets monthly. Security forbids storing standing admin credentials in scripts. The company already has PagerDuty Automation Actions and an approved job runner, but only three runbooks are automated. The VP wants to cut toil by 50% this quarter and has asked which incidents to automate first, how to keep security satisfied, and how to avoid automating a bad process.
1. Rank the automation candidates and justify the order.
Read the worked answer
Score candidates on frequency, time cost, determinism, risk, and reversibility. Certificate-expiry warnings are frequent and highly deterministic, but the right action is scheduled renewal, not an incident-time script; automate the renewal job and keep an alert for failures. Service restarts are frequent and time-consuming, but only safe if a health check and blast-radius limit exist; start with a single-pod restart and require approval. Disk-pressure alerts are lower priority because the safe action depends on the cause: log rotation, image cleanup, or node replacement. A blind disk-deletion script would automate a bad process. Start with read-only diagnostics that collect disk usage, pod events, and certificate status, because they are low risk and reduce the 22-minute context-gathering phase. Then automate one reversible remediation with a human approval step. The sequence is diagnostics, low-risk scheduled jobs, then approved restarts. This cuts toil without pretending that every runbook is safe to encode.
2. What guardrails would satisfy security and operations?
Read the worked answer
Use short-lived, scoped credentials issued by the job runner rather than stored admin passwords. Require human approval for any write action, with the approval and result recorded in the incident timeline. Make actions idempotent so a retry cannot restart a service twice or delete the wrong node. Enforce blast-radius limits such as one pod or one node per run, and block actions on production databases without a second approver. Add dry-run mode, timeouts, and automatic rollback or escalation on failure. Keep production and staging separate, and test each automation against a staging incident before enabling it. Log who triggered the action, what parameters were used, and what changed. Rate-limit automation so a storm cannot trigger hundreds of concurrent jobs. Finally, define a kill switch and require review after any failed or rolled-back automation. These controls let the team move faster without giving a script standing production power.
3. How would you measure success and detect harmful automation?
Read the worked answer
Measure both toil reduction and safety. Track automation coverage of eligible incidents, median time saved per incident, MTTR for automated versus manual cases, approval latency, success rate, rollback rate, and the number of incidents where automation made the situation worse. Survey responders on whether automation reduced context switching. Baseline before rollout and compare by incident type. Detect harmful automation through failed job rate, repeated retries, rollback frequency, and incidents reopened after an automated action. Review a sample of automated incidents weekly. A 50% toil reduction target is only meaningful if the automated incidents were safe to automate. If restart automation succeeds often but causes flapping services, the metric looks good while reliability gets worse. Pair toil hours saved with change failure rate and customer-impacting regressions so the VP sees the trade-off.