Automating the Gate Instead of Removing It
An insurer's 300-person IT group runs two automation initiatives. The first auto-approves change tickets whose fields match a template, cutting approval time from three days to four hours; the change board still exists and reviews everything else. The second is infrastructure as code: 2 of 40 production servers are described in Terraform, engineers may still make console changes, and nobody runs a drift check. A recent audit found 11 undocumented configuration changes, and six of last quarter's incidents trace to configuration not held in source control. Separately, 30% of the automated tests are Selenium UI tests; the pipeline is usually green, yet production breaks about once a fortnight.
1. Assess the auto-approval automation. Is it a DevOps improvement, and how would you test that claim empirically?
Read the worked answer
Partly, and it is a bridge rather than a destination.
It is a genuine improvement in one respect: the 97%-style wait that a ticket queue imposes on routine work is removed for the templated case, and three days becomes four hours. That is real flow.
But it automates the control rather than removing the need for it. The ticket is still the unit of change; the automation rewards field-matching, not risk. A change can be routine-looking and dangerous, and a template check will wave it through. Nothing here verifies that the change was tested, reviewed, scanned or reversible.
The empirical test Compare change failure rate and restore time for auto-approved changes against board-reviewed changes over a quarter, matched for service criticality. Two outcomes matter:
- If failure rates are similar and low, the board is not adding safety for the standard category and the real fix is making the pipeline produce the evidence automatically.
- If auto-approved changes fail more, the template is not a risk classifier and needs tightening before expanding scope.
Direction of travel Move the control into the pipeline: automated tests, security and policy scanning, peer review recorded, canary, documented rollback. Then the audit trail is a by-product rather than a form.
2. Address the infrastructure drift problem. What would you put in place, in what order, and what is the hardest part?
Read the worked answer
Order
- Inventory and import. Get all 40 servers into code, starting with the 38 outside Terraform. During import, accept the current state as the baseline - the goal is to make reality legible, not to correct it in the same step.
- Make the pipeline the only route to change. This is a policy and access change, not a tooling one: remove standing console credentials and require a break-glass path that is logged, time-boxed and reviewed.
- Continuous drift detection. Schedule a plan-and-compare on every managed resource and alert on any difference between code and reality. Without alerting, drift detection is just a weekly disappointment.
- Replaceable infrastructure. Move toward rebuild-from-code rather than in-place mutation, which is what makes drift expensive in the first place.
The hard part It is socio-technical, not technical. Console access is faster during an incident, and engineers will use it - that is a rational response to a break-glass process that is slow or punitive. Legacy stateful servers may be genuinely unreproducible. So define an explicit exception route with an expiry, make break-glass fast and audited, and use the six incidents as the shared evidence for why the change is worth it. Pairing and teaching during the transition matters more than the tooling choice.
Also note the causal link to the pipeline problem: 'green but broke' is often drift - the tests ran against something that is no longer what production looks like.
3. There is one quarter of budget. Option A: raise UI test coverage from 30% to 60%. Option B: build API and contract tests plus fast unit tests, and gate the pipeline on them and on a post-deploy smoke check. Advise, and say how you would judge success.
Read the worked answer
Take B.
- UI tests are slow, brittle and sit at the top of the pyramid. Doubling Selenium coverage typically doubles suite duration and flake rate while still exercising the system through its most volatile layer. Feedback arrives too late to be useful.
- The stated problem is not low coverage, it is a green pipeline that does not predict production. 30% is a vanity number; coverage measures code executed, not risk covered. Adding more of the same kind of test will not change the pipeline's predictive power.
- API and contract tests catch the integration and schema mismatches that actually break services, run in seconds or minutes, and can be trusted because they are stable. Gating on them makes green mean something.
- The post-deploy smoke check directly addresses 'green but broke', and combined with the drift work in the previous question it verifies the artefact that is really running, in the environment it is really running in.
Judging success
- Escaped defect rate and the proportion of incidents caught before customer impact.
- Pipeline duration and flake rate, which determine whether anyone trusts the gate.
- Change failure rate and restore time, as the outcome measures.
- Deliberately not: coverage percentage, or test count.