400 Scripts, 30% Green

Automation is a capital investment repaid by repeated execution, so the criteria are economic and technical, not aspirational.

Selection criteria:

1. Execution frequency. A case run 12 times a year or more is a candidate; a case run once is not.
2. Stability of the feature and its UI. Automating a screen that changes every sprint guarantees rewrite cost. Stable business rules are the best targets.
3. Deterministic, machine-checkable outcome. If deciding pass or fail needs human judgement, automation will produce false verdicts.
4. Test level. Business logic belongs at API or service level: faster, cheaper, less brittle. UI automation should be reserved for a thin set of end-to-end journeys and rendering checks that cannot be verified lower down.
5. Risk coverage. Critical paths justify automation even at higher cost, because the repeatability buys release confidence.
6. Setup and data feasibility. Cases needing complex manual data preparation cost far more than the three-day estimate suggests.

Apply to the 1,600: most fail criteria 1-4. A rough first target is 150-250 cases - the 100-150 high-frequency regression rules that can be pushed down to the API layer, the data-driven boundary sets (boundary value tables are ideal automation material), and 20-30 UI journeys covering login, payment authorisation and refund end to end.

Do the arithmetic for the director: 1,600 x 3 person-days = 4,800 person-days, roughly 2.6 years of the entire five-person team before a single maintenance hour, and the estimate excludes framework work, test data, environment and triage. Even if the estimate is five times too pessimistic, the proposal does not survive contact with the schedule.

Thirty percent false failures means the gate is already ignored in practice, so the first job is to restore trust before adding anything.

Policy:

1. Detect. Any test that fails and then passes on re-run with no product change, or fails in more than one run in twenty, is tagged flaky within 24 hours and logged with the failure output, screenshot and environment.
2. Quarantine, do not delete on sight. Remove from the blocking smoke gate immediately, keep running in a separate non-blocking job so the failure data keeps accumulating. A flaky test in the gate is worse than no test, because it teaches people to ignore red.
3. Assign and time-box. Each quarantined test gets a named owner and a fix-or-delete deadline, for example ten working days. At the deadline: fix it, or delete it and replace it with either a lower-level test or a manual case. A permanently quarantined test is a silent coverage hole.
4. No blanket re-run-to-green. Automatic retries hide real intermittent product defects, which are exactly the defects that reach production. If retries are permitted at all, restrict them to quarantined tests, cap at one, and log every retry.
5. Fix root causes, not symptoms. The usual culprits are fixed sleeps instead of explicit waits, shared test data and order dependency, environment and network instability, and non-deterministic oracles. Each quarantined test should be categorised so the pattern is visible.
6. Publish the metric. Flaky rate as a percentage of the suite, with a target under 2%, reviewed weekly. It is a quality measure of the test asset, not of the product.

Announce the rule that a red gate always means a real problem. That single commitment is what makes the six-hour suite worth its runtime.

Current manual baseline: 200 critical cases x 12 releases = 2,400 case-executions per year at five person-days per pass, so 60 person-days a year just for that regression, excluding the existing 55% maintenance load on the 400 scripts.

Naive automation case: automating 120 of those cases at three person-days each is 360 build days, plus roughly 20-30% of build effort per year in maintenance. Against a 60-day annual saving, payback is many years out. On labour arbitrage alone, the investment does not hold - and saying so first is what makes the rest of the argument credible.

Where automation actually pays:

• Feedback latency. A six-hour suite versus a five-day manual pass changes when defects are found, and defects found earlier are cheaper by an order of magnitude. That is the real return, and it is invisible in a person-day comparison.
• Consistency and auditability. Every release runs the identical check; no step is skipped because it is Friday.
• Coverage humans cannot do. Large data permutations, cross-browser matrices, concurrency and load.
• Regression confidence per change, which lets the team refactor instead of freezing.

Where it does not pay: one-off cases, unstable UI, anything needing human judgement, and anything better tested at API level. Record-and-playback sits badly with all four: it produces brittle selectors, no reusable data setup, and no meaningful assertions.

Recommended sequence: first stabilise and speed the existing 400 (parallelise, push logic to API, fix flakiness); second, automate the high-frequency regression rules at API level; third, expand UI coverage only for critical journeys. Ask the director to fund the stabilisation phase before the expansion phase, and to review the flaky rate as the gate for scaling up.