Tool selection: codeless vendor versus an open-source stack for a mixed mainframe, API and web estate
Northwind Insurance is selecting an automation tool for a claims platform with three interfaces: 400 IBM 3270 green screens, 260 REST endpoints, and an Angular back-office UI. Six delivery teams and 40 testers support a release every four weeks; only three testers write code comfortably. The current stack is open source (Selenium, REST Assured, Robot Framework) orchestrated by Jenkins, with 1,900 scripts that take 11 hours nightly and are maintained by one part-time engineer. A vendor, AutoSuite Pro, demonstrated a codeless AI record-and-playback tool on its own environment using six cherry-picked cases, and quoted EUR 180,000 over three years for 40 named users. The CIO has asked for a recommendation in four weeks, and the vendor is offering a discount if a contract is signed before the proof of concept finishes.
1. Design the evaluation so that the recommendation is defensible. What must the proof of concept contain and measure, and what should be rejected about the vendor's current evidence?
Read the worked answer
The vendor's demonstration fails four tests of evidence: it ran on the vendor's environment, on cases the vendor selected, with no failure or change scenario, and it is coupled to a discount deadline that removes the ability to walk away. All four must be neutralised.
Criteria and weights first. Agree weighted criteria with the six teams before any demo: coverage of 3270, REST and Angular; maintainability under change; authoring effort for a non-programmer; CI integration with Jenkins; reporting and traceability for audit; skills ramp; total cost over three years; exit cost. Publish the weights so scores are not retrofitted to a preferred answer.
Representative, adversarial case set. Select 20-25 cases jointly, including the hardest ones: a 3270 flow with paging and function keys, a dynamic Angular grid with generated element IDs, an asynchronous claims status transition, a negative validation path, and a data-dependent case. The same case set runs on both candidates.
Own environment, own data. Run in Northwind's infrastructure against a representative test environment and a masked or synthetic dataset, integrated with the existing Jenkins pipeline and reporting.
Measure, don't opine. Record authoring time per case by a tester and by a developer, first-run pass rate, and critically a change simulation: after the case set works, modify a screen label, reorder two fields, and change an API response field name, then measure repair effort and elapsed time in each tool. This is the only credible proxy for the 35% maintenance burden now carried by one engineer.
Include an exit assessment. How portable are the assets if the contract ends? Ask both candidates to explain what a script becomes outside the tool.
2. How would you structure the three-year cost comparison, and which cost lines are most often omitted?
Read the worked answer
The EUR 180,000 licence figure is a floor, not a total. Build the comparison as total cost of ownership per year, with both candidates costed on the same assumptions: 40 named users, 6 teams, 1,900 existing scripts, four-week release cadence.
Licence and infrastructure: named versus concurrent users, add-on modules for 3270 or API, test management and reporting modules, execution agents, environment cost, and the residual value of the existing open-source tooling that would be retired.
People cost, which dominates. Estimate person-days for initial training, ramp to productivity for 40 testers with mixed coding skill, and the ongoing maintenance ratio. Ask each candidate for a reference customer who will discuss maintenance effort after 18 months; a demo cannot answer this. Also cost the retention risk: if only three testers can code, a tool that requires coding concentrates knowledge in three people.
Frequently omitted lines: scripting or professional services for the first large migration; environment and data provisioning upgrades needed to make either tool performant; orchestration and reporting integration work with Jenkins; version upgrades and regression of the tool itself; parallel-execution licensing; and the cost of the discount deadline itself, which removes competitive tension and any option to re-tender.
Exit and lock-in. Cost the effort to extract 1,900 scripts into a neutral form, or accept that a codeless tool's assets are not portable and treat three years of licence as the sunk cost.
The comparison should be presented as a range with stated assumptions, plus sensitivity: if maintenance effort under AutoSuite Pro is twice the vendor estimate, does the decision flip? If it does, the decision is not really about price and should be made on maintainability evidence from the change simulation.
3. Regardless of which tool wins, what do you recommend for the 1,900 existing scripts and the 11-hour nightly run?
Read the worked answer
Tool choice does not fix an 11-hour suite maintained by one part-time engineer; that is a design and ownership problem. Recommendations:
Freeze and triage before migrating. Inventory the 1,900 scripts against requirements and defect history, and classify: keep (covers high-risk, frequently changing behaviour), refactor (valuable but brittle, e.g. record-and-playback artefacts with absolute XPaths), retire (duplicates, tests of trivial getters, cases whose requirement no longer exists). Realistically 20-30% is retirable, which reduces both migration cost and runtime before any tool decision is made. Migrating a redundant script to a new tool simply buys a more expensive redundant script.
Introduce a maintainable structure as a prerequisite. Page objects or screen objects per 3270 screen map, keyword or task-level abstractions for business flows, and API-level setup for data. This makes a later tool change survivable and gives a fair baseline for comparing candidates.
Cut runtime by design, not by budget. Most of 11 hours is likely sequential execution, fixed waits and redundant login/data setup. Parallelise by team-owned packs, replace waits with condition polling, and move data setup to APIs. A 40-50% reduction is normally achievable before buying anything.
Strangler migration, not big bang. New and changed tests go to the chosen approach immediately; legacy scripts are converted only when touched for other reasons, with a target retirement date. Big-bang migration of 1,900 scripts in a four-week release cycle will fail and will be blamed on the tool.
Fix ownership. Assign named maintainers per pack, with a maintenance budget and a definition of done that includes the test asset, not just the product change.