A 1,180-case regression library, six testers, and an audit obligation
Northwind Distribution runs Dynamics 365 Supply Chain Management with 14 extensions and is preparing to move from version 10.0.37 to 10.0.41. Its regression library holds 1,180 manual test cases accumulated over four years, and nobody can say which still reflect the current processes. Six testers are available for a three-week window. Microsoft hotfixes reach production twice a month. The functional leads want a full manual regression before every update; the IT director wants the entire suite automated in the Regression Suite Automation Tool; internal audit requires evidence, after every update, that segregation-of-duties controls and financial posting behave correctly. There is no dedicated automation engineer. Two earlier RSAT attempts were abandoned because recordings broke whenever a form changed, and the test environment is shared with a project team that rebuilds configuration weekly.
1. Design the regression strategy. Which tests run when, and how is the audit requirement satisfied?
Read the worked answer
Build the suite from risk, not from the library. First, retire: each functional lead marks every case keep, rewrite, or delete, with a rule that a case with no defect found in two years and no matching current process is deleted. Expect to lose 300–400 cases, and that is a gain.
Then tier by business consequence and change exposure. Tier 0, roughly 20–30 cases: period close and posting, bank reconciliation, sales tax calculation, inventory valuation, and the segregation-of-duties role checks the auditors care about. These run on every update without exception, with results stored against the update build so the evidence exists when audit asks. Tier 1, 120–180 cases: end-to-end flows per value stream, run whenever the update or a changed extension touches that stream. Tier 2: the long tail, executed at major releases only.
Translate the audit requirement precisely - it is not "all tests passed," it is "the controls operated and postings were correct after the change." Write four or five control tests that assert on the resulting ledger entries and on role assignments, and retain the artefacts. Add a short time-boxed exploratory session per area on each update, because scripted regression only finds what someone already thought of.
2. Both prior RSAT initiatives failed. What would you do differently so automation survives this time?
Read the worked answer
The two failures tell you the problem is ownership and maintenance economics, not the tool. RSAT replays Task Recorder recordings through the client, so any changed form, new mandatory field, or shifted lookup breaks a step. That is a recurring cost that must be budgeted like code, not treated as a one-off project.
Three changes. Assign ownership and version the assets. The suite gets a named owner, lives in source control alongside the build it validates, and is versioned with that build; an unowned suite is dead within two releases. Automate selectively. Only cases that run at least monthly, are data-driven, and sit on a stable UI are worth automating - realistically 60–120 of the 1,180, not all of them. Use the developer recording option so recordings are not tied to one user's security setup, externalise data into parameters, and split long flows into composable modules so a single page change invalidates one module rather than an entire scenario. Push logic testing left. For posting logic, price calculation, and dimension derivation, use ATL/SysTest in the build pipeline, where maintenance cost is an order of magnitude lower and failures surface before the test window starts.
Accept manual execution for judgement-heavy flows and anything touched less than quarterly, and say so explicitly so the IT director's expectation is reset rather than quietly ignored.
3. How should UAT be structured and what should constitute sign-off?
Read the worked answer
Define exit criteria before UAT starts, in writing: which scripts must execute, against which data set, and what defect threshold blocks sign-off - for example no open severity A, no severity B in the top ten processes, and all control tests evidenced. Without that, sign-off becomes an end-of-window negotiation.
Two structural issues to fix now. First, if the key users who designed the process also accept it, UAT confirms the design rather than the requirement. Bring in users from a site outside the design team and give them business scenarios, not scripts, for at least the critical flows. Second, the shared environment and weekly configuration rebuilds will invalidate UAT runs - freeze configuration for the UAT window and record the exact build and data snapshot that was signed off, or the evidence is worthless.
Run UAT on migrated data rather than hand-keyed data, and include at least one full period-end inside the window, because most post-go-live surprises are close-related. Track defect ageing daily, and agree in advance what happens to defects deferred past go-live: a named owner, a target date, and a compensating manual control, accepted by the business process owner rather than by IT alone.