Post-Upgrade Failures Across a Distributed Environment

Roll back is unsafe here, so the decision is roll forward - but only after establishing what actually happened.

Why rollback is unsafe. The backup was taken after the schema migration had partially run. Restoring it restores a partially migrated schema, not the pre-upgrade state. A true rollback would need a backup from before the first script execution, which does not exist. Restoring the available backup could leave the database in a state neither the old nor the new application server supports.

Why roll forward is feasible. The application server and database are the authoritative pair. If the schema migration completed on a re-run, the database is at the target version; the remaining work is bringing every client and runtime resource to match. The failures are consistent with version mismatch, not with data loss.

First actions:

1. Determine the exact schema version the database is at, using the vendor's version query.
2. Re-run the migration only if the tooling is idempotent and the vendor supports it; otherwise engage vendor support before touching the schema again.
3. Inventory every application server and runtime resource version by querying each host, since no pre-upgrade inventory exists.
4. Freeze changes until the version matrix is known.

Document the backup timing failure as a process defect - the backup must precede the first migration execution.

The application server and runtime resources must speak a compatible protocol. Upgrading runtime resources first puts newer clients in front of an older application server, which can register but then fail on work dispatch because the message contract differs. The vendor's ordering exists precisely to prevent this window.

Recovery for the two resources:

1. Confirm the application server is now at the target version. If it is not, finish that upgrade first - nothing else will work until the server and database agree.
2. Compare each resource's installed version against the target. Reinstall or repair the resource to match the server exactly.
3. Clear any stale registration for those resources and re-register them, since a rejected work item can leave state that survives a version fix.
4. Run a controlled test process on each resource and confirm it completes before returning it to the pool.

Broader fix. Apply the same version check to all 30 resources and 60 attended users' access points. Build a version matrix as a standing artefact, and add a pre-upgrade step that records every installed version so the next upgrade starts from a known baseline rather than reconstructing one under incident pressure.

The most likely distinction is version or configuration of the client the user connects through, not the user's Blue Prism role. Users whose client was updated to match the server work; users still on the old client fail at launch because the process definition references features or object versions the old client cannot interpret.

Confirm by:

1. Grouping users by outcome and recording the client version for each group. A clean split along version lines confirms the hypothesis.
2. Checking process-level dependencies. If only some processes fail for all users, the cause is the process, not the client. If all processes fail for some users, the cause is the client.
3. Reviewing release notes for changes to process behaviour between versions, and testing one failing process under a matched client.
4. Confirming role changes were not made during the upgrade - a role change would produce access errors, not immediate process failures.

Remediation is to bring every client to the target version through a controlled distribution, not to ask users to update themselves. Publish the version matrix and require a version check at login during the transition period so the gap cannot reopen silently.