Four-Node Upgrade in a Four-Hour Window

Decision: single-writer upgrade, full outage, no rolling start.

1. Put the load balancer in maintenance mode and drain sessions so nothing lands mid-upgrade.
2. Pause the scheduler on both scheduler-enabled nodes; let in-flight tasks finish or cancel them deliberately.
3. Stop all four nodes. The staging failure is the evidence: two nodes writing schema objects concurrently left tables half-populated, and mixed-version nodes running the same task are not a supported state.
4. Export all custom rules, workflows and plugin artifacts to source control before touching the schema - after the upgrade you need a diff to prove nothing was mutated.
5. Take the database backup. This is the rollback plan; the schema change is forward-only, so the backup is not optional insurance.
6. Run the upgrade on one node with the others stopped.
7. Deploy the identical WAR and identical plugin JARs to every node. Plugins are filesystem artifacts, not database objects - node drift is guaranteed if this step is skipped.
8. Start nodes one at a time, health-check between each, then re-enable the scheduler on exactly one node and exit maintenance mode.

Four hours holds only if steps 4 and 5 are rehearsed and timed in advance.

Do not finish the window with a mixed plugin version. Restart node three with the correct artifact before re-enabling the scheduler.

Reasoning:

• Plugin classes are loaded per JVM from the local filesystem. A node with the older JAR will throw ClassNotFoundException/NoClassDefFoundError when a task, rule or custom connector touches plugin-provided classes - and it will do so unpredictably, at the moment a task happens to be scheduled onto that node.
• Version metadata is recorded in the database, so the mismatch is detectable; use that to verify consistency across all four nodes rather than eyeballing file dates.
• The failure mode is worse than the delay: a half-executed provisioning or aggregation task can leave accounts in an inconsistent state that the next run will not automatically repair.

Cost of the correct choice: minutes. Cost of the wrong choice: a corrupt run discovered days later with no clean rollback. Restart, re-verify, then proceed.

Reject it. The proposal trades a known, bounded outage for an unbounded recovery risk.

• The old application version caches schema metadata at startup. Serving reads against altered tables produces stale-object errors at best and wrong results at worst - and a certification reviewer acting on wrong results is a compliance problem, not just a technical one.
• There is no meaningful read-only mode in the product; UI traffic fires background queries against role, entitlement and identity tables, exactly the tables the upgrade touches.
• The only safe way to shrink the window is to shrink the number of nodes participating, or to move backup and exports outside the window - not to mix versions.

Quantify the trade-off for the change board: a planned two-hour outage versus an unknown recovery that could consume the entire window plus a rollback from backup. Recommend rehearsing the runbook on the staging clone with timing per step, and pre-staging artifacts so the window contains only the actions that require the database offline.