4.0 to 4.1 upgrade on a cluster that is already dropping mutations

Three findings block it, and only one of them is about version compatibility.

1. Dropped mutations. 4,300 dropped MUTATION messages in 24 hours means the mutation stage queue is overflowing and writes are already being discarded. A rolling restart adds hinted handoff, cache warm-up and streaming pressure - it will make an existing write-loss problem worse, and the Saturday window explicitly cannot absorb failed writes. Fix the cause first: 85% CPU with 30% iowait suggests compaction backlog, a hot partition or wide rows, GC pauses, or genuine overload. The phi_convict warnings on the same two nodes are the same symptom seen through gossip: a saturated node looks unresponsive, gets marked down intermittently, and accrues hints. Do not upgrade until drops are zero and phi_convict is quiet for several days.
2. Disk headroom. 210 GB free on 6 TB is 3.5%. Restart-driven compaction, streaming and new SSTables need room; aim for 30–50% or add a node first.
3. Driver matrix. Java driver 3.11 and 4.9 both work against 4.1, but standardise and upgrade drivers before the server, not after, so a client regression is not confused with a server regression.

Before the first restart: pending compactions near zero, no dropped mutations, phi_convict clean, ample disk, a tested rollback (4.1 to 4.0 is supported only if you have not run upgradesstables), and one node at a time with nodetool drain, waiting for UN and drained hints before moving on.

This is the most common way a smooth minor upgrade turns into a multi-day incident.

upgradesstables rewrites every SSTable into the newest on-disk format. On 6 TB per node that is a full rewrite of the dataset: sustained heavy I/O, a compaction queue that will not drain for hours, transient disk growth while old and new files coexist, and a node that is effectively unusable for reads during the rewrite. Multiply by fifteen nodes and the Saturday window is long gone.

More importantly, it is not required. Cassandra 4.1 reads 4.0 'big' format SSTables natively, and older formats too; there is no functional reason to convert. Normal compaction gradually rewrites files into the new format on its own.

There is also a rollback cost. Converting every file removes the option of downgrading to 4.0 without a full restore, because 4.0 cannot read the newer format. Keeping the old format until you are confident in 4.1 is a cheap insurance policy.

If you do eventually run it - for a format-dependent feature or before decommissioning old hardware - then: one node at a time, off-peak, with pending compactions and netstats monitored, -j limited to a couple of threads, disk headroom verified first, and a pause if the queue grows without bound. Never in the same window as the version upgrade itself.

UDFs execute arbitrary code inside the coordinator's JVM. Enabling them is a cluster-wide security decision, not a client library choice, and it should not be bundled into a version upgrade window.

Risk:

• A UDF runs on every node that coordinates a query touching it, so a buggy or hostile function can exhaust heap or spin CPU across the whole cluster - the same cluster already at 85% CPU and dropping writes.
• The scripting sandbox (Nashorn on Java 11) has a history of escapes and is deprecated in the JDK. enable_user_defined_functions alone allows non-scripted UDFs; enable_scripted_user_defined_functions is a second, separate flag that allows JavaScript. Enabling the second flag is materially more dangerous than the first.
• The settings must match on every node or functions fail unpredictably depending on which coordinator handles the query, so 'just turn it on for one node' is not an option.

Decision: deny the JavaScript variant. Require the team to justify the function against built-in CQL functions or client-side computation first, which covers most real cases.

Conditions if a non-scripted UDF is genuinely required:

• Enable only enable_user_defined_functions: true, never the scripted flag.
• Grant CREATE FUNCTION / ALTER only to the team's deployment role, not its runtime role, and keep UDFs inside that team's keyspace.
• Stage on a canary node with the team's real queries and watch heap and CPU before cluster-wide enablement.
• Schedule it as its own rolling change after the upgrade is stable - not in the same Saturday window.