Memory upgrade on a NUMA-sensitive database host

Both options reach 512 GB, so the decision is about bandwidth, downtime, and future headroom, not capacity.

• Option A - add 8 x 32 GB RDIMMs. No removal of existing parts, fits the 90-minute window (power down, populate 8 empty slots, POST, boot). Result: 16 DIMMs, 2 DIMMs per channel, 512 GB at 2933 MT/s. That is 2933 / 3200 = 91.7% of peak bandwidth, roughly an 8% loss. All slots are then consumed, so future growth means throwing away the whole set.
• Option B - replace with 8 x 64 GB LRDIMMs. Mixing types is prohibited, so all eight 32 GB RDIMMs come out: longer downtime, higher cost, resale/ disposal of the old parts. Result: 512 GB at full 3200 MT/s at 1 DIMM per channel, with 8 slots free for a later 1 TB upgrade.

Decision rule: for an in-memory OLTP engine, find out whether the platform is actually bandwidth-bound before paying for speed. Pull the database engine's memory-bandwidth and NUMA-locality counters at peak load.

• If peak memory bandwidth utilization is already above ~80% of theoretical, take Option B; the 8% is real throughput.
• If it sits at 50-60%, take Option A; the speed difference will be invisible and you keep budget and window risk down.
• With no measurement available and a reasonable expectation that the database outgrows 512 GB within the server's life, choose B for the free slots.

A vendor support statement that the application is not certified at 2 DIMMs per channel would also force B.

Before:

• Confirm BIOS/firmware supports the new DIMM density. A stale BIOS can either refuse the parts or train them at a fallback speed (2400 MT/s), which silently costs you more than the 2DPC penalty you were trying to avoid.
• Record the current memory speed, per-socket capacity, and NUMA topology as a baseline (dmidecode -t memory or the BMC memory page). Without a before-picture you cannot prove an after-problem.
• Confirm DIMM population rules: identical DIMMs, populated in matched sets across channels, and the same count per CPU so the two NUMA nodes stay symmetric.
• Take a database-visible performance baseline (transactions/sec, p95 latency) and confirm a valid backup exists.

After:

• BIOS and OS both report 512 GB (not 480 or 256), with two symmetric NUMA nodes of 256 GB each. An asymmetric topology re-introduces remote-memory access for half the workload and is easy to miss.
• The reported speed is the expected 2933 or 3200 MT/s.
• Run at least one full pass of the vendor memory diagnostic or a memory stress test before returning to service; a badly seated DIMM often only shows under load.
• Re-run the same database benchmark and compare against the baseline. If throughput improved less than the capacity increase predicts, the new bottleneck is likely bandwidth, which is the signal to revisit the choice in the previous task.

Do not install six. Populating 6 of the 8 second-channel slots produces an uneven channel map: some channels run 2 DIMMs, some run 1, on one or both sockets. The memory controller interleaves across populated channels, so the channels with a single DIMM serve fewer ranks and the interleave becomes unbalanced. The practical result is worse average bandwidth than a clean configuration, plus unpredictable NUMA behaviour, and you have still spent the window and the risk.

Options, in order of preference:

1. Hold the window, keep 256 GB, reschedule for the full set. No production risk, no partial state to unwind. If the OLTP workload is already close to its 256 GB working set, this is uncomfortable but recoverable.
2. Install a symmetric subset only if it maps cleanly. If you could populate whole matched pairs across both sockets - for example four DIMMs, two per CPU in the same channel position on each - you would gain capacity with symmetry preserved. Two DIMMs per CPU out of eight channels leaves 6 of 8 channels idle per socket, so per-socket bandwidth drops substantially. That is a bigger loss than the 2DPC penalty you were weighing earlier, so it is rarely the right trade.
3. Borrow or expedite the remaining two DIMMs from the same vendor part number. Do not substitute a different density or a different rank count to make up the pair.

If the application truly cannot wait, the honest answer is that the shortfall is a supply problem, not a configuration problem, and it needs a business decision about running degraded rather than a clever DIMM layout.