Mixed DIMM Population and Correctable ECC Errors on a New Node
A four-node cluster is being staged with two-socket servers that have 12 DIMM slots per socket across six memory channels (two slots per channel), for 24 slots per node. The design calls for identical population on every node, but a supply shortage means node 3 was built with 12 x 32 GB dual-rank RDIMMs on socket 1 and 12 x 16 GB single-rank RDIMMs on socket 2, and four of the socket 2 modules are a different vendor with different timings. The node boots and reports 576 GB. The BIOS memory map shows socket 1 training at a higher speed than socket 2, and vSphere sees two NUMA nodes with very different memory sizes. Over the first week, node 3 logs more than 300 correctable ECC errors, nearly all on one socket 2 module, and after an unplanned reboot the CIMC reports that module as disabled and capacity drops to 560 GB.
1. Assess the population and rank mix. What is technically wrong and what is the practical performance consequence?
Read the worked answer
The channel count is satisfied on both sockets, since 12 modules over six channels is two per channel and that is a balanced 2DPC layout. The problem is that the two sockets are not populated alike: dual-rank 32 GB 2DPC on socket 1 has different electrical loading and typically trains one or two speed bins lower than single-rank 2DPC, so the sockets run at different speeds. Mixing vendors and timings within socket 2 forces the platform to use the slowest common timing on that socket, and the modules may also differ in rank or refresh behavior. The practical consequences are an asymmetric NUMA topology (384 GB versus 192 GB) and unequal memory bandwidth per socket, which skews VM placement, breaks any assumption of uniform vNUMA, and makes benchmark numbers non-comparable across nodes. This is not a configuration to standardize on. If the node must run temporarily, keep the population identical within each socket and within each channel, and place memory-heavy VMs where the capacity actually is; long term, rebuild the node with matched modules so all four nodes are identical.
2. The correctable errors and the disabled module: how do you separate a DIMM fault from a slot or CPU fault, and what is the urgency?
Read the worked answer
Treat correctable errors as a signal to investigate, not as an emergency, unless they are climbing fast or multi-bit errors appear. Collect the event log entries with the failing module's slot and FRU, the per-DIMM error counters, the BIOS version and any memory configuration changes, and the part numbers of all modules. Then isolate by substitution: move the suspect module to a known-good identical slot, ideally on the other socket, and watch whether the errors follow the module or stay with the slot. If they follow the module, replace the module. If they stay, the fault is the slot, the memory buffer, or the memory controller in the CPU, and the board or processor becomes the suspect. Before condemning hardware, confirm the module is fully seated with the latch closed, and check inlet temperature, since heat accelerates retention failures. The disabled state is a fault-management action taken after a threshold is crossed, so the drop to 560 GB is an expected consequence rather than a second fault. Schedule the swap in a maintenance window unless uncorrectable errors appear.
3. What baseline would you record before returning node 3 to service, and why does each item matter?
Read the worked answer
Record the full slot map: for all 24 slots, the size, rank, vendor, part number, and rated speed of the installed module, so the next engineer can tell at a glance whether the node matches its three peers. Record the trained memory speed per socket and the resulting NUMA topology and total capacity, because that is what vSphere sizing and any vNUMA-aware workload will actually see, and it is the number that changes silently when a module is disabled. Capture the corrected-error threshold configuration and whether error polling or patrol scrub is enabled, so a future spike can be compared against this baseline. Note the BIOS version and memory mode settings, and confirm nothing was left in a non-default mode during troubleshooting. Finally, attach the hardware case number and the replacement module serial, and check that no VM with a large memory reservation has been placed on the smaller NUMA node in the meantime. Without this baseline the next ECC event starts from zero again.