Periodic 40 ms write-latency spikes on a RAID 10 database server

Root cause: the controller's periodic battery relearn forces the cache from write-back to write-through, so every committed write waits for the SSD media instead of the cache.

The causal chain:

1. The controller runs an automatic learn cycle on a roughly 30-day schedule - this matches the observed interval exactly.
2. During relearn the battery is not trusted to protect cached data, so the firmware drops the virtual disk to WriteThrough as a protective measure. The event log and the cache policy read-out confirm it directly.
3. On an OLTP workload dominated by small synchronous log and commit writes, losing write-back means each write pays full media latency plus queueing. That is a two-order-of-magnitude jump, not a minor one.
4. The cycle completes, the firmware restores WriteBack, and latency returns to normal without intervention - which is why it self-heals.

Exclusions:

• CPU, NIC errors, and switch utilization are flat, so this is not a network or host-compute event.
• No deployment, so the application is not the trigger.
• 62% capacity and no mention of SSD wear or garbage collection overhead means capacity pressure is not the driver.
• Queue depth is a symptom, not a cause: writes are slower, so they accumulate in the controller queue. Treating queue depth as the root cause leads to pointless HBA or driver work.
• Battery capacity at 92% means the battery is not failing; a dead battery would leave you in permanent write-through, not 30-minute episodes.

Immediate mitigation:

• Use the controller management utility to take the learn cycle off its automatic schedule and run it manually inside a maintenance window, or reschedule it to a quiet period. The spike does not disappear, it moves somewhere it does not hurt.
• While a relearn is unavoidable, warn the application team and shift batch or reporting jobs out of the window.

Permanent fix, in order of preference:

1. Flash-backed cache module (supercapacitor plus NVRAM) instead of a battery. It has no learn cycle, so the write-back policy never flips. This eliminates the class of problem rather than scheduling around it.
2. Separate the database log onto its own device with independent power-loss protection, so a cache-policy change on the data array cannot stall commits.
3. Some controllers support running write-back without a battery when the data is replicated or the workload can tolerate loss. That is a risk decision to be signed off, not a default - it trades a monthly stall for exposure on power loss.

What you do not do: set the array to write-through permanently. You would trade a 30-minute monthly spike for a permanently worse write latency, which is a net loss.

Replace the battery when: capacity trends below the vendor threshold (commonly around 90% and falling), relearn frequency increases beyond the expected monthly cadence, charger or learn-complete faults appear in the log, the battery ages past its rated service life, or firmware release notes identify a relearn-related defect fixed in a later version - check that before spending money on hardware.

Because the fault is periodic, absence of symptoms for a week proves nothing. Build the proof into monitoring.

• Alert on the leading indicator, not the latency. Create a monitoring trigger on the controller event 'cache policy changed' and on 'relearn started/complete'. Those fire the moment the fault begins, whereas a p95 threshold fires only after users are already affected.
• Trend p95 and p99 write latency weekly and annotate the chart with relearn events. If latency spikes line up with relearn after the change, the scheduling or the module swap did not take effect.
• Verify the next cycle actually occurred where you intended. If you rescheduled relearn to Sunday 02:00, confirm 30 days later that it ran on Sunday 02:00 and that production latency stayed flat during it. If you installed a flash-backed module, confirm in the utility that no learn cycle exists and that cache policy is WriteBack.
• Track battery or supercap health monthly so degradation is caught by trend rather than by an outage, and record it in the same place as the latency data.
• Re-baseline after any controller firmware update - relearn behaviour and default cache policies do change between firmware releases, and the original symptom can return under a new cause.

Post-incident note for the problem record: the real gap was that a protective firmware behaviour was indistinguishable from an application problem for 30 days. Alerting on the controller event converts a monthly mystery into a one-line ticket.