Heap sizing, memory locking and a yellow cluster on a six-node cluster

Reject 56 GB; set -Xms31g -Xmx31g (equal values).

Two constraints bind before any GC tuning:

1. The 50% rule. Heap is only for in-flight segments, cluster state, field data and per-request structures. Lucene reads segment files through the OS page cache, which lives outside the heap. Giving the heap 58% of RAM takes memory away from that cache, so searches and merges start hitting disk.
2. Compressed oops. Above roughly 30–31 GB, HotSpot must use uncompressed 64-bit object references, so every reference grows and the usable heap drops. A 40 GB heap can hold less than a 31 GB heap while producing longer pauses. Confirm from the JVM log line about compressed ordinary object pointers, not from a remembered number.

Budget per host, not per node: if two ES processes share a host, their heaps must still sum to ≤ 50% of RAM. A combined master+data role shares one heap; a dedicated master could run at 4 GB. Also confirm swap is off (bootstrap.memory_lock or swapoff), otherwise the extra heap is wasted.

bootstrap.memory_lock: true makes the node call mlockall at startup. If that call fails, the bootstrap check fails and the node deliberately aborts - this is not a warning. The classic cause is a memlock limit that the ES user (or the container) does not have.

Checks, in order:

# as the user/process that runs Elasticsearch
ulimit -l                  # must be 'unlimited'
cat /proc/<pid>/limits | grep -i 'locked memory'

Fixes, depending on how the process is launched:

• systemd (packaged install): add LimitMEMLOCK=infinity and LimitNOFILE=65535 via drop-in, then systemctl daemon-reload.
• shell / init script: /etc/security/limits.conf → elasticsearch - memlock unlimited, nofile 65535.
• containers: --ulimit memlock=-1:-1.

Also verify the other startup prerequisites before restarting: vm.max_map_count=262144, open file descriptors ≥ 65535, and that the data directory is writable by the ES user.

After restart, verify with GET _nodes?filter_path=nodes.*.process.mlockall - it must return true. If the platform genuinely cannot grant memlock, set bootstrap.memory_lock: false and disable swap instead; that is a documented, supported fallback.

Diagnosis first, not guesswork:

GET _cluster/allocation/explain
{"index": "<index>", "shard": 0, "primary": false}

You will get a message of the form "cannot allocate replica shard because a copy of this shard is already allocated to node X" combined with a _tier_preference: data_hot filter that matched only one node. A replica can never live on the same node as its primary, so the available pool is empty.

Options:

1. Add a second node with the data_hot role - correct fix. Restores redundancy and keeps tier semantics. Cost: capacity.
2. Widen the tier preference: PUT <index>/_settings {"index.routing.allocation.include._tier_preference": "data_hot,data_content"} - replicas can then land on the content nodes. Fast, no new hardware, but hot-tier indices now share I/O with the content workload.
3. Promote an existing content node to data_hot - requires a restart and careful sequencing so primaries are not stranded.
4. number_of_replicas: 0 - reaches green but removes all redundancy; a single node loss makes the index red. Acceptable only for rebuildable data.

Note that adding a seventh node does not fix the memory-lock failure and does not help unless that node carries the data_hot role. State this explicitly to the operations team: the restart failure and the yellow status are two independent problems.