ILM rollover, warm-tier migration and a read-only index

allocate only applies the routing rules you give it - it does not infer them from node.attr.box_type. If the action is {"allocate": {}} (or has no require/include), it is a no-op that reports complete.

Second common cause: setting index.routing.allocation.require.box_type: hot at index creation. A require rule is absolute - no ILM action that only sets include can override it, and the shard will never satisfy a warm requirement while the hot requirement stands.

Third: mixing the 7.10+ tier system with attribute rules. If index.routing.allocation.include._tier_preference is present it is also evaluated; a conflicting attribute rule wins and can pin the shard.

A working warm phase:

{"warm": {
  "min_age": "2d",
  "actions": {
    "allocate": {
      "number_of_replicas": 1,
      "require": {"box_type": "warm"}
    },
    "forcemerge": {"max_num_segments": 1},
    "shrink": {"number_of_shards": 1}
  }
}}

If you stay on the tier system instead, drop the attribute rule and use "include": {"_tier_preference": "data_warm,data_hot"} plus node.roles: [data_warm] on the warm nodes.

Validate before rolling out to 30 days of indices: POST _ilm/move/logs-000010 {"current_step": {"phase": "hot", "action": "complete", "name": "complete"}, "next_step": {"phase": "warm", "action": "allocate", "name": "allocate"}}, then re-check _ilm/explain - the step should move to migrating, not instantly to complete.

A shard on a node above the flood-stage watermark (default 95%) causes Elasticsearch to set index.blocks.read_only_allow_delete: true on the affected indices. The block is a symptom; the disk is the cause.

Order matters because clearing the block while the disk is still over the flood stage just re-triggers it, and the index will flap between writable and read-only - often mid-bulk.

1. Stop the bleeding: roll over immediately (POST logs-write/_rollover) so new writes go to a fresh index; optionally index.refresh_interval: 30s temporarily on the new index.
2. Free space on the hot tier: delete or ILM-expire the oldest indices, force-merge read-only indices, or move data off. Confirm with GET _cat/allocation?v that the node is now below the high watermark (default 90%) - below high is not enough if you only skim just under flood.
3. Clear the block once, cluster-wide:

   PUT _all/_settings
   {"index.blocks.read_only_allow_delete": null}
   

   null removes the setting; false also works but leaves an explicit setting behind.
4. Verify: GET _cat/indices?v&health=red,yellow and a small test bulk into logs-write.

Longer-term: the hot tier at 91% is already over the high watermark, so shard relocation pressure is running continuously. Either add hot capacity, shorten the rollover threshold, or lower max_primary_shard_size so shards are smaller and easier to relocate.

Rollover needs an alias pointing at exactly one index that has is_write_index: true, and the index name should end in a zero-padded number so the next index can be derived. A manually created index with a plain alias produces either "Rollover alias [logs-write] can point to multiple indices, found duplicated alias" or "index.lifecycle.rollover_alias [logs-write] does not point to an index with [is_write_index] set to true".

Repair, without dropping the alias:

POST _aliases
{
  "actions": [
    {"add": {"index": "logs-000042", "alias": "logs-write", "is_write_index": true}}
  ]
}

Then confirm the policy is attached (GET logs-000042/_ilm/explain) - an index created outside the template has no index.lifecycle.name, so add it:

PUT logs-000042/_settings
{"index.lifecycle.name": "logs-policy", "index.lifecycle.rollover_alias": "logs-write"}

Finally, fix the root cause: the index template for logs-* should carry index.lifecycle.name, index.lifecycle.rollover_alias: logs-write, the shard settings and the tier/attribute rules, so no index is ever hand-created. The one-time bootstrap (logs-000001 with is_write_index: true) is the only manual step that should ever be needed.