ILM rollover stuck for six weeks on a 2 TB per day log index
An observability team manages a 14-node cluster (10 hot, 4 warm) for application logs from 600 Kubernetes pods. Logs are written to the alias app-logs-write, which points to a single index app-logs-prod-000001 with 12 primaries and 1 replica. The ILM policy has a hot phase with rollover at max_primary_shard_size 50gb or max_age 1d, a warm phase at 7 days, and delete at 30 days. Daily ingest is 2.1 TB. After six weeks, app-logs-prod-000001 holds 5.8 TB across its shards, searches over the last hour take 8-12 seconds, and some nodes exceed the 85 percent disk watermark. GET app-logs-prod-000001/_ilm/explain shows the rollover action in the check-rollover-ready step with is_auto_retryable true and the error: index.lifecycle.rollover_alias [app-logs-write] does not point to index [app-logs-prod-000001]. The alias actually points to a newer index created by a sidecar process, and older data is split across two naming schemes. The team cannot reindex 5.8 TB during peak hours and must keep 30-day retention.
1. Explain why ILM never rolled the index over and what the alias and index state should have been.
Read the worked answer
ILM rollover is evaluated against the index that carries the policy, and it requires the configured rollover alias to resolve to exactly that index. The explain output says it plainly: the alias no longer points at app-logs-prod-000001. A sidecar process re-pointed app-logs-write at its own newer index, so every policy poll fails the check-rollover-ready condition. Because is_auto_retryable is true, ILM retries forever and never advances to warm or delete; the 12 primaries keep absorbing writes and grow far past the 50 GB target.
Correct state is one write alias pointing to one index, with that index named to match the rollover pattern (a trailing hyphen and number, such as app-logs-prod-000001). A separate read alias may point at many indices for search. Confirm with GET _cat/aliases/app-logs-write?v, GET app-logs-prod-000001/_ilm/explain, and GET _ilm/policy/<policy>; also check which process re-pointed the alias, because it will do so again unless stopped. Simply deleting the policy hides the problem and leaves a 5.8 TB index with no retention.
2. Propose the target data stream and ILM design, including shard sizing arithmetic and tiering for 2.1 TB per day with 30-day retention.
Read the worked answer
Write to a data stream such as logs-app-default rather than a hand-managed alias. A data stream owns the write index and performs rollover itself, so a sidecar cannot re-point the write target. Put an index template at a higher priority than any existing template, with data_stream: {}, explicit mappings for the fields the teams query, and a rollover condition on max_primary_shard_size plus max_age as a safety net.
At 2.1 TB per day, 50 GB shards give roughly 42 primary shards per day. Over 30 days that is about 1,260 primaries, or 2,520 shards with one replica; on 14 nodes that is a high but survivable shard count, so consider 100 GB shards in warm tiers or fewer replicas after the hot phase. Keep hot at 3 days, warm from day 4 with force-merge, cold or frozen with searchable snapshots from day 8, and delete at day 30. Searchable snapshots cut local disk for the 22 oldest days, which is where most of the 63 TB lives.
Set index.codec: best_compression for the warm phase and avoid over-sharding: one primary per backing index keeps rollover arithmetic predictable. Validate the policy with a short max_age in a test data stream before production.
3. Describe a migration that avoids a 5.8 TB reindex and stops the failed rollover loop without losing query access to historical logs.
Read the worked answer
Freeze the old path first. Stop the sidecar from creating or re-pointing aliases, and remove the ILM policy from app-logs-prod-000001 with POST app-logs-prod-000001/_ilm/remove. That stops the retry loop without deleting data. Leave a read alias such as app-logs-read pointing at the old index so dashboards and saved searches still work.
Create the new data stream and its index template, then switch producers to write to logs-app-default. Update the Elastic Agent or Filebeat output and any sidecar configuration in the same change window, and verify new documents with GET logs-app-default/_stats and a Discover query. New data now rolls over automatically.
For the old index, mark it read-only after writes stop and let a scheduled delete remove it once it passes 30 days. If the old index is needed longer, move it to warm and force-merge during off-peak hours; do not reindex it. Keep the read alias until the retention window expires, then remove it. Document the cutover so a future sidecar cannot reintroduce a second naming scheme.