Expand "Configure a lifecycle policy" docs #1906

kilfoyle · 2025-06-24T17:26:39Z

This PR takes a chunk out of internal issue Requested fixes for data lifecycle docs that was initiated by a review of the ILM docs by our Support team.

It updates the "Configure a lifecycle policy" page to:

Add Kibana steps where we currently show only the API steps.
- In particular, when creating an index template users aren't always sure what to specify on the "Index settings" tab, so this adds an example config.
Add a page overview
Add a section about viewing the ILM status for an index or datastream
Fix up smaller items, such as:
- Explicitly call out the Kibana "Data retention" option.
- Emphasize that lifecycle phase changes are based on time since rollover rather than index creation time
- Warn about updating the logs@lifecycle and metrics@lifecycle policies since they affect a LOT of indices.
Provide links for things like the index lifecycle actions, mappings, etc., to help people understand these options.

Please see preview pages:

(For reference, here's the original version of the first page).

ES Data Management team, if any of you can please give this a technical review I'd be very grateful! 🙏 The API instructions aren't changed, with the exception that I added this section about calling the ILM explain API. The Kibana steps are all new.

github-actions · 2025-06-24T17:26:49Z

🔍 Preview links for changed docs:

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

samxbr

Thanks for making this documentation change, I love the new Kibana steps! I just left a few comments.

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

gmarouli

@kilfoyle thank you for doing this, how nice to see it getting some love ❤️ .

I found some spots that might be misleading or incorrect, let me know if you want to go over through some of them offline as well, if that helps.

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

gmarouli · 2025-06-25T08:46:42Z

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

+    1. If you're storing continuously generated, append-only data, you can opt to create [data streams](/manage-data/data-store/data-streams.md) instead of indices for more efficient storage. If you enable this option, you can also enable **Data retention** to configure how long your indexed data is kept.

-::::{tip}
-An `index.lifecycle.rollover_alias` setting is only required if using {{ilm}} with an alias. It is unnecessary when using [Data Streams](../../data-store/data-streams.md).
-::::
+        :::{important}
+        When the **Data retention** option is set, data is guaranteed to be stored for the specified retention duration. Elasticsearch is allowed at a later time to delete data older than this duration. This setting replaces any data retention settings that may be defined in an ILM policy. Refer to the [Data stream retention](/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md) tutorial to learn more.
+        :::


Referring to data retention here is incorrect. Data retention as described in the referenced tutorial is only applicable if the user is using the data stream lifecycle which is an alternative to ILM. Considering this is an ILM tutorial I think we should refrain from mentioning it all together.

A user can still enable the data stream option, it's just that their data stream will be managed by ILM.

Thanks! I've changed it. The Support person who reviewed these docs with us told me that that data retention setting is causing some confusion and that it's not currently documented anywhere. I'll try to fix that separately, but for here, rather than not mentioning data retention do you think this note would be okay instead of what I have above?

NOTE: Since you're creating an index lifecycle policy to manage indices, the Data retention option should be left disabled. Data retention is applicable only if you're using a data stream lifecycle, which is an alternative to ILM. Refer to the Data stream lifecycle to learn more.

Hm, I see. Can you explain to me what the confusion is about and what do you mean when you say Data retention?

I am asking because it's not clear to me what we mean when we say he Data retention option should be left disabled. Is there a screen in kibana or something?

@gmarouli Yup. I understand that it's a new Kibana setting.

On the "Create template" wizard, "Create data stream" is enabled by default and the "Data retention" setting appears but is disabled by default. If I disable "Create data stream" the "Data retention" setting disappears.

The concern that Zoia mentioned is that if someone enables "Data retention" and sets the retention to, say, 30 days, "it doesn't matter how many tiers you have, Elasticsearch will only keep the data on the hot tier for 30 days and then will delete it." So I guess we want people to understand that this setting would effectively enable data stream lifecycle and override any ILM policy they have configured.

(By the way, I'm happy to share the recording of that feedback session or my long, messy set of notes.)

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

gmarouli · 2025-06-25T09:02:51Z

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

+:::::{warning}
+Be careful when changing either the `logs@lifecycle` or `metrics@lifecycle` policies as these typically manage many indices. In {{kib}}, the **Index Lifecycle Policies** table shows the number of indices currently associated with each policy.
+:::::


This is a bit misleading. If I am not mistaken these are managed policies, meaning they are shipped along with elasticsearch. The recommendation for such policies is that the user should not change them, ever. If a user changes them, there are no guarantees that a future upgrade will not overwrite them. In general we recommend to create a new policy and associate them with the intended index templates or the index they want.

Perfect. I've changed this to:

(see here)

Should we add a course of action to ensure the user can recover from removing a policy? Although it's not easy to anticipate everything that could go wrong.

a course of action to ensure the user can recover from removing a policy

I think that's something we could add to the Troubleshooting section of the docs, and then have a link from this page. If that's something we should write up I'd open a separate issue for it rather than tackle it in this PR.

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

kilfoyle · 2025-06-25T22:43:09Z

@gmarouli and @samxbr Thanks so much for the careful review. 🙏

If you don't mind I'd love if you can take a second look. And do let me know if you don't think the new structure makes sense (I tried to explain the rationale in this comment).

stefnestor · 2025-06-25T23:21:00Z

manage-data/lifecycle/index-lifecycle-management/policy-apply.md

+You can also manually apply a lifecycle policy to an existing index, as described here. You can do this in {{kib}} or using the {{es}} API.
+
+::::{important}
+Do not manually apply a policy that uses the rollover action. Policies that use rollover must be applied by the index template. Otherwise, the policy is not carried forward when the rollover action creates a new index.


This statement is good advice & also we apply policies with rollover all the time in Support outside the template. I believe your call out is to not apply a policy to an index where rollover has yet to occur.

Ex: index 1+2 associated to policy A, policy A removed from 1 for manual intervention (ex rehydrating frozen tier), policy A re-added to 1, 1 ILM Move Step pushed past rollover.

@stefnestor Thanks for the explanation! I'm still not sure exactly what the warning should be (it's from the current docs), so based on what you've said I've changed it to:

WARNING: Do not manually apply a policy that uses the rollover action to an index which has not yet rolled over. Otherwise, the policy may not be carried forward when the rollover action creates a new index.

manage-data/lifecycle/index-lifecycle-management/policy-apply.md

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

stefnestor · 2025-06-26T23:43:46Z

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

-::::
+* To use a policy to manage a single index, you can specify a lifecycle policy when you create the index, or apply a policy directly to an existing index.
+
+* {{ilm-init}} policies are stored in the global cluster state and can be included in snapshots by setting `include_global_state` to `true` when you [take the snapshot](../../../deploy-manage/tools/snapshot-and-restore/create-snapshots.md). When the snapshot is restored, all of the policies in the global state are restored and any local policies with the same names are overwritten.


I believe also any existing policies not in previous snapshot delete, no? It's not just update existing, it'll fully reset to previous.

I'm sorry but I'm really not sure. Maybe someone else can weigh in here?

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

stefnestor · 2025-06-27T00:02:31Z

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md

+1. On the **Index settings** page:
+    1. Configure ILM by specifying the [ILM settings](https://www.elastic.co/docs/api/doc/elasticsearch/configuration-reference/index-lifecycle-management-settings#_index_level_settings_2) to apply to the indices:
+        * `index.lifecycle.name` - The lifecycle policy to manage the created indices.
+        * `index.lifecycle.rollover_alias` - The index [alias](/manage-data/data-store/aliases.md) used for querying and managing the set of indices associated with a lifecycle policy that contains a rollover action.


At this point, could we consider collapsing the existing doc into this? This has screenshots but the previous does exist & is commonly linked by Support : https://www.elastic.co/docs/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover

@stefnestor I'm not sure about why we have both the "Tutorial: Automate rollover" and "Configure a lifecycle policy" that I'm working on here. There's overlap for sure. I suppose the former is intended only for data streams while the latter is more general, more basic. Combining these would be tricky I think but I'm open to ideas.

One note: In the recording I have of the review of these docs the number one reported problem is that users aren't sure what to put in the "index settings" tab when they create an index template for ILM. I was asked to make sure there's an example that they can follow, so this is how it would look. If we should change this or pull in some content from that rollover tutorial I'm happy to do that.

--

Anyhow, I think we'd need to keep this page and the rollover tutorial separate, or else this PR will become super complicated. Let me if you think that's okay, please.

manage-data/lifecycle/index-lifecycle-management/policy-apply.md

stefnestor · 2025-06-27T00:11:38Z

manage-data/lifecycle/index-lifecycle-management/policy-apply.md

+You can do this procedure in {{kib}} or using the {{es}} API.
+
+::::{warning}
+Do not manually apply a policy that uses the rollover action to an index which has not yet rolled over. Otherwise, the policy may not be carried forward when the rollover action creates a new index.


🙈 Sorry, why not as long as the template is setup to catch?

This is the note as it appears in the current docs here:

Please let me know if you think we should rephrase it somehow, or otherwise I can just remove it.

stefnestor · 2025-06-27T00:15:37Z

manage-data/lifecycle/index-lifecycle-management/policy-apply.md

+
+:::{tab-item} API
+:sync: api
+Use the [update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) to apply a lifecycle policy to an index.


Question on what you're going for vs what I'd expect this section to append:

when you apply an ILM policy it always sequentially runs from the top (so hot maybe rollover) so even if your index previously rolled over or something and you apply a new ilm policy or one for the first time, you're going back to the top of the queue. so if you may actually want the rollover_alias and/or indexing_complete mentioned before and/or do an ILM Move step to move to wherever in the new policy flow you want to be.

This content comes from the current docs: Apply a lifecycle policy manually section which is part of the "Configure a lifecycle policy" page. As Sam explained here, it doesn't seem to fit on that page because if I create an index with the right name, the ILM policy will be applied automatically.

I thought we could move this "Apply a lifecycle policy manually" content to be a separate page, describing the simplest case of someone adding an ILM policy to an index that doesn't already have one. We could also just remove the content if it's not useful. If we want something more complex like using indexing_complete, ILM Move, etc., I can document that but it would help me a lot if someone can demo the procedure since I'm a complete ILM amateur. :-)

stefnestor · 2025-06-27T00:18:03Z

manage-data/lifecycle/index-lifecycle-management/policy-view-status.md

+  - id: elasticsearch
+---
+
+# View the lifecycle status of an index [view-lifecycle-status]


If I may request there is no built-in way to view this across all indices (only filter index management to a phase/errors but no aggregate stats) so I had requested https://github.com/elastic/elasticsearch/pull/99612/files which Dev didn't want to document before rather than improving the product but Support still ends up sending out that JQ as our only option for aggrgate "how's it doing"

What about the ILM indicator in the health report? This should report any stagnating indices. Would that be sufficient?

Hypothetically yes, but in practice it's a good data point but an insufficient overview for Support due to bugs

(HealthAPI) Oscillating report for ILM Health elasticsearch#113553

(HealthAPI) flags 0 doc indices for overdue ILM Rollover elasticsearch#116894

manage-data/lifecycle/index-lifecycle-management/policy-view-status.md

stefnestor · 2025-06-27T00:21:44Z

👋 @kilfoyle hiya

I'm feeling a little frazzled at my EOD today so sorry if any of my comments below or in-line are miss-not-hit.

Apologies if it's out of scope of what you're intending, but I have comments which don't currently fit into the PR>files ➕ icon to just edit in-line & am not sure how to express

data tiers
- could/should header#4 show on the right-side TOC?
- Per this searchable snapshots can happen on hot+cold+frozen for quotes but only guaranteed frozen. there's various wrong logic from not understanding SS can happen on hot in the steps after these quotes
  
  The hot and warm tiers store regular indices, while the frozen tier stores searchable snapshots. However, the cold tier can store either regular indices or searchable snapshots.
  
  When data reaches the cold or frozen phases, it is automatically converted to a searchable snapshot by ILM.
- "Move shards off the nodes to be removed from the cluster."
  - they may want to push farther down not up (e.g. warm>cold not warm>hot, maybe noting frozen would be invalid option)
  - needs to ensure to call out they need to drain all shards (confirm with CAT Shards) not just update allocation (not sure how you want to handle node attrs saying "we'll do that during plan") but calling out because frequent support volume for total_shards_per_node and/or watermark on destination to block migrations off
- "If you do not intend to delete this data, you should manually restore each of the searchable snapshot indices to a regular index before disabling the data tier, by following these steps" fully mounted could just be ported across hot+cold, no? it doesn't have to be rehydrated.
- "Capture a comprehensive list of index and searchable snapshot names." I believe this assumes repo is found-snapshots but users can have custom repos so this isn't sufficiently valid
- "Remove the associated ILM policy (set it to null). If you want to apply a different ILM policy, follow the steps to Switch lifecycle policies." if you're modifying it you always set to null and then remove and then add the new ilm, you do not set <new-policy-name>
- "Optionally, specify the desired number of replica shards." > "index.lifecycle.rollover_alias": "<alias-for-rollover>": we usually don't re-add a rollover alias because we're not ingesting more into the index, instead we usually set "index.lifecycle.indexing_complete": true to bypass the potential new ilm policy's rollover action (which noop if NA)
- FWIW
  - the searchable snapshot rehydration section comes up if you need to fix data not just remove a data tier so it may potentially be worth a different section in the docs
  - see also KB > Resolve > 4) Clean-up > Searchable Snapshots
- these are not equal nor even generally related sub-sections. IMO maybe "remove a data tier" should be a par-header since it applies across the board saying something like "to remove a data tier we recommend draining off shards first to avoid data loss, ECE+ECK+ECH will enforce this"
  - if you disagree then FWIW "Elastic Cloud Hosted and Elastic Cloud Enterprise try to move all data from the nodes that are removed during plan changes." also applies to ECK software-side.

kilfoyle · 2025-06-27T17:14:59Z

Thanks @stefnestor. For your comments above about the Data Tiers docs I've opened an issue: #1956. I'd need guidance from you or others about exactly what doc updates to make based on the questions in that issue, but for now your feedback is stored safely. :-)

I've tried to address your other comments but I had a few questions. Please reply whenever you can (no rush :-) ).

github-actions · 2025-07-03T15:40:00Z

🔍 Preview links for changed docs

kilfoyle added 2 commits June 20, 2025 20:02

Rework 'Configure a lifecycle policy'

a288748

Expand 'Configure a lifecycle policy' docs

694c55e

github-actions bot deployed to docs-preview June 24, 2025 17:27 View deployment

kilfoyle changed the title ~~1572/ilm create template~~ Expand "Configure a lifecycle policy" docs Jun 24, 2025

kilfoyle marked this pull request as ready for review June 24, 2025 17:38

kilfoyle requested a review from a team as a code owner June 24, 2025 17:38

kilfoyle requested a review from a team June 24, 2025 17:39

typo fix

9ccf2be

github-actions bot deployed to docs-preview June 24, 2025 18:11 View deployment

samxbr reviewed Jun 25, 2025

View reviewed changes

gmarouli requested changes Jun 25, 2025

View reviewed changes

Updates for review feedback

a9ebfce

github-actions bot deployed to docs-preview June 25, 2025 21:57 View deployment

Small fix

d6c1431

github-actions bot deployed to docs-preview June 25, 2025 22:26 View deployment

stefnestor reviewed Jun 25, 2025

View reviewed changes

manage-data/lifecycle/index-lifecycle-management/policy-apply.md Outdated Show resolved Hide resolved

stefnestor reviewed Jun 25, 2025

View reviewed changes

manage-data/lifecycle/index-lifecycle-management/policy-apply.md Show resolved Hide resolved

stefnestor reviewed Jun 25, 2025

View reviewed changes

manage-data/lifecycle/index-lifecycle-management/policy-apply.md Show resolved Hide resolved

Apply Stef's suggestions and fix links

51d70a7

kilfoyle requested a review from a team as a code owner June 26, 2025 15:13

github-actions bot deployed to docs-preview June 26, 2025 15:14 View deployment

one more link fix

689b1f8

github-actions bot deployed to docs-preview June 26, 2025 15:17 View deployment

kilfoyle requested review from gmarouli, samxbr and stefnestor June 26, 2025 15:35

stefnestor reviewed Jun 26, 2025

View reviewed changes

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md Show resolved Hide resolved

stefnestor reviewed Jun 26, 2025

View reviewed changes

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md Show resolved Hide resolved

stefnestor reviewed Jun 26, 2025

View reviewed changes

manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md Show resolved Hide resolved

stefnestor reviewed Jun 27, 2025

View reviewed changes

manage-data/lifecycle/index-lifecycle-management/policy-apply.md Outdated Show resolved Hide resolved

stefnestor reviewed Jun 27, 2025

View reviewed changes

manage-data/lifecycle/index-lifecycle-management/policy-view-status.md Outdated Show resolved Hide resolved

Small fixes

a8a5c3f

github-actions bot deployed to docs-preview June 27, 2025 15:45 View deployment

Update 'View ILM status' example to System Int indices

365caef

github-actions bot deployed to docs-preview June 27, 2025 16:12 View deployment

kilfoyle mentioned this pull request Jun 27, 2025

Feedback on "Data tiers" docs #1956

Open

kilfoyle requested a review from stefnestor June 27, 2025 17:44

Add note about datastream lifecycle management at top

94494e2

github-actions bot deployed to docs-preview June 30, 2025 22:20 View deployment

Merge branch 'main' into 1572/ilm-create-template

7123e9b

github-actions bot deployed to docs-preview June 30, 2025 23:44 View deployment

Add note about built-in ILM policies

b9f9662

github-actions bot deployed to docs-preview July 3, 2025 15:38 View deployment

Fix list of topics on ILM main page; fix note about custom ILM

4f0e90b

github-actions bot deployed to docs-preview July 3, 2025 19:15 View deployment

Expand "Configure a lifecycle policy" docs #1906

Are you sure you want to change the base?

Expand "Configure a lifecycle policy" docs #1906

Uh oh!

Conversation

kilfoyle commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samxbr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gmarouli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kilfoyle Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kilfoyle commented Jun 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kilfoyle Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kilfoyle Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kilfoyle Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

kilfoyle commented Jun 24, 2025 •

edited

Loading

github-actions bot commented Jun 24, 2025 •

edited

Loading

kilfoyle Jun 25, 2025 •

edited

Loading

kilfoyle Jun 26, 2025 •

edited

Loading

kilfoyle Jun 27, 2025 •

edited

Loading

kilfoyle Jun 27, 2025 •

edited

Loading

kilfoyle commented Jun 27, 2025 •

edited

Loading