Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto migrate datanodes to real chunks v2 #457

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

l0kix2
Copy link
Collaborator

@l0kix2 l0kix2 commented Mar 5, 2025

Fix #396

The issue For successful master update to 24.2 it is required there are no active datanodes registered in cypress with use_imaginary_chunk_locations=%true, otherwise master couldn't be started.
To achieve it on existing cluster, one must set //sys/@config/node_tracker/enable_real_chunk_locations=%true and force datanodes with imaginary chunks location to unregister at master (by restarting them or somehow else). After that datanodes will re-register and have use_imaginary_chunk_locations=%false, but for successful master update only absence of active nodes with use_imaginary_chunk_locations=%true is required.

Previous implementation only set //sys/@config/node_tracker/enable_real_chunk_locations=%true and doesn't restart datanodes. It is removed in this PR.

New implementation adds new update flow status WaitingForImaginaryChunksAbsence, which is automatically added if master or data nodes are updated. For this status:

  1. yt client component does the following:
  • if enable_real_chunk_locations==%false — it sets it to true, otherwise proceed
  • if there >0 datanodes with status=online && use_imaginary_chunk_locations=%true — it sets DataNodesNeedPodsRemoval condition and wait until there are no nodes with status=online && use_imaginary_chunk_locations=%true. Otherwise or after condition is satisfied it is proceed to master update (or datanodes update if master excluded from update by updatePlan/updateSelector).
  1. if data node components detect DataNodesNeedPodsRemoval condition, it removes its pods (which ultimately leads to unregistering datanodes in master). The pods will be re-created (with possible image update if needed) at WaitingForPodsCreation stage later as usual.

@l0kix2 l0kix2 marked this pull request as ready for review March 6, 2025 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable enable_real_chunk_locations before update to 24.2
1 participant