🐛 Fix e2e test for dockermachinePool #11440

serngawy · 2024-11-18T22:51:25Z

What this PR does / why we need it:
This PR fix the missing return error for non exist machine and add more logs

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #11162

k8s-ci-robot · 2024-11-18T22:51:34Z

This PR is currently missing an area label, which is used to identify the modified component when generating release notes.

Area labels can be added by org members by writing /area ${COMPONENT} in a comment

Please see the labels list for possible areas.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2024-11-18T22:51:35Z

Hi @serngawy. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

sbueringer · 2024-11-19T07:42:59Z

/ok-to-test

/assign @AndiDog
/assign @fabriziopandini (for the parts affecting "regular non-MP CAPD"

sbueringer · 2024-11-19T07:47:07Z

test/infrastructure/docker/exp/internal/controllers/dockermachinepool_controller_phases.go

-	// Providers should iterate through their infrastructure instances and ensure that each instance has a corresponding InfraMachine.
-	for _, machine := range externalMachines {
-		if existingMachine, ok := dockerMachineMap[machine.Name()]; ok {
-			log.V(2).Info("Patching existing DockerMachine", "DockerMachine", klog.KObj(&existingMachine))


Are we losing this entire branch?

I don't follow how this change solves the problem

I moved the creation of the DockerMachine with the Container creation here to avoid the loop for creating DockerMachine CR based on the previously created container.
Based on my investigation ex logs blow, for some reason while the DockerMachinePool cleaning all dockerMachine a delay happen for deleting the container which make the DockerMachine get re-created here at same time the DockerMachinePool is been deleted

logs ex; the dockerMachine is failed to patch as it is deleted , next log is the dockerMachine created (I think dockerMachinePool go to delete same time the patch is happening to create missing dockerMachine) and then we stuck waiting for the machine get created but never happen as the dockermachinePool and machinePool are gone.

I1119 02:08:36.160435 1 dockermachinepool_controller_phases.go:147] "Patching existing DockerMachine" controller="dockermachinepool" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="DockerMachinePool" DockerMachinePool="self-hosted-jqke01/self-hosted-r77ad9-mp-0-8xsxc" namespace="self-hosted-jqke01" name="self-hosted-r77ad9-mp-0-8xsxc" reconcileID="904d6ace-3baa-48e1-b000-97ed711f9802" MachinePool="self-hosted-r77ad9-mp-0-s8xbz" Cluster="self-hosted-jqke01/self-hosted-r77ad9" DockerMachine="self-hosted-jqke01/worker-1tpa4a" E1119 02:08:36.181156 1 controller.go:316] "Reconciler error" err="failed to patch DockerMachine self-hosted-jqke01/worker-1tpa4a: dockermachines.infrastructure.cluster.x-k8s.io \"worker-1tpa4a\" not found" controller="dockermachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="DockerMachine" DockerMachine="self-hosted-jqke01/worker-1tpa4a" namespace="self-hosted-jqke01" name="worker-1tpa4a" reconcileID="f3f06dcd-f11a-4c3f-ae30-8bbef471ca00" E1119 02:08:36.183408 1 controller.go:316] "Reconciler error" err="failed to update DockerMachine \"self-hosted-jqke01/worker-1tpa4a\": failed to apply DockerMachine self-hosted-jqke01/worker-1tpa4a: Operation cannot be fulfilled on dockermachines.infrastructure.cluster.x-k8s.io \"worker-1tpa4a\": uid mismatch: the provided object specified uid fb561549-45d7-4a09-af27-6931b56bd020, and no existing object was found" controller="dockermachinepool" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="DockerMachinePool" DockerMachinePool="self-hosted-jqke01/self-hosted-r77ad9-mp-0-8xsxc" namespace="self-hosted-jqke01" name="self-hosted-r77ad9-mp-0-8xsxc" reconcileID="904d6ace-3baa-48e1-b000-97ed711f9802" I1119 02:08:36.186594 1 dockermachinepool_controller_phases.go:158] "Creating a new DockerMachine for Docker container" controller="dockermachinepool" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="DockerMachinePool" DockerMachinePool="self-hosted-jqke01/self-hosted-r77ad9-mp-0-8xsxc" namespace="self-hosted-jqke01" name="self-hosted-r77ad9-mp-0-8xsxc" reconcileID="9cc3622a-5a30-41c7-8b0a-6f0b212d0bdb" MachinePool="self-hosted-r77ad9-mp-0-s8xbz" Cluster="self-hosted-jqke01/self-hosted-r77ad9" container="worker-1tpa4a" I1119 02:08:36.253031 1 dockermachine_controller.go:104] "Waiting for Machine Controller to set OwnerRef on DockerMachine" controller="dockermachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="DockerMachine" DockerMachine="self-hosted-jqke01/worker-1tpa4a" namespace="self-hosted-jqke01" name="worker-1tpa4a" reconcileID="9799d2ef-c1a7-4924-b347-875155993995" DockerMachinePool="self-hosted-jqke01/self-hosted-r77ad9-mp-0-8xsxc"

for some reason while the DockerMachinePool cleaning all dockerMachine a delay happen for deleting the container which make the DockerMachine get re-created here at same time the DockerMachinePool is been deleted

have you considered to prevent creation of new machines in this func when the DockerMachinePool has a deletion timestamp? (it should probably be an if around L155-L161)

Well, the code logic shouldn't allow this to happen (it should go through the delete reconcile). In any case, tying the creation of dockerMachine after the container creation is better implementation to avoid such random execution to happen.

Well, the code logic shouldn't allow this to happen (it should go through the delete reconcile).

This is confusing me a little bit (and my lack of knowledge in MP doesn't help)
If I got it right, we are trying to fix a race condition that happens when deleting the MP.
But, to fix this error on deletion, we are shuffling machine creation from reconcileDockerMachines to reconcileDockerContainers, and both are not called on reconcile delete 🤔

Sorry for confusion, this changes fix the race condition. here during reconcile the dockerMachines if the container has no dockerMachine the dockerMachine get created, while the delete could happen at sametime. Before this changes; all containers get created first then all dockerMachines get created (in between race condition could happen). So this changes create every container with its dockerMachine at the sametime to avoid race condition. Hope that make it clear :)

Thanks for bearing with me with the explanation.
Ok, we are probably getting to to the point that confuses me.

When I review the PR, I start from the assumption that, if we consider a single MP, controller runtime ensures us that only one concurrent reconcile can happen anytime.

That's means, that if we are running the code here we are in reconcile normal, and reconcileDelete won't start until the operation completes or fails.

This rules out most of the race conditions. Also, assuming normal completes, the order of operations doesn't matter much, because at the end both dockerMachine and container should exist.

However, what might happen is that it takes some time for the cache to see the newly created dockerMachines, so it will seems it is missing when reconcileDelete starts, even if it it actually exists on the API server (but this requires a different fix).

Does this match with what are you observing?
To check this you can look e.g. at reconcile IDs in logs + look at errors when creating machines

sbueringer · 2024-11-19T07:48:15Z

test/infrastructure/docker/exp/internal/controllers/dockermachinepool_controller_phases.go

@@ -107,15 +113,13 @@ func createDockerContainer(ctx context.Context, name string, cluster *clusterv1.

 // reconcileDockerMachines creates and deletes DockerMachines to match the MachinePool's desired number of replicas and infrastructure spec.
 // It is responsible for
-// - Ensuring each Docker container has an associated DockerMachine by creating one if it doesn't already exist.
-// - Ensuring that deletion for Docker container happens by calling delete on the associated Machine so that the node is cordoned/drained and the infrastructure is cleaned up.


The delete call seems unchanged

I kept the delete cause if the container get deleted out side of capi. It has to reflect what is exist in the infrastructure. Also it handle the replica scale down use-case if something wrong happen.

But then we probably shouldn't drop the comment mentioning the deletion?

k8s-ci-robot · 2024-11-19T18:32:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from fabriziopandini. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fabriziopandini · 2024-11-21T21:41:26Z

test/infrastructure/docker/exp/internal/controllers/dockermachinepool_controller.go

-		return ctrl.Result{}, r.reconcileDelete(ctx, cluster, machinePool, dockerMachinePool)
+	if !dockerMachinePool.DeletionTimestamp.IsZero() {
+		// perform dockerMachinePool delete and requeue in 30s giving time for containers to be deleted.
+		return ctrl.Result{RequeueAfter: 30 * time.Second}, r.reconcileDelete(ctx, cluster, machinePool, dockerMachinePool)


CR will complain if we return a non zero result and an error, so we should split this into:

Suggested change

return ctrl.Result{RequeueAfter: 30 * time.Second}, r.reconcileDelete(ctx, cluster, machinePool, dockerMachinePool)

if err:= r.reconcileDelete(ctx, cluster, machinePool, dockerMachinePool); err != nil {

return ctrl.Result{}, err

}

return ctrl.Result{RequeueAfter: 30 * time.Second}, nil

Also, for my better understanding, could you kindly expand a little bit on why do you want to always reconcile every 30s instead of relying on watches/events or error backoff?

Okay, The e2e-Blocking job was failing as the docker cluster delete-wait-time is 3m here and it takes longer than this. Testing it locally, it takes up to 20min to reconcile after deleting the container to let the CR be deleted. So to avoid increasing the wait-time force reconcile and fail fast if there is an error.

Thanks for details. Considering we usually resync every 10m, 20m is something unusual.

If the 20m are due to race between creating machines and deleting them, and with this PR we are going to prevent this race to happen, I would prefer to avoid the requeue given that we have a proper fix in place.

If instead the 20m are due to some unknown reason, then we should probably try to root cause (requeing will probably make the test pass, but not fix the underlying issue)

There is no race condition, this how the code logic work check here in the reconcileDelete, We list the dockerMachines delete them and return (I believe to give time for containers & machines) then next reconcile if there is no dockerMachine we remove the dockerMachinePool finalizer. So as you said if we rysync in 10m and in the docker.yaml->wait-delete-cluster we wait 3m for delete so its normal to fail. Either we increase the docker.yaml->wait-delete-cluster to 15-20m (long time to fail) OR can we set the global resync to 1m during the e2e test ?

fabriziopandini · 2024-11-21T22:05:36Z

test/infrastructure/docker/exp/internal/controllers/dockermachinepool_controller_phases.go

-	// Providers should iterate through their infrastructure instances and ensure that each instance has a corresponding InfraMachine.
-	for _, machine := range externalMachines {
-		if existingMachine, ok := dockerMachineMap[machine.Name()]; ok {
-			log.V(2).Info("Patching existing DockerMachine", "DockerMachine", klog.KObj(&existingMachine))


for some reason while the DockerMachinePool cleaning all dockerMachine a delay happen for deleting the container which make the DockerMachine get re-created here at same time the DockerMachinePool is been deleted

have you considered to prevent creation of new machines in this func when the DockerMachinePool has a deletion timestamp? (it should probably be an if around L155-L161)

Signed-off-by: serngawy <[email protected]>

serngawy · 2024-12-02T15:23:40Z

@fabriziopandini @sbueringer , Do you have any other comments ? can we merge this ?

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-area PR is missing an area label labels Nov 18, 2024

k8s-ci-robot requested review from jackfrancis and JoelSpeed November 18, 2024 22:51

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 18, 2024

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 18, 2024

serngawy force-pushed the e2efix branch from 0213ff1 to 49502cd Compare November 19, 2024 03:31

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 19, 2024

serngawy changed the title ~~🐛 (WIP) Fix e2e test for dockermachinePool~~ 🐛 Fix e2e test for dockermachinePool Nov 19, 2024

k8s-ci-robot assigned AndiDog and fabriziopandini Nov 19, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 19, 2024

sbueringer reviewed Nov 19, 2024

View reviewed changes

serngawy force-pushed the e2efix branch from 49502cd to fa64f09 Compare November 19, 2024 18:31

serngawy force-pushed the e2efix branch from fa64f09 to 75b147f Compare November 20, 2024 15:09

fabriziopandini reviewed Nov 21, 2024

View reviewed changes

Fix e2e test for dockermachinePool

a1865cd

Signed-off-by: serngawy <[email protected]>

serngawy force-pushed the e2efix branch from 75b147f to a1865cd Compare November 22, 2024 18:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Fix e2e test for dockermachinePool #11440

🐛 Fix e2e test for dockermachinePool #11440

serngawy commented Nov 18, 2024 •

edited

Loading

k8s-ci-robot commented Nov 18, 2024

k8s-ci-robot commented Nov 18, 2024

sbueringer commented Nov 19, 2024 •

edited

Loading

sbueringer Nov 19, 2024

serngawy Nov 19, 2024 •

edited

Loading

fabriziopandini Nov 21, 2024 •

edited

Loading

serngawy Nov 22, 2024

fabriziopandini Nov 25, 2024 •

edited

Loading

serngawy Nov 25, 2024

fabriziopandini Dec 2, 2024 •

edited

Loading

sbueringer Nov 19, 2024

serngawy Nov 19, 2024

sbueringer Nov 19, 2024

k8s-ci-robot commented Nov 19, 2024

fabriziopandini Nov 21, 2024

serngawy Nov 22, 2024 •

edited

Loading

fabriziopandini Nov 25, 2024

serngawy Nov 25, 2024

fabriziopandini Nov 21, 2024 •

edited

Loading

serngawy commented Dec 2, 2024

-		return ctrl.Result{RequeueAfter: 30 * time.Second}, r.reconcileDelete(ctx, cluster, machinePool, dockerMachinePool)
+		if err:= r.reconcileDelete(ctx, cluster, machinePool, dockerMachinePool); err != nil {
+		     return ctrl.Result{}, err
+		}
+		return ctrl.Result{RequeueAfter: 30 * time.Second}, nil

🐛 Fix e2e test for dockermachinePool #11440

Are you sure you want to change the base?

🐛 Fix e2e test for dockermachinePool #11440

Conversation

serngawy commented Nov 18, 2024 • edited Loading

k8s-ci-robot commented Nov 18, 2024

k8s-ci-robot commented Nov 18, 2024

sbueringer commented Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

serngawy Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

fabriziopandini Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabriziopandini Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabriziopandini Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 19, 2024

Choose a reason for hiding this comment

serngawy Nov 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabriziopandini Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

serngawy commented Dec 2, 2024

serngawy commented Nov 18, 2024 •

edited

Loading

sbueringer commented Nov 19, 2024 •

edited

Loading

serngawy Nov 19, 2024 •

edited

Loading

fabriziopandini Nov 21, 2024 •

edited

Loading

fabriziopandini Nov 25, 2024 •

edited

Loading

fabriziopandini Dec 2, 2024 •

edited

Loading

serngawy Nov 22, 2024 •

edited

Loading

fabriziopandini Nov 21, 2024 •

edited

Loading