Double spec reconciliation of etcd controller upon adding druid.gardener.cloud/operation="reconcile" annotation to Etcd CR

[anveshreddy18]

How to categorize this issue?

/area control-plane
/kind bug

What happened:

As the title suggests, I have seen double spec reconciliation of etcd controller upon adding druid.gardener.cloud/operation="reconcile" annotation to Etcd CR and traced it back to the PR #996 which moved the removing of operation annotation functionality from reconcileSpec flow earlier to after the reconcileStatus now. Now in the next reconciliation cycle when we check whether reconciliation annotation is still present here, it evaluates to true signalling that the informer cache is not updated for the patch request that was made. Due to this staleness, the shouldReconcileSpec method evaluates to true so the reconciliation happens again. The probable reason for why we didn't see this earlier was because of the time delay between when we attempt to remove the annotation and when we check it back later, as there was status reconciliation in between which induced the necessary delay.

What you expected to happen:

Spec Reconciliation of an Etcd resource should only happen once after the druid.gardener.cloud/operation="reconcile" is added.

How to reproduce it (as minimally and precisely as possible):

• Spin up a kind cluster with make kind-up
• Install CRDs and deploy druid with make deploy ( with CLI flag enable-spec-auto-reconcile is set to false by setting enableEtcdSpecAutoReconcile set to the default value of false here)
• Install a Etcd CR by doing kubectl apply -f examples/etcd/druid_v1alpha1_etcd.yaml
• Observe the logs of etcd-druid while adding a reconcile annotation like kubectl annotate etcd <etcd-name> druid.gardener.cloud/operation="reconcile"

Anything else we need to know?:

Environment:

• etcd-druid version: v0.27.0, can just use master to replicate the issue.

[shreyas-s-rao]

Might be related to #898