Cannot get CockroachDB-Cluster to work when pods are meshed

[sebschlue]

Hello :)

I deploy linkerd 2.13.5 in HA-mode

linkerd install --crds ...
linkerd install --ha ...
linkerd viz install --ha ...

Then I set a namespace to policy "all-authenticated":

kubectl annotate namespace development config.linkerd.io/default-inbound-policy=all-authenticated

Then I deploy CockroachDB-Cluster via helm chart with default values:

helm upgrade --install cockroachdb cockroachdb/cockroachdb --version 11.1.3 --namespace development

CockroachDB-Cluster works fine afterwards.

Then I try to perform linkerd injection:

kubectl -n development get sts cockroachdb -o yaml | linkerd inject - | kubectl apply -f -

Rollout process get stuck because first restarted pod does not become ready, so I manually restart the other pods.

But even after all pods have been restarted and contain linkerd init- and sidecar-containers, CockroachDB-Cluster does not work anymore - nodes cannot reach each other:

E230801 17:25:57.509642 927 2@rpc/context.go:2404 ⋮ [T1,n1,rnode=2,raddr=‹cockroachdb-2.cockroachdb.development.svc.cluster.local:26257›,class=default,rpc] 108  unable to connect (is the peer up and reachable?): initial connection heartbeat failed: grpc: ‹connection error: desc = "transport: authentication handshake failed: EOF"› [code 14/Unavailable]

linkerd-proxy sidecar container does not log anything related to port 26257. (It only logs unauthoized connection attempts from Prometheus to port 8080, which is true, but unrelated to the cockroachdb-cluster-communication issue.)

Also

linkerd viz tap -n development sts/cockroachdb

does not show anything related to port 26257.

I don't know how to further debug this issue.

I have tried to set annotation

config.linkerd.io/opaque-ports: 26257,8080

but this did not change anything.

k8s cluster is EKS cluster 1.26.6

Can anybody give me a hint on how to further debug this issue?

Thanks in advance :)