Scalability issues in 2.13.x

[njohnstone2]

Overview

Upgrading linkerd from 2.12.5 to 2.13.5 results in the destination pods entering a crashloop on startup. The logs show that the destination container's gRPC server fails to start and there is a flood of proxy errors which eventually clear when running a smaller set of pods in the mesh.

The cluster consists of ~2300 meshed pods. In terms of workarounds to these errors two options have worked:

1. Rollback to v2.12.5
2. Remove ~100 pods from the mesh by setting linkerd.io/inject:"disabled" on a set of deployments

Another point of interest is that on 2.12.5 the linkerd-proxy container used on average ~200MiB memory. Since upgrading to 2.13.5 the linkerd-proxy container now consistently uses an average of 600MiB memory.

Attempted fixes

• increased replica count on all linkerd deployments from 3 -> 6. All destination pods still entered a crash loop.
• increased destination deployments liveness/readiness probes for all containers to 30s, to provide ample time for the containers to start up. All destination pods still entered a crash loop.

Versions

NAME                             	NAMESPACE                	STATUS  	CHART                              	APP VERSION
linkerd                          	linkerd                  	deployed	linkerd-control-plane-1.12.5       	stable-2.13.5
linkerd-crds                     	linkerd                  	deployed	linkerd-crds-1.6.1
linkerd-smi                      	linkerd-smi              	deployed	linkerd-smi-1.0.1                  	v0.2.1
linkerd-viz                      	linkerd-viz              	deployed	linkerd-viz-30.8.5                 	stable-2.13.5

Check command

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ can retrieve the control plane version
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
√ control plane proxies are up-to-date
√ control plane proxies and cli versions match

linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ can initialize the client
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
√ viz extension proxies are up-to-date
√ viz extension proxies and cli versions match
√ viz extension self-check

linkerd-smi
-----------
√ linkerd-smi extension Namespace exists
√ SMI extension service account exists
√ SMI extension pods are injected
√ SMI extension pods are running
√ SMI extension proxies are healthy

Status check results are √

Destination pod logs snippet (all containers)

stern linkerd-destination-944fcff6d-7x2rs
+ linkerd-destination-944fcff6d-7x2rs › destination
+ linkerd-destination-944fcff6d-7x2rs › sp-validator
+ linkerd-destination-944fcff6d-7x2rs › policy
+ linkerd-destination-944fcff6d-7x2rs › linkerd-proxy
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.548303s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.162.253:60848}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.162.253:60848: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs policy 2023-07-12T22:54:27.898891Z  INFO linkerd_policy_controller: Lease already exists, no need to create it
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.548361s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.162.253:60848}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.162.253:60848: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.550793s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.159.202:52870}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.159.202:52870: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.550842s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.166.165:57806}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.166.165:57806: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.550890s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.166.165:57806}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.166.165:57806: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.550917s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.166.165:57806}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.166.165:57806: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.550976s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.183.219:54752}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.183.219:54752: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551014s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.183.219:54752}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.183.219:54752: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs policy 2023-07-12T22:54:27.903144Z  INFO grpc{port=8090}: linkerd_policy_controller: policy gRPC server listening addr=0.0.0.0:8090
linkerd-destination-944fcff6d-7x2rs sp-validator time="2023-07-12T22:54:27Z" level=info msg="running version stable-2.13.5"
linkerd-destination-944fcff6d-7x2rs destination time="2023-07-12T22:54:27Z" level=info msg="running version stable-2.13.5"
linkerd-destination-944fcff6d-7x2rs sp-validator time="2023-07-12T22:54:27Z" level=info msg="starting admin server on :9997"
linkerd-destination-944fcff6d-7x2rs sp-validator time="2023-07-12T22:54:27Z" level=info msg="waiting for caches to sync"
linkerd-destination-944fcff6d-7x2rs sp-validator time="2023-07-12T22:54:27Z" level=info msg="caches synced"
linkerd-destination-944fcff6d-7x2rs sp-validator time="2023-07-12T22:54:27Z" level=info msg="listening at :8443"
linkerd-destination-944fcff6d-7x2rs destination time="2023-07-12T22:54:27Z" level=info msg="starting admin server on :9996"
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551045s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.190.38:50832}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.190.38:50832: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551072s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.190.38:50832}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.190.38:50832: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs destination time="2023-07-12T22:54:27Z" level=info msg="Using default opaque ports: map[25:{} 587:{} 3306:{} 4444:{} 5432:{} 6379:{} 9300:{} 11211:{}]"
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551130s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.190.38:50832}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.190.38:50832: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551168s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.162.152:43468}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.162.152:43468: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551200s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.162.152:43468}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.162.152:43468: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551235s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.162.152:43468}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.162.152:43468: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551265s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.162.152:43468}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.162.152:43468: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551332s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.190.242:36544}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.190.242:36544: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs destination time="2023-07-12T22:54:27Z" level=info msg="waiting for caches to sync"
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551363s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.190.242:36544}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.190.242:36544: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs destination time="2023-07-12T22:54:31Z" level=info msg="caches synced"
linkerd-destination-944fcff6d-7x2rs destination time="2023-07-12T22:54:31Z" level=info msg="waiting for caches to sync"
linkerd-destination-944fcff6d-7x2rs destination time="2023-07-12T22:54:31Z" level=info msg="caches synced"
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551389s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.190.242:36544}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.190.242:36544: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551412s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.171.242:40368}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.171.242:40368: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551431s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.171.242:40368}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.171.242:40368: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551450s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.171.242:40368}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.171.242:40368: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551478s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.171.242:40368}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.171.242:40368: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs destination time="2023-07-12T22:54:31Z" level=info msg="starting gRPC server on :8086"
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551500s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.176.4:42680}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.176.4:42680: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551632s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.176.4:42680}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.176.4:42680: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551667s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.176.4:42680}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.176.4:42680: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551700s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.160.169:53804}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.160.169:53804: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551731s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.160.169:53804}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.160.169:53804: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551762s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.160.169:53804}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.160.169:53804: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551797s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.186.103:56044}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.186.103:56044: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551832s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.186.103:56044}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.186.103:56044: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551863s]  INFO ThreadId(01) inbound:server{port=8090}:rescue{client.addr=10.202.186.103:56044}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.186.103:56044: server: 10.202.174.177:8090: server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable error.sources=[server 10.202.174.177:8090: service linkerd-policy.linkerd.svc.cluster.local:8090: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551892s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.161.100:46850}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.161.100:46850: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551919s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.161.100:46850}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.161.100:46850: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.551969s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.161.100:46850}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.161.100:46850: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.552000s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.161.100:46850}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.161.100:46850: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.552028s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.161.100:46850}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.161.100:46850: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.552075s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.161.100:46850}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.161.100:46850: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.552105s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.161.100:46850}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.161.100:46850: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.552133s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.161.100:46850}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.161.100:46850: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.552179s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.161.100:46850}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.161.100:46850: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]
linkerd-destination-944fcff6d-7x2rs linkerd-proxy [    38.552201s]  INFO ThreadId(01) inbound:server{port=8086}:rescue{client.addr=10.202.161.100:46850}: linkerd_app_core::errors::respond: gRPC request failed error=client 10.202.161.100:46850: server: 10.202.174.177:8086: server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable error.sources=[server 10.202.174.177:8086: service linkerd-dst-headless.linkerd.svc.cluster.local:8086: service unavailable, service unavailable]

Destination container only logs

k logs linkerd-destination-78c6bdcd5d-gtrt2 destination -p
time="2023-07-12T23:04:07Z" level=info msg="running version stable-2.13.5"
time="2023-07-12T23:04:07Z" level=info msg="starting admin server on :9996"
time="2023-07-12T23:04:07Z" level=info msg="Using default opaque ports: map[25:{} 587:{} 3306:{} 4444:{} 5432:{} 6379:{} 9300:{} 11211:{}]"
time="2023-07-12T23:04:08Z" level=info msg="waiting for caches to sync"
time="2023-07-12T23:04:12Z" level=info msg="caches synced"
time="2023-07-12T23:04:12Z" level=info msg="waiting for caches to sync"
time="2023-07-12T23:04:13Z" level=info msg="caches synced"
time="2023-07-12T23:04:13Z" level=info msg="starting gRPC server on :8086"
time="2023-07-12T23:05:20Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=info msg="shutting down gRPC server on :8086"
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=warning msg="unable to delete server_port_subscribers metric with labels map[name:jaeger namespace:jaeger port:5778]" addr=":8086" component=server
time="2023-07-12T23:05:31Z" level=error msg="failed to start destination admin server: http: Server closed"

[alpeb]

Can you please provide all the logs for that Destination pod, with the containers log levels set to debug?

[njohnstone2]

Hi @alpeb, really appreciate you taking a look at this. I've attached each of the container logs for the destination pod with the debug log level set.

NAME                                          READY   STATUS             RESTARTS         AGE
pod/linkerd-destination-7f65fdfd99-b9hzh      1/4     CrashLoopBackOff   40 (66s ago)     34m
pod/linkerd-destination-7f65fdfd99-cbp6x      1/4     CrashLoopBackOff   38 (2m3s ago)    34m
pod/linkerd-destination-7f65fdfd99-vdh6b      1/4     CrashLoopBackOff   38 (4m27s ago)   35m
pod/linkerd-identity-55d46785cb-7zq9n         2/2     Running            0                34m
pod/linkerd-identity-55d46785cb-hcstd         2/2     Running            0                35m
pod/linkerd-identity-55d46785cb-kwgn7         2/2     Running            0                34m
pod/linkerd-proxy-injector-6c9fd89756-p5lm7   2/2     Running            0                35m
pod/linkerd-proxy-injector-6c9fd89756-qt6lh   2/2     Running            0                34m
pod/linkerd-proxy-injector-6c9fd89756-zlxk6   2/2     Running            0                34m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/linkerd-destination      0/3     3            0           21d
deployment.apps/linkerd-identity         3/3     3            3           21d
deployment.apps/linkerd-proxy-injector   3/3     3            3           21d

NAME                              SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/linkerd-heartbeat   32 09 * * *   False     0        26m             21d

linkerd-destination-7f65fdfd99-vdh6b_proxy.log
linkerd-destination-7f65fdfd99-vdh6b_sp-validator.log
linkerd-destination-7f65fdfd99-vdh6b_destination.log
linkerd-destination-7f65fdfd99-vdh6b_policy.log
linkerd-destination-7f65fdfd99-vdh6b_linkerd-init.log

[alpeb]

I don't see any clear signs in the logs of an error that would cause the containers to crash. Could you share the kubectl describe output for those pods to see if there's more info about what component is causing the failure?

[njohnstone2]

here's a describe of one of the destination pods. I've attached files for all events so you can see the order of events, and the individual pod describes.

Name:             linkerd-destination-6c9b766689-qx9fk
Namespace:        linkerd
Priority:         0
Service Account:  linkerd-destination
Node:             ip-10-202-175-174.ap-southeast-2.compute.internal/10.202.175.174
Start Time:       Fri, 04 Aug 2023 19:11:32 +1000
Labels:           linkerd.io/control-plane-component=destination
                  linkerd.io/control-plane-ns=linkerd
                  linkerd.io/proxy-deployment=linkerd-destination
                  linkerd.io/workload-ns=linkerd
                  pod-template-hash=6c9b766689
Annotations:      cluster-autoscaler.kubernetes.io/safe-to-evict: true
                  config.linkerd.io/default-inbound-policy: all-unauthenticated
                  kubectl.kubernetes.io/restartedAt: 2023-07-18T22:04:23+10:00
                  linkerd.io/created-by: linkerd/helm stable-2.13.5
                  linkerd.io/helm-release-version: 13
                  linkerd.io/proxy-version: stable-2.13.5
                  linkerd.io/trust-root-sha256: 70710c0d4cd23d9ab43b4f2500ad027594ddcaf3cdc63ad4964e2c614da1e944
                  vector.dev/exclude: true
                  viz.linkerd.io/tap-enabled: true
Status:           Running
SeccompProfile:   RuntimeDefault
IP:               10.202.170.42
IPs:
  IP:           10.202.170.42
Controlled By:  ReplicaSet/linkerd-destination-6c9b766689
Init Containers:
  linkerd-init:
    Container ID:    containerd://10e20065e52683aa27656fa688c87c26ecdf57acca511587bf1d519ffd2bc2a6
    Image:           cr.l5d.io/linkerd/proxy-init:v2.2.1
    Image ID:        cr.l5d.io/linkerd/proxy-init@sha256:20349a461f9fb76fde33741a90f9de2a647068f506325ac5e0faf7b7bc2eea72
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Args:
      --incoming-proxy-port
      4143
      --outgoing-proxy-port
      4140
      --proxy-uid
      2102
      --inbound-ports-to-ignore
      4190,4191,4567,4568
      --outbound-ports-to-ignore
      443,6443
      --log-level
      debug
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 04 Aug 2023 19:11:35 +1000
      Finished:     Fri, 04 Aug 2023 19:11:35 +1000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  20Mi
    Requests:
      cpu:        100m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /run from linkerd-proxy-init-xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-269tr (ro)
Containers:
  linkerd-proxy:
    Container ID:    containerd://38fa30a9374d2944fb3cfac71bf5effdf7b8af6a1529ea7203328a1ef742c271
    Image:           cr.l5d.io/linkerd/proxy:stable-2.13.5
    Image ID:        cr.l5d.io/linkerd/proxy@sha256:0dfde3a6216a91720c42a3fd1ac014f7f2c4eca174a6a5680e55a570815b913b
    Ports:           4143/TCP, 4191/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    State:           Running
      Started:       Fri, 04 Aug 2023 19:11:35 +1000
    Ready:           True
    Restart Count:   0
    Requests:
      cpu:      50m
      memory:   20Mi
    Liveness:   http-get http://:4191/live delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:4191/ready delay=2s timeout=1s period=10s #success=1 #failure=3
    Environment:
      _pod_name:                                                linkerd-destination-6c9b766689-qx9fk (v1:metadata.name)
      _pod_ns:                                                  linkerd (v1:metadata.namespace)
      _pod_nodeName:                                             (v1:spec.nodeName)
      LINKERD2_PROXY_LOG:                                       warn,linkerd=info,trust_dns=error
      LINKERD2_PROXY_LOG_FORMAT:                                plain
      LINKERD2_PROXY_DESTINATION_SVC_ADDR:                      localhost.:8086
      LINKERD2_PROXY_DESTINATION_PROFILE_NETWORKS:              10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16,172.20.0.0/16
      LINKERD2_PROXY_POLICY_SVC_ADDR:                           localhost.:8090
      LINKERD2_PROXY_POLICY_WORKLOAD:                           $(_pod_ns):$(_pod_name)
      LINKERD2_PROXY_INBOUND_DEFAULT_POLICY:                    all-unauthenticated
      LINKERD2_PROXY_POLICY_CLUSTER_NETWORKS:                   10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16,172.20.0.0/16
      LINKERD2_PROXY_INBOUND_CONNECT_TIMEOUT:                   100ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_TIMEOUT:                  1000ms
      LINKERD2_PROXY_OUTBOUND_DISCOVERY_IDLE_TIMEOUT:           5s
      LINKERD2_PROXY_INBOUND_DISCOVERY_IDLE_TIMEOUT:            90s
      LINKERD2_PROXY_CONTROL_LISTEN_ADDR:                       0.0.0.0:4190
      LINKERD2_PROXY_ADMIN_LISTEN_ADDR:                         0.0.0.0:4191
      LINKERD2_PROXY_OUTBOUND_LISTEN_ADDR:                      127.0.0.1:4140
      LINKERD2_PROXY_INBOUND_LISTEN_ADDR:                       0.0.0.0:4143
      LINKERD2_PROXY_INBOUND_IPS:                                (v1:status.podIPs)
      LINKERD2_PROXY_INBOUND_PORTS:                             8086,8090,8443,9443,9990,9996,9997
      LINKERD2_PROXY_DESTINATION_PROFILE_SUFFIXES:              svc.cluster.local.
      LINKERD2_PROXY_INBOUND_ACCEPT_KEEPALIVE:                  10000ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_KEEPALIVE:                10000ms
      LINKERD2_PROXY_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION:  25,587,3306,4444,5432,6379,9300,11211
      LINKERD2_PROXY_DESTINATION_CONTEXT:                       {"ns":"$(_pod_ns)", "nodeName":"$(_pod_nodeName)"}
                                                                
      _pod_sa:                                                   (v1:spec.serviceAccountName)
      _l5d_ns:                                                  linkerd
      _l5d_trustdomain:                                         cluster.local
      LINKERD2_PROXY_IDENTITY_DIR:                              /var/run/linkerd/identity/end-entity
      LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS:                    <set to the key 'ca-bundle.crt' of config map 'linkerd-identity-trust-roots'>  Optional: false
      LINKERD2_PROXY_IDENTITY_TOKEN_FILE:                       /var/run/secrets/tokens/linkerd-identity-token
      LINKERD2_PROXY_IDENTITY_SVC_ADDR:                         linkerd-identity-headless.linkerd.svc.cluster.local.:8080
      LINKERD2_PROXY_IDENTITY_LOCAL_NAME:                       $(_pod_sa).$(_pod_ns).serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_IDENTITY_SVC_NAME:                         linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_DESTINATION_SVC_NAME:                      linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_POLICY_SVC_NAME:                           linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_TAP_SVC_NAME:                              tap.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
    Mounts:
      /var/run/linkerd/identity/end-entity from linkerd-identity-end-entity (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-269tr (ro)
      /var/run/secrets/tokens from linkerd-identity-token (rw)
  destination:
    Container ID:    containerd://aca182b3cf2c276f950a718ea05b20391307de58355f8bedaad8726bbd07ce51
    Image:           cr.l5d.io/linkerd/controller:stable-2.13.5
    Image ID:        cr.l5d.io/linkerd/controller@sha256:5b493b54f5205cc8c1a68e5e82882aae93ce9819f371829c6e49fd287c3c6fe1
    Ports:           8086/TCP, 9996/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      destination
      -addr=:8086
      -controller-namespace=linkerd
      -enable-h2-upgrade=true
      -log-level=debug
      -log-format=plain
      -enable-endpoint-slices=true
      -cluster-domain=cluster.local
      -identity-trust-domain=cluster.local
      -default-opaque-ports=25,587,3306,4444,5432,6379,9300,11211
      -enable-pprof=false
    State:          Running
      Started:      Fri, 04 Aug 2023 19:13:15 +1000
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 04 Aug 2023 19:12:26 +1000
      Finished:     Fri, 04 Aug 2023 19:13:15 +1000
    Ready:          False
    Restart Count:  2
    Limits:
      cpu:     2
      memory:  2Gi
    Requests:
      cpu:        100m
      memory:     50Mi
    Liveness:     http-get http://:9996/ping delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:9996/ready delay=0s timeout=1s period=10s #success=1 #failure=7
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-269tr (ro)
  sp-validator:
    Container ID:    containerd://ada4ddf6cfb40fa0050dd277893c7c0f33633e85ce02cf63d6259e54f948215d
    Image:           cr.l5d.io/linkerd/controller:stable-2.13.5
    Image ID:        cr.l5d.io/linkerd/controller@sha256:5b493b54f5205cc8c1a68e5e82882aae93ce9819f371829c6e49fd287c3c6fe1
    Ports:           8443/TCP, 9997/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      sp-validator
      -log-level=debug
      -log-format=plain
      -enable-pprof=false
    State:          Running
      Started:      Fri, 04 Aug 2023 19:13:17 +1000
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 04 Aug 2023 19:12:26 +1000
      Finished:     Fri, 04 Aug 2023 19:13:16 +1000
    Ready:          True
    Restart Count:  2
    Limits:
      cpu:     2
      memory:  2Gi
    Requests:
      cpu:        100m
      memory:     50Mi
    Liveness:     http-get http://:9997/ping delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get http://:9997/ready delay=0s timeout=1s period=10s #success=1 #failure=7
    Environment:  <none>
    Mounts:
      /var/run/linkerd/tls from sp-tls (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-269tr (ro)
  policy:
    Container ID:    containerd://79688752dcbd5c409a60e9aeba7958f005f01f498d728e731c2a30891b59155d
    Image:           cr.l5d.io/linkerd/policy-controller:stable-2.13.5
    Image ID:        cr.l5d.io/linkerd/policy-controller@sha256:1e3498d238fa7f08dd087ddeb6683224ae1213221e12b2be454a6e1d645d2522
    Ports:           8090/TCP, 9990/TCP, 9443/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      --admin-addr=0.0.0.0:9990
      --control-plane-namespace=linkerd
      --grpc-addr=0.0.0.0:8090
      --server-addr=0.0.0.0:9443
      --server-tls-key=/var/run/linkerd/tls/tls.key
      --server-tls-certs=/var/run/linkerd/tls/tls.crt
      --cluster-networks=10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16,172.20.0.0/16
      --identity-domain=cluster.local
      --cluster-domain=cluster.local
      --default-policy=all-unauthenticated
      --log-level=debug
      --log-format=plain
      --default-opaque-ports=25,587,3306,4444,5432,6379,9300,11211
      --probe-networks=0.0.0.0/0
    State:          Running
      Started:      Fri, 04 Aug 2023 19:13:17 +1000
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 04 Aug 2023 19:12:27 +1000
      Finished:     Fri, 04 Aug 2023 19:13:16 +1000
    Ready:          False
    Restart Count:  2
    Liveness:       http-get http://:admin-http/live delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:admin-http/ready delay=10s timeout=1s period=10s #success=1 #failure=7
    Environment:    <none>
    Mounts:
      /var/run/linkerd/tls from policy-tls (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-269tr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  sp-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  linkerd-sp-validator-k8s-tls
    Optional:    false
  policy-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  linkerd-policy-validator-k8s-tls
    Optional:    false
  linkerd-proxy-init-xtables-lock:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  linkerd-identity-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  linkerd-identity-end-entity:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  kube-api-access-269tr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 infrastructure=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  118s                 default-scheduler  Successfully assigned linkerd/linkerd-destination-6c9b766689-qx9fk to ip-10-202-175-174.ap-southeast-2.compute.internal
  Normal   Pulled     117s                 kubelet            Container image "cr.l5d.io/linkerd/proxy-init:v2.2.1" already present on machine
  Normal   Created    117s                 kubelet            Created container linkerd-init
  Normal   Started    115s                 kubelet            Started container linkerd-init
  Normal   Pulled     115s                 kubelet            Container image "cr.l5d.io/linkerd/proxy:stable-2.13.5" already present on machine
  Normal   Created    115s                 kubelet            Created container linkerd-proxy
  Normal   Started    115s                 kubelet            Started container linkerd-proxy
  Normal   Pulled     114s                 kubelet            Container image "cr.l5d.io/linkerd/controller:stable-2.13.5" already present on machine
  Normal   Created    114s                 kubelet            Created container destination
  Normal   Started    114s                 kubelet            Started container destination
  Normal   Pulled     114s                 kubelet            Container image "cr.l5d.io/linkerd/controller:stable-2.13.5" already present on machine
  Normal   Created    114s                 kubelet            Created container sp-validator
  Normal   Started    114s                 kubelet            Started container sp-validator
  Normal   Pulled     114s                 kubelet            Container image "cr.l5d.io/linkerd/policy-controller:stable-2.13.5" already present on machine
  Normal   Created    114s                 kubelet            Created container policy
  Normal   Started    113s                 kubelet            Started container policy
  Warning  Unhealthy  108s (x4 over 113s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy  87s                  kubelet            Readiness probe failed: Get "http://10.202.170.42:9996/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  87s                  kubelet            Liveness probe failed: Get "http://10.202.170.42:9996/ping": dial tcp 10.202.170.42:9996: i/o timeout (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  87s                  kubelet            Readiness probe failed: Get "http://10.202.170.42:9990/ready": dial tcp 10.202.170.42:9990: i/o timeout (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  87s                  kubelet            Liveness probe failed: Get "http://10.202.170.42:9997/ping": dial tcp 10.202.170.42:9997: i/o timeout (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  87s                  kubelet            Readiness probe failed: Get "http://10.202.170.42:9997/ready": dial tcp 10.202.170.42:9997: i/o timeout (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  87s                  kubelet            Liveness probe failed: Get "http://10.202.170.42:9990/live": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

linkerd-destination-6c9b766689-qx9fk_describe.txt
linkerd-proxy-injector-77b9f448b6-9snr4_describe.txt
linkerd-identity-55d46785cb-rhbsh_describe.txt
all_events.txt

[wmorgan]

Just following up here. We haven't been able to repro this issue yet. We've fixed something similar in #11135 but since there are no panics here it's not clear whether it's the same underlying issue. We've also fixed #11162 and #11055 but not obvious that these are related. We're going to 2.13.6 later this week and it would be great if you could try with that release; at a minimum it should reduce some of the log noise.

[njohnstone2]

Thanks for the update @wmorgan! #11055 sounds somewhat promising. I'll test the upgrade again once 2.13.6 is released and report back.

[njohnstone2]

I would like to report that we have been successfully running version v2.13.6 for over 12 hours and the issues with the destination pods have been entirely resolved.

Thank you to the Linkerd team for investigating the issue and getting the fix deployed.