Linkerd proxy injector failed readiness probe

[chbalbas]

Hi everyone,

$ k logs linkerd-proxy-injector-84c8cbc6d4-jwj6b -c proxy-injector
time="2023-05-12T07:45:56Z" level=info msg="running version stable-2.11.5"
time="2023-05-12T07:45:56Z" level=info msg="waiting for caches to sync"
time="2023-05-12T07:45:56Z" level=info msg="listening at :8443

$ k describe pod linkerd-proxy-injector-84c8cbc6d4-jwj6b
(...)
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  32m                   default-scheduler  Successfully assigned linkerd/linkerd-proxy-injector-84c8cbc6d4-jwj6b to aks-poolcipdev01-37231988-vmss00000i
  Normal   Pulling    32m                   kubelet            Pulling image "****/linkerd/proxy-init:v1.5.3"
  Normal   Pulled     32m                   kubelet            Successfully pulled image "****/linkerd/proxy-init:v1.5.3" in 463.334236ms
  Normal   Created    32m                   kubelet            Created container linkerd-init
  Normal   Started    32m                   kubelet            Started container linkerd-init
  Normal   Started    32m                   kubelet            Started container linkerd-proxy
  Normal   Pulled     32m                   kubelet            Successfully pulled image "****/linkerd/proxy:stable-2.11.5" in 626.803708ms
  Normal   Created    32m                   kubelet            Created container linkerd-proxy
  Normal   Pulling    32m                   kubelet            Pulling image "****/linkerd/proxy:stable-2.11.5"
  Normal   Pulled     32m                   kubelet            Successfully pulled image "****/linkerd/controller:stable-2.11.5" in 139.596995ms
  Normal   Created    32m                   kubelet            Created container proxy-injector
  Normal   Started    32m                   kubelet            Started container proxy-injector
  Warning  Unhealthy  31m (x3 over 31m)     kubelet            Liveness probe failed: HTTP probe failed with statuscode: 502
  Normal   Killing    31m                   kubelet            Container proxy-injector failed liveness probe, will be restarted
  Normal   Pulling    27m (x6 over 32m)     kubelet            Pulling image "****/linkerd/controller:stable-2.11.5"
  Warning  Unhealthy  6m57s (x99 over 32m)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 502
  Warning  BackOff    118s (x90 over 26m)   kubelet            Back-off restarting failed container

$ k get all
NAME                                          READY   STATUS             RESTARTS         AGE
pod/linkerd-destination-964665745-fwbvz       4/4     Running            0                49m
pod/linkerd-identity-5445697fd9-dfmbf         2/2     Running            0                49m
pod/linkerd-proxy-injector-84c8cbc6d4-jwj6b   1/2     CrashLoopBackOff   13 (3m54s ago)   37m

NAME                                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/linkerd-dst                 ClusterIP   172.17.0.68    <none>        8086/TCP   49m
service/linkerd-dst-headless        ClusterIP   None           <none>        8086/TCP   49m
service/linkerd-identity            ClusterIP   172.17.2.54    <none>        8080/TCP   49m
service/linkerd-identity-headless   ClusterIP   None           <none>        8080/TCP   49m
service/linkerd-policy              ClusterIP   None           <none>        8090/TCP   49m
service/linkerd-policy-validator    ClusterIP   172.17.2.180   <none>        443/TCP    49m
service/linkerd-proxy-injector      ClusterIP   172.17.2.139   <none>        443/TCP    49m
service/linkerd-sp-validator        ClusterIP   172.17.2.177   <none>        443/TCP    49m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/linkerd-destination      1/1     1            1           49m
deployment.apps/linkerd-identity         1/1     1            1           49m
deployment.apps/linkerd-proxy-injector   0/1     1            0           49m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/linkerd-destination-964665745       1         1         1       49m
replicaset.apps/linkerd-identity-5445697fd9         1         1         1       49m
replicaset.apps/linkerd-proxy-injector-84c8cbc6d4   1         1         0       49m

Any ideas? Thanks in advance 😄

[alpeb]

Please run linkerd check to run some diagnostics. Also try setting the log level for the injector at debug and retrieve the logs again. OTOH, 2.11 is pretty old an unsupported; can you try with the latest 2.13.3?

[chbalbas]

Hello,

Thanks for helping.

$ linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
\ pod/linkerd-proxy-injector-84c8cbc6d4-jwj6b container proxy-injector is not ready

I have deployed it with helm, the chart latest version is 2.11.5, I do not know if it is possible deploy the 2.13.3 version with the chart 2.11.5. Do you know it?

$ helm ls 
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION  
linkerd2        linkerd         1               2023-05-12 07:02:21.422526051 +0000 UTC deployed        linkerd2-2.11.5 stable-2.11.5

[alpeb]

Oh we're using different charts as of 2.12. The docs here explain how to get ahold of the new chart repos and how to install. If you absolutely require upgrading from 2.11 (as opposed to reinstalling from scratch, which would be much easier) there are some instructions here.

[chbalbas]

Thanks for helping! 😀

Fine, I have reinstalled linkerd with the new chart, but now i have another problem with linkerd-destination pod.

$ helm ls 
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                           APP VERSION  
linkerd-control-plane   linkerd         1               2023-05-12 11:04:44.154343985 +0000 UTC deployed        linkerd-control-plane-1.12.3    stable-2.13.3

$ k get pods
NAME                                      READY   STATUS             RESTARTS         AGE
linkerd-destination-5fdf999774-t99lm      2/4     CrashLoopBackOff   10 (2m54s ago)   28m
linkerd-identity-7cb68859df-dsnmf         2/2     Running            0                30m
linkerd-proxy-injector-5879f4f6cb-bxhwl   2/2     Running            0                30m

$ k describe linkerd-destination-5fdf999774-t99lm
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  23m                    default-scheduler  Successfully assigned linkerd/linkerd-destination-5fdf999774-t99lm to aks-poolcipdev01-37231988-vmss00000j
  Normal   Pulling    23m                    kubelet            Pulling image "acrweucipmgmt01.azurecr.io/linkerd/proxy-init:v2.2.1"
  Normal   Pulled     23m                    kubelet            Successfully pulled image "acrweucipmgmt01.azurecr.io/linkerd/proxy-init:v2.2.1" in 188.360057ms
  Normal   Created    23m                    kubelet            Created container linkerd-init
  Normal   Started    23m                    kubelet            Started container linkerd-init
  Normal   Pulling    23m                    kubelet            Pulling image "acrweucipmgmt01.azurecr.io/linkerd/proxy:stable-2.13.3"
  Normal   Pulled     23m                    kubelet            Successfully pulled image "acrweucipmgmt01.azurecr.io/linkerd/proxy:stable-2.13.3" in 169.288539ms
  Normal   Created    23m                    kubelet            Created container linkerd-proxy
  Normal   Started    23m                    kubelet            Started container linkerd-proxy
  Normal   Pulled     23m                    kubelet            Successfully pulled image "acrweucipmgmt01.azurecr.io/linkerd/policy-controller:stable-2.13.3" in 222.287745ms
  Normal   Started    23m                    kubelet            Started container sp-validator
  Normal   Started    23m                    kubelet            Started container policy
  Normal   Created    23m                    kubelet            Created container policy
  Normal   Pulling    23m                    kubelet            Pulling image "acrweucipmgmt01.azurecr.io/linkerd/controller:stable-2.13.3"
  Normal   Pulled     23m                    kubelet            Successfully pulled image "acrweucipmgmt01.azurecr.io/linkerd/controller:stable-2.13.3" in 133.949234ms
  Normal   Created    23m                    kubelet            Created container sp-validator
  Normal   Pulled     23m                    kubelet            Successfully pulled image "acrweucipmgmt01.azurecr.io/linkerd/controller:stable-2.13.3" in 132.496018ms
  Normal   Pulling    23m                    kubelet            Pulling image "acrweucipmgmt01.azurecr.io/linkerd/policy-controller:stable-2.13.3"
  Normal   Pulling    23m (x2 over 23m)      kubelet            Pulling image "acrweucipmgmt01.azurecr.io/linkerd/controller:stable-2.13.3"
  Normal   Started    23m (x2 over 23m)      kubelet            Started container destination
  Normal   Created    23m (x2 over 23m)      kubelet            Created container destination
  Normal   Pulled     23m                    kubelet            Successfully pulled image "acrweucipmgmt01.azurecr.io/linkerd/controller:stable-2.13.3" in 107.536931ms
  Warning  Unhealthy  8m49s (x173 over 23m)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  BackOff    3m56s (x106 over 23m)  kubelet            Back-off restarting failed container

Container logs

linkerd-destination - container: policy

2023-05-12T12:04:54.535981Z  INFO servers: kubert::errors: stream failed error=failed to perform initial object list: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })
2023-05-12T12:04:54.623485Z  WARN serverauthorizations: kube_client::client: Unsuccessful data error parse: 404 page not found

2023-05-12T12:04:54.623525Z  INFO serverauthorizations: kubert::errors: stream failed error=failed to perform initial object list: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })
2023-05-12T12:04:54.623588Z  WARN authorizationpolicies: kube_client::client: Unsuccessful data error parse: 404 page not found

2023-05-12T12:04:54.623599Z  INFO authorizationpolicies: kubert::errors: stream failed error=failed to perform initial object list: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })
2023-05-12T12:04:54.623636Z  WARN networkauthentications: kube_client::client: Unsuccessful data error parse: 404 page not found

2023-05-12T12:04:54.623646Z  INFO networkauthentications: kubert::errors: stream failed error=failed to perform initial object list: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })
2023-05-12T12:04:54.623872Z  WARN meshtlsauthentications: kube_client::client: Unsuccessful data error parse: 404 page not found

2023-05-12T12:04:54.623895Z  INFO meshtlsauthentications: kubert::errors: stream failed error=failed to perform initial object list: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })
2023-05-12T12:04:59.539390Z  WARN servers: kube_client::client: Unsuccessful data error parse: 404 page not found

2023-05-12T12:04:59.539423Z  INFO servers: kubert::errors: stream failed error=failed to perform initial object list: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })
2023-05-12T12:04:59.627342Z  WARN authorizationpolicies: kube_client::client: Unsuccessful data error parse: 404 page not found

2023-05-12T12:04:59.627370Z  INFO authorizationpolicies: kubert::errors: stream failed error=failed to perform initial object list: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })
2023-05-12T12:04:59.627498Z  WARN networkauthentications: kube_client::client: Unsuccessful data error parse: 404 page not found

2023-05-12T12:04:59.627510Z  INFO networkauthentications: kubert::errors: stream failed error=failed to perform initial object list: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })
2023-05-12T12:04:59.627555Z  WARN meshtlsauthentications: kube_client::client: Unsuccessful data error parse: 404 page not found

2023-05-12T12:04:59.627562Z  INFO meshtlsauthentications: kubert::errors: stream failed error=failed to perform initial object list: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })
2023-05-12T12:04:59.627597Z  WARN serverauthorizations: kube_client::client: Unsuccessful data error parse: 404 page not found

2023-05-12T12:04:59.627609Z  INFO serverauthorizations: kubert::errors: stream failed error=failed to perform initial object list: ApiError: "404 page not found\n": Failed to parse error data (ErrorResponse { status: "404 Not Found", message: "\"404 page not found\\n\"", reason: "Failed to parse error data", code: 404 })

linkerd-destination - container: sp-validator

$ k logs linkerd-destination-5fdf999774-t99lm -c sp-validator
time="2023-05-12T11:34:47Z" level=info msg="running version stable-2.13.3"
time="2023-05-12T11:34:47Z" level=info msg="starting admin server on :9997"
time="2023-05-12T11:34:47Z" level=info msg="waiting for caches to sync"
time="2023-05-12T11:34:47Z" level=info msg="caches synced"
time="2023-05-12T11:34:47Z" level=info msg="listening at :8443"

linkerd-destination - container: destination

$ k logs linkerd-destination-5fdf999774-t99lm -c destination
time="2023-05-12T12:05:49Z" level=info msg="running version stable-2.13.3"
time="2023-05-12T12:05:49Z" level=info msg="starting admin server on :9996"
time="2023-05-12T12:05:49Z" level=info msg="Using default opaque ports: map[25:{} 587:{} 3306:{} 4444:{} 5432:{} 6379:{} 9300:{} 11211:{}]"
time="2023-05-12T12:05:49Z" level=fatal msg="Failed to initialize K8s API: the server could not find the requested resource"

linkerd-destination - container: linkerd-proxy

[     0.002115s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.002879s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.002899s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.002902s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.002904s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.002906s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.002909s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.002911s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
[     0.003317s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     0.110173s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     0.198367s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[     0.319864s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     0.738690s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[     1.239583s]  WARN ThreadId(01) watch{port=9997}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[  1256.178889s]  INFO ThreadId(01) inbound:server{port=9996}:rescue{client.addr=10.16.128.11:60186}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]

[alpeb]

Did you install the linkerd-crds chart?

[chbalbas]

I was installing crds via linkerd cli util, I have installed the chart and now it works

[chbalbas]

Thanks! 😄