Proxy containers are randomly not injected

[57vn]

I have a problem with Linkerd, proxy contaienrs are randomly not injected. We are using AKS, ArgoCD and Externalsecrets.
I did not found any relevant info in proxy injector logs or kubernetes events. What I noticed, that trustanchor secret age is the same, as the affected pod.
There are also some TLS handshake errors in the logs, and it seems k8s wasn't even trying to inject the container. At this point I have no clue in which direction should I start the investigation, how could I debug this.
Could you please help?

[alpeb]

Injection events are surfaced at the workload level. So if some given pod wasn't injected you might be able to see the reason issuing kubectl describe on its deployment resource. For more detailed information on the linkerd-proxy-injector pod, you should set the logging level to debug. Also check the linkerd-proxy-injector-webhook-config MutatingWebhookConfiguration to see what namespaceSelector and objectSelector it's using to determine which workloads to inject.

[xolott]

@alpeb
I have the same issue on multiple clusters. I have to check every time we roll out an update to make sure all the replicas are injected.

Checking the deployment, there are no reasons at all, only scale events, and after multiple pod deletions, a linkerd event:

  Normal  Injected           62s (x12 over 6h25m)  linkerd-proxy-injector  Linkerd sidecar proxy injected

I'm using:

• Linkerd
• Linkerd CRD
• Linkerd CNI

All are installed using helm on an AKS cluster (multiple, actually)

PS: I can't find the MutatingWebhookConfiguration. Is this a CRD? I couldn't find it

[alpeb]

PS: I can't find the MutatingWebhookConfiguration. Is this a CRD? I couldn't find it

It's a cluster-level resource. You can get it with kubectl get mutatingwebhookconfigurations linkerd-proxy-injector-webhook-config -oyaml

[xolott]

This is the value. I don't see any red flag. The behaviour in the same namespace with the same deployments is random, some times the linkerd proxy is injected, some times it doesn't, so I need to kill those pods until the linkerd proxy is injected

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    meta.helm.sh/release-name: linkerd-control-plane
    meta.helm.sh/release-namespace: linkerd
  creationTimestamp: "2023-05-20T02:47:06Z"
  generation: 2
  labels:
    app.kubernetes.io/managed-by: Helm
    linkerd.io/control-plane-component: proxy-injector
    linkerd.io/control-plane-ns: linkerd
  name: linkerd-proxy-injector-webhook-config
  resourceVersion: "3170"
  uid: some-uuid
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    caBundle: some-ca-bundle-value
    service:
      name: linkerd-proxy-injector
      namespace: linkerd
      path: /
      port: 443
  failurePolicy: Ignore
  matchPolicy: Equivalent
  name: linkerd-proxy-injector.linkerd.io
  namespaceSelector:
    matchExpressions:
    - key: config.linkerd.io/admission-webhooks
      operator: NotIn
      values:
      - disabled
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values:
      - kube-system
      - cert-manager
    - key: control-plane
      operator: DoesNotExist
  objectSelector:
    matchExpressions:
    - key: linkerd.io/control-plane-component
      operator: DoesNotExist
    - key: linkerd.io/cni-resource
      operator: DoesNotExist
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    - services
    scope: '*'
  sideEffects: None
  timeoutSeconds: 10

[alpeb]

You can set webhookFailurePolicy=Fail (Helm value) to fail pod creation if it couldn't be injected. If that results in too many failures, you can the linkerd-proxy-injector log (container proxy-injector) to see what's causing the issue.