GRPC load balance is not working

[spgyip]

I have followed getting-started and installed linkerd in my k8s cluster. Also the emojivoto project is working as fine as expected.

But there is load balance problem with my own programs, these are simple grpc applications which greeter-client make grpc call to the greeter-server . The greeter-client make connection to greeter-server and makes calls in a long-run loop. I have injected both client side and server side just as the gettint-started guiding. However, I found the requests is not balanced among the 2 greeter-server pods, but only 1 pod gets requests.

I have no idea why this is happening, and how to debug.

• The linkerd dashboard observing

You can see only one pod of greeter-server is processing request.

• k8s version

[root@k8m ~/deployment/greeter]# kubectl version --short
Client Version: v1.24.0
Kustomize Version: v4.5.4
Server Version: v1.24.0

• linker check -o short

[root@k8m ~/deployment/greeter]# linkerd check -o short
linkerd-version
---------------
‼ cli is up-to-date
    is running version 2.13.4 but the latest stable version is 2.13.5
    see https://linkerd.io/2.13/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    is running version 2.13.4 but the latest stable version is 2.13.5
    see https://linkerd.io/2.13/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
        * linkerd-destination-58cf68b74c-5wkgh (stable-2.13.4)
        * linkerd-identity-7544cc5c8d-gt5n5 (stable-2.13.4)
        * linkerd-proxy-injector-776fcc448-pnzzb (stable-2.13.4)
    see https://linkerd.io/2.13/checks/#l5d-cp-proxy-version for hints

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
        * metrics-api-6c494f97c6-w5wvx (stable-2.13.4)
        * prometheus-6bdfc9c44-tp2zz (stable-2.13.4)
        * tap-5484df5888-wznxk (stable-2.13.4)
        * tap-injector-5c4984f8cf-mcjkh (stable-2.13.4)
        * web-847d864fb6-5cxnt (stable-2.13.4)
    see https://linkerd.io/2.13/checks/#l5d-viz-proxy-cp-version for hints

Status check results are √

• greeter yaml

[root@k8m ~/deployment/greeter]# cat greeter-1-0-1.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: greeter-server-ds
  labels:
    app: greeter-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: greeter-server
  template:
    metadata:
      labels:
        app: greeter-server
    spec:
      containers:
        - name: greeter-server
          image: docker-images.music.woa.com/cargo/greeter_server:v1.0.2
          command: ["./greeter_server"]
          args: ["-l", ":80", "-promhttp", ":8080"]
          ports:
            - containerPort: 80
              protocol: TCP
            - containerPort: 8080
              protocol: TCP
          resources:
            requests:
              cpu: 0.5
            limits:
              cpu: 2
---
apiVersion: v1
kind: Service
metadata:
  name: greeter-server-svc
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
  labels:
    microservice.io/name: example.helloworld.greeter
spec:
  selector:
    app: greeter-server
  type: ClusterIP
  clusterIP: None
  ports:
  - name: rpc
    protocol: TCP
    port: 80
    targetPort: 80
  - name: promhttp
    protocol: TCP
    port: 8080
    targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: greeter-client-ds
  labels:
    app: greeter-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: greeter-client
  template:
    metadata:
      labels:
        app: greeter-client
    spec:
      containers:
        - name: greeter-client
          image: docker-images.music.woa.com/cargo/greeter_client:latest
          command: ["./greeter_client"]
          args: ["-addr", "greeter-server-svc:80", "-nzeros", "15"]

• greeter-client pod desc

[root@k8m ~/deployment/greeter]# kubectl describe pod/greeter-client-ds-7b8765597b-frpwd
Name:         greeter-client-ds-7b8765597b-frpwd
Namespace:    default
Priority:     0
Node:         k8w0/9.135.231.137
Start Time:   Tue, 27 Jun 2023 17:43:15 +0800
Labels:       app=greeter-client
              linkerd.io/control-plane-ns=linkerd
              linkerd.io/proxy-deployment=greeter-client-ds
              linkerd.io/workload-ns=default
              pod-template-hash=7b8765597b
Annotations:  container.seccomp.security.alpha.kubernetes.io/linkerd-init: runtime/default
              container.seccomp.security.alpha.kubernetes.io/linkerd-proxy: runtime/default
              kubectl.kubernetes.io/restartedAt: 2023-06-27T17:43:15+08:00
              linkerd.io/created-by: linkerd/proxy-injector stable-2.13.4
              linkerd.io/inject: enabled
              linkerd.io/proxy-version: stable-2.13.4
              linkerd.io/trust-root-sha256: 2e1b0607b42e1a804a62dd02770786302c0e644b51cb2ffee40eb661d913e748
              viz.linkerd.io/tap-enabled: true
Status:       Running
IP:           10.244.1.192
IPs:
  IP:           10.244.1.192
Controlled By:  ReplicaSet/greeter-client-ds-7b8765597b
Init Containers:
  linkerd-init:
    Container ID:  containerd://abdb665eee2762e46bf1b1b956146114639fba0b2f481b21d88b4f806ed016b6
    Image:         cr.l5d.io/linkerd/proxy-init:v2.2.1
    Image ID:      cr.l5d.io/linkerd/proxy-init@sha256:20349a461f9fb76fde33741a90f9de2a647068f506325ac5e0faf7b7bc2eea72
    Port:          <none>
    Host Port:     <none>
    Args:
      --incoming-proxy-port
      4143
      --outgoing-proxy-port
      4140
      --proxy-uid
      2102
      --inbound-ports-to-ignore
      4190,4191,4567,4568
      --outbound-ports-to-ignore
      4567,4568
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 27 Jun 2023 17:43:17 +0800
      Finished:     Tue, 27 Jun 2023 17:43:17 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  20Mi
    Requests:
      cpu:        100m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /run from linkerd-proxy-init-xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x7bc9 (ro)
Containers:
  linkerd-proxy:
    Container ID:   containerd://4bcbcc76476b8c7c9cd1e89bde7f795851becd96cd8b5d690f1db5adbcc3a0f6
    Image:          cr.l5d.io/linkerd/proxy:stable-2.13.4
    Image ID:       cr.l5d.io/linkerd/proxy@sha256:0f873d84e059859eabbdb07dbc2311acc37802d5f4f100372fa06d11e63def16
    Ports:          4143/TCP, 4191/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Tue, 27 Jun 2023 17:43:18 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:4191/live delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:4191/ready delay=2s timeout=1s period=10s #success=1 #failure=3
    Environment:
      _pod_name:                                                greeter-client-ds-7b8765597b-frpwd (v1:metadata.name)
      _pod_ns:                                                  default (v1:metadata.namespace)
      _pod_nodeName:                                             (v1:spec.nodeName)
      LINKERD2_PROXY_LOG:                                       warn,linkerd=info,trust_dns=error
      LINKERD2_PROXY_LOG_FORMAT:                                plain
      LINKERD2_PROXY_DESTINATION_SVC_ADDR:                      linkerd-dst-headless.linkerd.svc.cluster.local.:8086
      LINKERD2_PROXY_DESTINATION_PROFILE_NETWORKS:              10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16
      LINKERD2_PROXY_POLICY_SVC_ADDR:                           linkerd-policy.linkerd.svc.cluster.local.:8090
      LINKERD2_PROXY_POLICY_WORKLOAD:                           $(_pod_ns):$(_pod_name)
      LINKERD2_PROXY_INBOUND_DEFAULT_POLICY:                    all-unauthenticated
      LINKERD2_PROXY_POLICY_CLUSTER_NETWORKS:                   10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16
      LINKERD2_PROXY_INBOUND_CONNECT_TIMEOUT:                   100ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_TIMEOUT:                  1000ms
      LINKERD2_PROXY_OUTBOUND_DISCOVERY_IDLE_TIMEOUT:           5s
      LINKERD2_PROXY_INBOUND_DISCOVERY_IDLE_TIMEOUT:            90s
      LINKERD2_PROXY_CONTROL_LISTEN_ADDR:                       0.0.0.0:4190
      LINKERD2_PROXY_ADMIN_LISTEN_ADDR:                         0.0.0.0:4191
      LINKERD2_PROXY_OUTBOUND_LISTEN_ADDR:                      127.0.0.1:4140
      LINKERD2_PROXY_INBOUND_LISTEN_ADDR:                       0.0.0.0:4143
      LINKERD2_PROXY_INBOUND_IPS:                                (v1:status.podIPs)
      LINKERD2_PROXY_INBOUND_PORTS:
      LINKERD2_PROXY_DESTINATION_PROFILE_SUFFIXES:              svc.cluster.local.
      LINKERD2_PROXY_INBOUND_ACCEPT_KEEPALIVE:                  10000ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_KEEPALIVE:                10000ms
      LINKERD2_PROXY_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION:  25,587,3306,4444,5432,6379,9300,11211
      LINKERD2_PROXY_DESTINATION_CONTEXT:                       {"ns":"$(_pod_ns)", "nodeName":"$(_pod_nodeName)"}

      _pod_sa:                                                   (v1:spec.serviceAccountName)
      _l5d_ns:                                                  linkerd
      _l5d_trustdomain:                                         cluster.local
      LINKERD2_PROXY_IDENTITY_DIR:                              /var/run/linkerd/identity/end-entity
      LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS:                    -----BEGIN CERTIFICATE-----
                                                                MIIBhzCCAS6gAwIBAgIBATAKBggqhkjOPQQDAjAcMRowGAYDVQQDExFpZGVudGl0
                                                                eS5saW5rZXJkLjAeFw0yMzA2MjEwODAxNDVaFw0yNDA2MjAwODAyMDVaMBwxGjAY
                                                                BgNVBAMTEWlkZW50aXR5LmxpbmtlcmQuMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcD
                                                                QgAEkh0wh0MMFlW25niTKIx3nR0COyrxn3w4QayZh0IUgvUJ1diUdd27k20ZV+Kq
                                                                u0GGzGjrYG18clIlVhyWUg+XNKNhMF8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdJQQW
                                                                MBQGCCsGAQUFBwMBBggrBgEFBQcDAjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQW
                                                                BBQt8lJK2bhWQkhSS/P9lTOR1UoshTAKBggqhkjOPQQDAgNHADBEAiBtsgnZlSUT
                                                                OHOm5GcaUB6+CcVDzx2r/Fo6K9dPawBn4gIgH9qvmkp34MZY+NF6R7skaS+3AT8h
                                                                u+Jc13aNxUGK5YY=
                                                                -----END CERTIFICATE-----

      LINKERD2_PROXY_IDENTITY_TOKEN_FILE:                       /var/run/secrets/tokens/linkerd-identity-token
      LINKERD2_PROXY_IDENTITY_SVC_ADDR:                         linkerd-identity-headless.linkerd.svc.cluster.local.:8080
      LINKERD2_PROXY_IDENTITY_LOCAL_NAME:                       $(_pod_sa).$(_pod_ns).serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_IDENTITY_SVC_NAME:                         linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_DESTINATION_SVC_NAME:                      linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_POLICY_SVC_NAME:                           linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_TAP_SVC_NAME:                              tap.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
    Mounts:
      /var/run/linkerd/identity/end-entity from linkerd-identity-end-entity (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x7bc9 (ro)
      /var/run/secrets/tokens from linkerd-identity-token (rw)
  greeter-client:
    Container ID:  containerd://55edc40f1c111059b129f224dcd62916fe226061e98ebf8483f371b8f93bbbb4
    Image:         docker-images.music.woa.com/cargo/greeter_client:latest
    Image ID:      docker-images.music.woa.com/cargo/greeter_client@sha256:57006cea8b2e75a7ec34e74124a5c3f49c2f5a579a8ec309a68c811b103e55d9
    Port:          <none>
    Host Port:     <none>
    Command:
      ./greeter_client
    Args:
      -addr
      greeter-server-svc:80
      -nzeros
      15
    State:          Running
      Started:      Tue, 27 Jun 2023 17:43:19 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x7bc9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-x7bc9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  linkerd-proxy-init-xtables-lock:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  linkerd-identity-end-entity:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  linkerd-identity-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  32m   default-scheduler  Successfully assigned default/greeter-client-ds-7b8765597b-frpwd to k8w0
  Normal  Pulled     32m   kubelet            Container image "cr.l5d.io/linkerd/proxy-init:v2.2.1" already present on machine
  Normal  Created    32m   kubelet            Created container linkerd-init
  Normal  Started    32m   kubelet            Started container linkerd-init
  Normal  Pulled     32m   kubelet            Container image "cr.l5d.io/linkerd/proxy:stable-2.13.4" already present on machine
  Normal  Created    32m   kubelet            Created container linkerd-proxy
  Normal  Started    32m   kubelet            Started container linkerd-proxy
  Normal  Pulling    32m   kubelet            Pulling image "docker-images.music.woa.com/cargo/greeter_client:latest"
  Normal  Pulled     32m   kubelet            Successfully pulled image "docker-images.music.woa.com/cargo/greeter_client:latest" in 442.404513ms
  Normal  Created    32m   kubelet            Created container greeter-client
  Normal  Started    32m   kubelet            Started container greeter-client

• greeter-server pod desc

[root@k8m ~/deployment/greeter]# kubectl describe pod/greeter-server-ds-57dc5c4cbf-jxmbj
Name:         greeter-server-ds-57dc5c4cbf-jxmbj
Namespace:    default
Priority:     0
Node:         k8w1/9.135.231.29
Start Time:   Sun, 25 Jun 2023 16:51:01 +0800
Labels:       app=greeter-server
              linkerd.io/control-plane-ns=linkerd
              linkerd.io/proxy-deployment=greeter-server-ds
              linkerd.io/workload-ns=default
              pod-template-hash=57dc5c4cbf
Annotations:  container.seccomp.security.alpha.kubernetes.io/linkerd-init: runtime/default
              container.seccomp.security.alpha.kubernetes.io/linkerd-proxy: runtime/default
              linkerd.io/created-by: linkerd/proxy-injector stable-2.13.4
              linkerd.io/inject: enabled
              linkerd.io/proxy-version: stable-2.13.4
              linkerd.io/trust-root-sha256: 2e1b0607b42e1a804a62dd02770786302c0e644b51cb2ffee40eb661d913e748
              viz.linkerd.io/tap-enabled: true
Status:       Running
IP:           10.244.2.113
IPs:
  IP:           10.244.2.113
Controlled By:  ReplicaSet/greeter-server-ds-57dc5c4cbf
Init Containers:
  linkerd-init:
    Container ID:  containerd://76cbed53ba9e3171afdb88bede41a53b5e2787e926b38efacc54b9c19a2118fd
    Image:         cr.l5d.io/linkerd/proxy-init:v2.2.1
    Image ID:      cr.l5d.io/linkerd/proxy-init@sha256:20349a461f9fb76fde33741a90f9de2a647068f506325ac5e0faf7b7bc2eea72
    Port:          <none>
    Host Port:     <none>
    Args:
      --incoming-proxy-port
      4143
      --outgoing-proxy-port
      4140
      --proxy-uid
      2102
      --inbound-ports-to-ignore
      4190,4191,4567,4568
      --outbound-ports-to-ignore
      4567,4568
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 25 Jun 2023 16:51:04 +0800
      Finished:     Sun, 25 Jun 2023 16:51:04 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  20Mi
    Requests:
      cpu:        100m
      memory:     20Mi
    Environment:  <none>
    Mounts:
      /run from linkerd-proxy-init-xtables-lock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dmrdm (ro)
Containers:
  linkerd-proxy:
    Container ID:   containerd://37641677ec8e0c4d1b3257331eddfa98d36ac20e464945f566831818ac4752b7
    Image:          cr.l5d.io/linkerd/proxy:stable-2.13.4
    Image ID:       cr.l5d.io/linkerd/proxy@sha256:0f873d84e059859eabbdb07dbc2311acc37802d5f4f100372fa06d11e63def16
    Ports:          4143/TCP, 4191/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Sun, 25 Jun 2023 16:51:05 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:4191/live delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:4191/ready delay=2s timeout=1s period=10s #success=1 #failure=3
    Environment:
      _pod_name:                                                greeter-server-ds-57dc5c4cbf-jxmbj (v1:metadata.name)
      _pod_ns:                                                  default (v1:metadata.namespace)
      _pod_nodeName:                                             (v1:spec.nodeName)
      LINKERD2_PROXY_LOG:                                       warn,linkerd=info,trust_dns=error
      LINKERD2_PROXY_LOG_FORMAT:                                plain
      LINKERD2_PROXY_DESTINATION_SVC_ADDR:                      linkerd-dst-headless.linkerd.svc.cluster.local.:8086
      LINKERD2_PROXY_DESTINATION_PROFILE_NETWORKS:              10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16
      LINKERD2_PROXY_POLICY_SVC_ADDR:                           linkerd-policy.linkerd.svc.cluster.local.:8090
      LINKERD2_PROXY_POLICY_WORKLOAD:                           $(_pod_ns):$(_pod_name)
      LINKERD2_PROXY_INBOUND_DEFAULT_POLICY:                    all-unauthenticated
      LINKERD2_PROXY_POLICY_CLUSTER_NETWORKS:                   10.0.0.0/8,100.64.0.0/10,172.16.0.0/12,192.168.0.0/16
      LINKERD2_PROXY_INBOUND_CONNECT_TIMEOUT:                   100ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_TIMEOUT:                  1000ms
      LINKERD2_PROXY_OUTBOUND_DISCOVERY_IDLE_TIMEOUT:           5s
      LINKERD2_PROXY_INBOUND_DISCOVERY_IDLE_TIMEOUT:            90s
      LINKERD2_PROXY_CONTROL_LISTEN_ADDR:                       0.0.0.0:4190
      LINKERD2_PROXY_ADMIN_LISTEN_ADDR:                         0.0.0.0:4191
      LINKERD2_PROXY_OUTBOUND_LISTEN_ADDR:                      127.0.0.1:4140
      LINKERD2_PROXY_INBOUND_LISTEN_ADDR:                       0.0.0.0:4143
      LINKERD2_PROXY_INBOUND_IPS:                                (v1:status.podIPs)
      LINKERD2_PROXY_INBOUND_PORTS:                             80,8080
      LINKERD2_PROXY_DESTINATION_PROFILE_SUFFIXES:              svc.cluster.local.
      LINKERD2_PROXY_INBOUND_ACCEPT_KEEPALIVE:                  10000ms
      LINKERD2_PROXY_OUTBOUND_CONNECT_KEEPALIVE:                10000ms
      LINKERD2_PROXY_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION:  25,587,3306,4444,5432,6379,9300,11211
      LINKERD2_PROXY_DESTINATION_CONTEXT:                       {"ns":"$(_pod_ns)", "nodeName":"$(_pod_nodeName)"}

      _pod_sa:                                                   (v1:spec.serviceAccountName)
      _l5d_ns:                                                  linkerd
      _l5d_trustdomain:                                         cluster.local
      LINKERD2_PROXY_IDENTITY_DIR:                              /var/run/linkerd/identity/end-entity
      LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS:                    -----BEGIN CERTIFICATE-----
                                                                MIIBhzCCAS6gAwIBAgIBATAKBggqhkjOPQQDAjAcMRowGAYDVQQDExFpZGVudGl0
                                                                eS5saW5rZXJkLjAeFw0yMzA2MjEwODAxNDVaFw0yNDA2MjAwODAyMDVaMBwxGjAY
                                                                BgNVBAMTEWlkZW50aXR5LmxpbmtlcmQuMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcD
                                                                QgAEkh0wh0MMFlW25niTKIx3nR0COyrxn3w4QayZh0IUgvUJ1diUdd27k20ZV+Kq
                                                                u0GGzGjrYG18clIlVhyWUg+XNKNhMF8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdJQQW
                                                                MBQGCCsGAQUFBwMBBggrBgEFBQcDAjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQW
                                                                BBQt8lJK2bhWQkhSS/P9lTOR1UoshTAKBggqhkjOPQQDAgNHADBEAiBtsgnZlSUT
                                                                OHOm5GcaUB6+CcVDzx2r/Fo6K9dPawBn4gIgH9qvmkp34MZY+NF6R7skaS+3AT8h
                                                                u+Jc13aNxUGK5YY=
                                                                -----END CERTIFICATE-----

      LINKERD2_PROXY_IDENTITY_TOKEN_FILE:                       /var/run/secrets/tokens/linkerd-identity-token
      LINKERD2_PROXY_IDENTITY_SVC_ADDR:                         linkerd-identity-headless.linkerd.svc.cluster.local.:8080
      LINKERD2_PROXY_IDENTITY_LOCAL_NAME:                       $(_pod_sa).$(_pod_ns).serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_IDENTITY_SVC_NAME:                         linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_DESTINATION_SVC_NAME:                      linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_POLICY_SVC_NAME:                           linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
      LINKERD2_PROXY_TAP_SVC_NAME:                              tap.linkerd-viz.serviceaccount.identity.linkerd.cluster.local
    Mounts:
      /var/run/linkerd/identity/end-entity from linkerd-identity-end-entity (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dmrdm (ro)
      /var/run/secrets/tokens from linkerd-identity-token (rw)
  greeter-server:
    Container ID:  containerd://49de3bfbcf6aa33a1174bcc27e4928119badcc58c94b05e625d1827ab25f838b
    Image:         docker-images.music.woa.com/cargo/greeter_server:v1.0.2
    Image ID:      docker-images.music.woa.com/cargo/greeter_server@sha256:d1e90be614ded64663f81be1a50cf7cafb6854f26f83cc58d19a29ca70da8d67
    Ports:         80/TCP, 8080/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      ./greeter_server
    Args:
      -l
      :80
      -promhttp
      :8080
    State:          Running
      Started:      Sun, 25 Jun 2023 16:51:05 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:  2
    Requests:
      cpu:        500m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dmrdm (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-dmrdm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
  linkerd-proxy-init-xtables-lock:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  linkerd-identity-end-entity:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  linkerd-identity-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

[spgyip]

The problem is with the headless type of svc/greeter-server-svc, change it to normal svc will fix the problem.
But still wonder why is problem with headless svc.

[mateiidavid]

@spgyip hi, thanks for the update! This is to be expected. Headless services are a convenient way for Kubernetes service owners to set up DNS records for their workloads without providing a frontend through a ClusterIP. Every Service object serves the same purpose: provide a frontend, which may either be a DNS name, or a long-lived IP address (normal Service objects fulfil both roles).

Headless services are used primarily for two things:

• Set-up DNS records to allow users to implement load balancing in their clients
• Set-up multiple records for StatefulSet objects, where network identifiers represent subdomains.

If a headless service selects two pods, for example, your client will resolve the hostname and get a random IP, this is the default behaviour for most clients. You can of course choose to resolve all IPs and then, like I said, implement your own load balancing.

Once your gRPC client resolves the DNS name, it will get a pod IP. Linkerd acts only on the pod IP, it detects it is an actual concrete endpoint and routes to it. For Linkerd's load balancing to work, we need to act on a ClusterIP (i.e. a logical target) so that we may discover all concrete endpoints and build a balancer.

It is impossible for Linkerd to detect it is dealing with a headless service, since it relies on packets being redirected to it. Packets will never contain a DNS name. I hope this clears it up.