linkerd-network-validator doesn't restart failed pod

[darek76000]

Hi,
We tested linkerd stable-2.13.3 and stable-2.13.5 with linkerd-cni enabled and both have the same issue.

In this article https://linkerd.io/2022/12/01/what-really-happens-at-startup-linkerd-init-containers-the-cni-and-more/#the-linkerd-cni-plugin you wrote that:

As of edge-22.10.3, though, the init container will validate that IPTables is really configured correctly: if not, it will fail, and allow Kubernetes to restart the Pod. This eliminates a startup race. (edge-22.10.3 will be incorporated into stable-2.13.0.)

We can see that pod is not restarted but only linkerd-network-validator init-container is.

Scenario:

1. Delete calico pod on one node, calico pod is started - now we don't have iptables configuration for linkerd because calico has overwritten it.
2. Delete nginx pod with linkerd sidecar injected. New nginx pod can't start because of linkerd-network-validator error:

2023-06-29T10:34:41.528461Z INFO linkerd_network_validator: Listening for connections on 0.0.0.0:4140
2023-06-29T10:34:41.528520Z DEBUG linkerd_network_validator: token="uwINdLTFiiUTXLtt55xhxttP6FIDgbWuKdwncqG2GNu8cXetaJTrTEQjd3stmAJ\n"
2023-06-29T10:34:41.528598Z INFO linkerd_network_validator: Connecting to 1.1.1.1:20001
2023-06-29T10:34:51.529591Z ERROR linkerd_network_validator: Failed to validate networking configuration. Please ensure iptables rules are rewriting traffi
Stream closed EOF for ingress-nginx/ingress-nginx-controller-ggwtt (linkerd-network-validator)

Describe of the pod:

Normal Created 19m (x5 over 21m) kubelet Created container linkerd-network-validator
Normal Started 19m (x5 over 21m) kubelet Started container linkerd-network-validator
Warning BackOff 101s (x86 over 21m) kubelet Back-off restarting failed container

Scenario part 2:
3. Delete and recreate linkerd-cni pod so now we have proper iptables configuration.
4. Wait for nginx pod to be ready, but during the linkerd-network-validator container restarts nothing happens. Same logs as before.
5. Delete and recreate nginx pod manually, everything is up and running. linkerd-network-validator container has no errors. Iptables are validated and nginx is going to start.

So my question is that linkerd-network-validator works ok? K8s should restart whole failed nginx pod or only linkerd-network-validator init container?

Regards!

[alpeb]

Thanks for the detailed write-up. Indeed, I think the claim in that post is wrong; when the network-validator catches a bad iptables config it will fail the pod but that doesn't make the scheduler restart it. It needs to be restarted manually. It would be great to have this be automated but I'm not aware of any workaround, besides some kind of external controller doing the restart after catching this situation.