Graceful shutdown doesn't complete + Fail-fast

[bjoernw]

I'm on stable 2.12.4 and all linkerd checks are ✅

I have 2 meshed services. Service A ----> Service B

Service B has

config.alpha.linkerd.io/proxy-wait-before-exit-seconds: "40"
config.linkerd.io/shutdown-grace-period: 45s

Sometimes when deploying a new version of Service B I get: WARN ThreadId(01) linkerd2_proxy: Graceful shutdown did not complete in 45s, terminating now and at the same time Service A logs: HTTP Balancer service in fail-fast and QPS from Service A to Service B drops dramatically.

Is the fail-fast mode essentially a circuit breaker? Can I configure it? Removing Linkerd from Service A removes the problem with the drop in QPS even tough I still see the graceful shutdown issue. It seems like the sending linkerd goes into fail-fast mode and stays there for a while.

What are my options here?

thanks for the help!