Linkerd Proxy issues "outbound: linkerd_app_core::serve: Connection closed error=connect timed out"

[ngc4579]

Running Linkerd 2.11.2, data plane proxies from time to time are issuing messages like the following:

Jun 22, 2022 @ 17:30:26.547
[111880.744572s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:30:19.756
[111954.853132s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:30:19.028
[111954.125126s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:29:52.992
[111767.170571s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:29:20.651
[111895.747281s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:29:18.753
[111893.849163s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:29:18.020
[111893.116227s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:28:51.986
[111706.165238s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:28:51.440
[111705.618545s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:28:48.891
[111703.069492s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:28:25.207
[111759.404242s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:28:19.647
[111834.743782s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:28:17.748
[111832.844548s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx
Jun 22, 2022 @ 17:28:17.00
[111832.103637s]  INFO ThreadId(01) outbound: linkerd_app_core::serve: Connection closed error=connect timed out after 1s client.addr=x.x.x.x:xxxxx

Any hints on what might cause this? linkerd check reports Status check results are √.

[olix0r]

This most likely indicates that the proxy timed out when trying to forward a (non-HTTP) TCP connection. That is, the TCP connection to another pod/etc could not be established within 1s. It's unfortunate that we don't also log the target address here... We can look into improving the log message.

You may be able to identify the addresses by running, for instance:

linkerd diagnostics proxy-metrics -n $ns pod/$pod | grep ^outbound_tcp_errors

[ngc4579]

Yep, thank you - at least that brought up the target addresses. Actually I'm not quite sure whether this is a condition to inspect more closely. Most of the affected target addresses are upstream (i.e. outside our cluster) HTTPS endpoints (presumably health endpoints), while one seems to be our Postgresql database. Interestingly though, the application container itself yields no errors whatsoever within the respective time slots.

[ngc4579]

That is, the TCP connection to another pod/etc could not be established within 1s

@olix0r Just do get this right - does that mean initiating the connection takes more than 1s rather than waiting for the response? I.e., the timeout indicates a connection couldn't even be established? Because of... what?

Is this a situation to merely keep an eye on, or rather something that requires closer attention?

[rsmartins78]

Is it possible to increase this 1s timeout to something higher? I couldn't find anything on the docs.

[ngc4579]

@rsmartins78 You can configure this by setting the config.linkerd.io/proxy-outbound-connect-timeout annotation, as stated in the proxy configuration guide.

[babinos87]

Hi guys, this occurs to me too. I am not clear from the above reply, IS this an issue? Or it can be ignored? If it IS an issue and causes problems, what is the best way to resolve it?

Also, as per @rsmartins78 question, is there a way to change this value?

Thank you.

Pinging @olix0r so he get notified too :)

[ngc4579]

@babinos87 Please refer to my answer in the thread above.