Linkerd load testing with slowcooker connection timeout

[sv6375261073]

Hi Team,

I am using linkerd 2.13.3 version,
I deployed linkerd in docker-desktop and meshed with a service then trying to load test to enable it in dev env for loadtest. But getting connection timeout issue and not getting the proper point of failure.

Please help me here, stuck on it not getting proper solution.

/slow_cooker # ./slow_cooker -qps 10 -concurrency 3000 -iterations 1 -timeout 600s -reportLatenciesCSV report.csv http://slackbot.default.svc

sending 30000 GET req/s with concurrency=3000 to http://slackbot.default.svc ...

####### iter good/b/f t goal% min [p50 p95 p99 p999] max bhash change
2023-07-14T03:34:08Z 0 168/0/0 300000 0% 10s 1355 [1835 2905 2957 2961 ] 2961 0 +
{
"p50": 1835,
"p75": 2789,
"p90": 2799,
"p95": 2905,
"p99": 2957,
"p999": 2961
}
Get "http://slackbot.default.svc": dial tcp: i/o timeout
Get "http://slackbot.default.svc": EOF
Get "http://slackbot.default.svc": EOF
Get "http://slackbot.default.svc": dial tcp: i/o timeout
Get "http://slackbot.default.svc": dial tcp: i/o timeout
Get "http://slackbot.default.svc": dial tcp: i/o timeout
Get "http://slackbot.default.svc": dial tcp: i/o timeout
Get "http://slackbot.default.svc": dial tcp: i/o timeout
Get "http://slackbot.default.svc": dial tcp 10.96.174.104:80: i/o timeout
Get "http://slackbot.default.svc": dial tcp 10.96.174.104:80: i/o timeout
Get "http://slackbot.default.svc": dial tcp 10.96.174.104:80: i/o timeouteout
Get "http://slackbot.default.svc": dial tcp 10.96.174.104:80: i/o timeout
Get "http://slackbot.default.svc": dial tcp 10.96.174.104:80: i/o timeout
Get "http://slackbot.default.svc": EOF
Get "http://slackbot.default.svc": EOF
Get "http://slackbot.default.svc": EOF
Get "http://slackbot.default.svc": http: server closed idle connection
Get "http://slackbot.default.svc": EOF
Get "http://slackbot.default.svc": EOF

###################################### linkerd pod log ################################

Defaulted container "linkerd-proxy" out of: linkerd-proxy, destination, sp-validator, policy, linkerd-init (init)
[ 0.008724s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[ 0.009751s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[ 0.009767s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[ 0.009768s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[ 0.009769s] INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[ 0.009770s] INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[ 0.009772s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.009773s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
[ 26.728578s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
[ 26.931276s] WARN ThreadId(01) watch{port=8443}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[ 27.434894s] WARN ThreadId(01) watch{port=8443}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[ 27.955622s] WARN ThreadId(01) watch{port=8443}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[ 28.463763s] WARN ThreadId(01) watch{port=8443}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[ 28.965985s] WARN ThreadId(01) watch{port=8443}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[ 29.869629s] WARN ThreadId(01) watch{port=8443}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[ 36.820816s] INFO ThreadId(01) linkerd_stack::failfast: Service has recovered

[sv6375261073]

##################. pod log ###############

Defaulted container "linkerd-proxy" out of: linkerd-proxy, slackbot, linkerd-init (init)
[ 0.021801s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[ 0.022706s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[ 0.022722s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[ 0.022723s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[ 0.022724s] INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[ 0.022725s] INFO ThreadId(01) linkerd2_proxy: Local identity is slackbot.default.serviceaccount.identity.linkerd.cluster.local
[ 0.022726s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.022727s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 3.027492s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_stack::failfast: Service entering failfast after 3s
[ 3.027898s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: "controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: service in fail-fast", details: [], metadata: MetadataMap { headers: {} } error.sources=[controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: service in fail-fast, service in fail-fast]
[ 10.025222s] WARN ThreadId(01) watch{port=4191}:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_stack::failfast: Service entering failfast after 10s
[ 15.024874s] WARN ThreadId(01) linkerd_app: Waiting for identity to be initialized...
[ 15.029089s] WARN ThreadId(01) dst:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: request timed out; failed to resolve A record: no record found for Query { name: Name("linkerd-dst-headless.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN } error.sources=[failed to resolve A record: no record found for Query { name: Name("linkerd-dst-headless.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN }, no record found for Query { name: Name("linkerd-dst-headless.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN }]
[ 15.029169s] WARN ThreadId(01) policy:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: request timed out; failed to resolve A record: no record found for Query { name: Name("linkerd-policy.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN } error.sources=[failed to resolve A record: no record found for Query { name: Name("linkerd-policy.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN }, no record found for Query { name: Name("linkerd-policy.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN }]
[ 15.031772s] WARN ThreadId(01) policy:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for Query { name: Name("linkerd-policy.linkerd.svc.cluster.local."), query_type: SRV, query_class: IN }; failed to resolve A record: no record found for Query { name: Name("linkerd-policy.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN } error.sources=[failed to resolve A record: no record found for Query { name: Name("linkerd-policy.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN }, no record found for Query { name: Name("linkerd-policy.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN }]
[ 15.031807s] WARN ThreadId(01) dst:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}: linkerd_app_core::control: Failed to resolve control-plane component error=failed SRV and A record lookups: failed to resolve SRV record: no record found for Query { name: Name("linkerd-dst-headless.linkerd.svc.cluster.local."), query_type: SRV, query_class: IN }; failed to resolve A record: no record found for Query { name: Name("linkerd-dst-headless.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN } error.sources=[failed to resolve A record: no record found for Query { name: Name("linkerd-dst-headless.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN }, no record found for Query { name: Name("linkerd-dst-headless.linkerd.svc.cluster.local."), query_type: AAAA, query_class: IN }]
[ 16.032313s] WARN ThreadId(02) identity:controller{addr=linkerd-identity-headless.linkerd.svc.cluster.local:8080}: linkerd_stack::failfast: Service entering failfast after 3s
[ 16.032443s] ERROR ThreadId(02) identity: linkerd_proxy_identity_client::certify: Failed to obtain identity error=status: Unknown, message: "controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: service in fail-fast", details: [], metadata: MetadataMap { headers: {} } error.sources=[controller linkerd-identity-headless.linkerd.svc.cluster.local:8080: service in fail-fast, service in fail-fast]
[ 26.112366s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=slackbot.default.serviceaccount.identity.linkerd.cluster.local
[ 45.241792s] INFO ThreadId(01) linkerd_stack::failfast: Service has recovered
[ 188.709710s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:40268
[ 188.775066s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:42650
[ 188.776064s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:42708
[ 188.776117s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:44510
[ 188.779964s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:45640
[ 188.780322s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:47016
[ 188.780389s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:47114
[ 188.780418s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:47382
[ 188.780562s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:52374
[ 188.780662s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:52522
[ 188.780691s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:52576
[ 188.781349s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=TLS detection timed out client.addr=10.1.1.133:52622