Constant errors in policy container

[xolott]

The policy container log is:

024-06-14T00:13:02.240507Z  INFO external_workloads: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-14T00:13:03.120965Z  INFO services: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-14T00:13:09.110952Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-14T00:22:59.126043Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-14T00:27:42.256990Z  INFO external_workloads: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-14T00:32:49.131801Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-14T00:42:39.143256Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-14T00:56:53.745039Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-14T01:01:44.742186Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-14T01:06:39.746578Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-14T01:06:43.764551Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out

Adding the debug log level to the policy controller I see this:

2024-06-14T22:59:56.038214Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-06-14T22:59:56.038221Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T22:59:56.039289Z DEBUG hyper::proto::h1::io: flushed 1217 bytes
2024-06-14T22:59:59.822101Z DEBUG kubert::lease: patch=Strategic(Object {"apiVersion": String("coordination.k8s.io/v1"), "kind": String("Lease"), "metadata": Object {"resourceVersion": String("228617849")}, "spec": Object {"leaseDurationSeconds": Number(30), "renewTime": String("2024-06-14T22:59:59.822071Z")}})
2024-06-14T22:59:59.822199Z DEBUG tower::buffer::worker: service.ready=true processing request
2024-06-14T22:59:59.822268Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: kube_client::client::builder: requesting
2024-06-14T22:59:59.822298Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: hyper::client::pool: reuse idle connection for ("https", 10.0.0.1)
2024-06-14T22:59:59.822499Z DEBUG hyper::proto::h1::io: flushed 2208 bytes
2024-06-14T22:59:59.828837Z DEBUG hyper::proto::h1::io: parsed 7 headers
2024-06-14T22:59:59.828858Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1483 bytes)
2024-06-14T22:59:59.828883Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-06-14T22:59:59.828917Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: hyper::client::pool: pooling idle connection for ("https", 10.0.0.1)
2024-06-14T22:59:59.829065Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder has changed
2024-06-14T22:59:59.829071Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-bcdf65946-6spv7
2024-06-14T22:59:59.829085Z DEBUG status::Controller: linkerd_policy_controller_k8s_status::index: Lease holder has changed
2024-06-14T23:00:01.056309Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:01.056456Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:01.056487Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:01.056495Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:01.056545Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:01.056554Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:01.056560Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:01.056563Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:01.056582Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:05.524733Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(21), flags: (0x4: END_HEADERS) }
2024-06-14T23:00:05.524808Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(21) }
2024-06-14T23:00:05.524827Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(21), flags: (0x1: END_STREAM) }
2024-06-14T23:00:05.525085Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(21), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:00:06.035161Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:06.035217Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-06-14T23:00:06.035227Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:06.036106Z DEBUG hyper::proto::h1::io: flushed 1223 bytes
2024-06-14T23:00:06.042658Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:06.042805Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:06.097162Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(23), flags: (0x4: END_HEADERS) }
2024-06-14T23:00:06.097237Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(23) }
2024-06-14T23:00:06.097251Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(23), flags: (0x1: END_STREAM) }
2024-06-14T23:00:06.097508Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(23), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:00:09.829717Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-bcdf65946-6spv7
2024-06-14T23:00:10.045804Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:11.039789Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:11.039845Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:11.039852Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:11.039918Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:11.039933Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:11.039939Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:11.039942Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:11.039966Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:15.042031Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:15.042183Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:15.268607Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xAD (173 bytes)
2024-06-14T23:00:15.698227Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xA9 (169 bytes)
2024-06-14T23:00:16.035059Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:16.035107Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-06-14T23:00:16.035114Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:16.036101Z DEBUG hyper::proto::h1::io: flushed 1216 bytes
2024-06-14T23:00:19.045492Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:19.831416Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-bcdf65946-6spv7
2024-06-14T23:00:21.040505Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:21.040560Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:21.040569Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:21.040651Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:21.040662Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:21.040689Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:21.040699Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:21.040750Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:24.042382Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:24.042546Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:25.520081Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(25), flags: (0x4: END_HEADERS) }
2024-06-14T23:00:25.520171Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(25) }
2024-06-14T23:00:25.520187Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(25), flags: (0x1: END_STREAM) }
2024-06-14T23:00:25.520442Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(25), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:00:26.034758Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:26.034811Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-06-14T23:00:26.034819Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:26.035596Z DEBUG hyper::proto::h1::io: flushed 1220 bytes
2024-06-14T23:00:26.101700Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(27), flags: (0x4: END_HEADERS) }
2024-06-14T23:00:26.101893Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(27) }
2024-06-14T23:00:26.101982Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(27), flags: (0x1: END_STREAM) }
2024-06-14T23:00:26.102300Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(27), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:00:28.823447Z DEBUG kubert::lease: patch=Strategic(Object {"apiVersion": String("coordination.k8s.io/v1"), "kind": String("Lease"), "metadata": Object {"resourceVersion": String("228618090")}, "spec": Object {"leaseDurationSeconds": Number(30), "renewTime": String("2024-06-14T23:00:28.823424Z")}})
2024-06-14T23:00:28.823549Z DEBUG tower::buffer::worker: service.ready=true processing request
2024-06-14T23:00:28.823685Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: kube_client::client::builder: requesting
2024-06-14T23:00:28.823722Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: hyper::client::pool: reuse idle connection for ("https", 10.0.0.1)
2024-06-14T23:00:28.823934Z DEBUG hyper::proto::h1::io: flushed 2208 bytes
2024-06-14T23:00:28.834125Z DEBUG hyper::proto::h1::io: parsed 7 headers
2024-06-14T23:00:28.834142Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1483 bytes)
2024-06-14T23:00:28.834165Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-06-14T23:00:28.834201Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: hyper::client::pool: pooling idle connection for ("https", 10.0.0.1)
2024-06-14T23:00:28.834370Z DEBUG status::Controller: linkerd_policy_controller_k8s_status::index: Lease holder has changed
2024-06-14T23:00:28.834390Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder has changed
2024-06-14T23:00:28.834394Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-bcdf65946-6spv7
2024-06-14T23:00:31.038996Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:31.039041Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:31.039049Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:31.039116Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:31.039131Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:31.039140Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:31.039143Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:31.039166Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:31.045529Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:36.035221Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:36.035311Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-06-14T23:00:36.035334Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:36.036319Z DEBUG hyper::proto::h1::io: flushed 1211 bytes
2024-06-14T23:00:36.041953Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:36.042065Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:38.834858Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-bcdf65946-6spv7
2024-06-14T23:00:40.045301Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:41.043639Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:41.043683Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:41.043691Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:41.043752Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:41.045706Z DEBUG hyper::proto::h1::role: resetting h1 header read timeout timer
2024-06-14T23:00:41.045730Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:41.045734Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:41.045769Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:45.042673Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:45.550411Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(29), flags: (0x4: END_HEADERS) }
2024-06-14T23:00:45.550484Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(29) }
2024-06-14T23:00:45.550496Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(29), flags: (0x1: END_STREAM) }
2024-06-14T23:00:45.550744Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(29), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:00:46.038191Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:46.038241Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-06-14T23:00:46.038248Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:46.039787Z DEBUG hyper::proto::h1::io: flushed 1215 bytes
2024-06-14T23:00:46.106032Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(31), flags: (0x4: END_HEADERS) }
2024-06-14T23:00:46.106100Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(31) }
2024-06-14T23:00:46.106984Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(31), flags: (0x1: END_STREAM) }
2024-06-14T23:00:46.107259Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(31), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:00:47.913884Z DEBUG h2::codec::framed_write: send frame=Settings { flags: (0x0), initial_window_size: 1048576, max_frame_size: 16384, max_header_list_size: 16777216 }
2024-06-14T23:00:47.914035Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Settings { flags: (0x0), enable_push: 0, initial_window_size: 65535, max_frame_size: 16384 }
2024-06-14T23:00:47.914056Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Settings { flags: (0x1: ACK) }
2024-06-14T23:00:47.914064Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=WindowUpdate { stream_id: StreamId(0), size_increment: 983041 }
2024-06-14T23:00:47.914082Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=WindowUpdate { stream_id: StreamId(0), size_increment: 983041 }
2024-06-14T23:00:47.914172Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
2024-06-14T23:00:47.914216Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(1) }
2024-06-14T23:00:47.914228Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(1), flags: (0x1: END_STREAM) }
2024-06-14T23:00:47.914364Z DEBUG linkerd_policy_controller_k8s_index::outbound::index: subscribing to service port key=ServicePort { service: "npi-service", port: 80 }
2024-06-14T23:00:47.914524Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
2024-06-14T23:00:47.914556Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Data { stream_id: StreamId(1) }
2024-06-14T23:00:47.914673Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Settings { flags: (0x1: ACK) }
2024-06-14T23:00:47.914688Z DEBUG Connection{peer=Server}: h2::proto::settings: received settings ACK; applying Settings { flags: (0x0), initial_window_size: 1048576, max_frame_size: 16384, max_header_list_size: 16777216 }
2024-06-14T23:00:48.835625Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-bcdf65946-6spv7
2024-06-14T23:00:49.045630Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:51.041277Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:51.041331Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:51.041339Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:51.041404Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:51.041427Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:51.041439Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:00:51.041442Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:51.041468Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:00:54.041777Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:55.919871Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Reset { stream_id: StreamId(1), error_code: CANCEL }
2024-06-14T23:00:55.919954Z DEBUG hyper::proto::h2::server: stream error: error writing a body to connection: send stream capacity unexpectedly closed
2024-06-14T23:00:56.034625Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:00:56.034679Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-06-14T23:00:56.034688Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:00:56.035688Z DEBUG hyper::proto::h1::io: flushed 1209 bytes
2024-06-14T23:00:57.042682Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:00:57.825240Z DEBUG kubert::lease: patch=Strategic(Object {"apiVersion": String("coordination.k8s.io/v1"), "kind": String("Lease"), "metadata": Object {"resourceVersion": String("228618345")}, "spec": Object {"leaseDurationSeconds": Number(30), "renewTime": String("2024-06-14T23:00:57.825215Z")}})
2024-06-14T23:00:57.825334Z DEBUG tower::buffer::worker: service.ready=true processing request
2024-06-14T23:00:57.825397Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: kube_client::client::builder: requesting
2024-06-14T23:00:57.825467Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: hyper::client::pool: reuse idle connection for ("https", 10.0.0.1)
2024-06-14T23:00:57.825661Z DEBUG hyper::proto::h1::io: flushed 2208 bytes
2024-06-14T23:00:57.832227Z DEBUG hyper::proto::h1::io: parsed 7 headers
2024-06-14T23:00:57.832248Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1483 bytes)
2024-06-14T23:00:57.832275Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-06-14T23:00:57.832314Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: hyper::client::pool: pooling idle connection for ("https", 10.0.0.1)
2024-06-14T23:00:57.832526Z DEBUG status::Controller: linkerd_policy_controller_k8s_status::index: Lease holder has changed
2024-06-14T23:00:57.832621Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder has changed
2024-06-14T23:00:57.832639Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-bcdf65946-6spv7
2024-06-14T23:01:01.040307Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:01:01.040309Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:01:01.040352Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:01:01.040355Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:01:01.040360Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:01.040361Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:01.040426Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:01:01.040426Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:01:01.044814Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:01:06.034531Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:01:06.034590Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-06-14T23:01:06.034626Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:06.035463Z DEBUG hyper::proto::h1::io: flushed 1219 bytes
2024-06-14T23:01:06.042159Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:01:06.042282Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:01:07.834240Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-bcdf65946-6spv7
2024-06-14T23:01:10.045126Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:01:11.037379Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:01:11.037434Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:01:11.037442Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:11.037498Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:01:11.040888Z DEBUG hyper::proto::h1::role: resetting h1 header read timeout timer
2024-06-14T23:01:11.040914Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:01:11.040925Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:11.040957Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:01:14.754181Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xAD (173 bytes)
2024-06-14T23:01:15.042318Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:01:15.530320Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(33), flags: (0x4: END_HEADERS) }
2024-06-14T23:01:15.530419Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(33) }
2024-06-14T23:01:15.530454Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(33), flags: (0x1: END_STREAM) }
2024-06-14T23:01:15.530756Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(33), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:01:15.744967Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0xA9 (169 bytes)
2024-06-14T23:01:15.751255Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(9), flags: (0x4: END_HEADERS) }
2024-06-14T23:01:15.751355Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(9) }
2024-06-14T23:01:15.751373Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(9), flags: (0x1: END_STREAM) }
2024-06-14T23:01:15.751587Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(9), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:01:16.034990Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:01:16.035035Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-06-14T23:01:16.035041Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:16.035908Z DEBUG hyper::proto::h1::io: flushed 1217 bytes
2024-06-14T23:01:16.103249Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(35), flags: (0x4: END_HEADERS) }
2024-06-14T23:01:16.103321Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(35) }
2024-06-14T23:01:16.103335Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(35), flags: (0x1: END_STREAM) }
2024-06-14T23:01:16.103532Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(35), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:01:16.187250Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x93 (147 bytes)
2024-06-14T23:01:17.834995Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-bcdf65946-6spv7
2024-06-14T23:01:18.042284Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x638 (1592 bytes)
2024-06-14T23:01:18.042583Z DEBUG services: linkerd_policy_controller_k8s_index::outbound::index: updating service name="keela-form-meteord" service=ServiceInfo { opaque_ports: {9300, 5432, 25, 3306, 587, 4444, 6379, 11211}, accrual: None }
2024-06-14T23:01:19.045512Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:01:21.039287Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:01:21.039341Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:01:21.039351Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:21.039404Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:01:21.039466Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:01:21.039480Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:01:21.039484Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:21.039509Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:01:24.042202Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:01:24.042341Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:01:26.038930Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:01:26.038978Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-06-14T23:01:26.038986Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:26.041356Z DEBUG hyper::proto::h1::io: flushed 1212 bytes
2024-06-14T23:01:26.826071Z DEBUG kubert::lease: patch=Strategic(Object {"apiVersion": String("coordination.k8s.io/v1"), "kind": String("Lease"), "metadata": Object {"resourceVersion": String("228618579")}, "spec": Object {"leaseDurationSeconds": Number(30), "renewTime": String("2024-06-14T23:01:26.826046Z")}})
2024-06-14T23:01:26.826165Z DEBUG tower::buffer::worker: service.ready=true processing request
2024-06-14T23:01:26.826298Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: kube_client::client::builder: requesting
2024-06-14T23:01:26.826369Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: hyper::client::pool: reuse idle connection for ("https", 10.0.0.1)
2024-06-14T23:01:26.826577Z DEBUG hyper::proto::h1::io: flushed 2208 bytes
2024-06-14T23:01:26.838175Z DEBUG hyper::proto::h1::io: parsed 7 headers
2024-06-14T23:01:26.838194Z DEBUG hyper::proto::h1::conn: incoming body is content-length (1483 bytes)
2024-06-14T23:01:26.838214Z DEBUG hyper::proto::h1::conn: incoming body completed
2024-06-14T23:01:26.838246Z DEBUG HTTP{http.method=PATCH http.url=https://10.0.0.1/apis/coordination.k8s.io/v1/namespaces/linkerd/leases/policy-controller-write?&fieldManager=kubert otel.name="patch" otel.kind="client"}: hyper::client::pool: pooling idle connection for ("https", 10.0.0.1)
2024-06-14T23:01:26.838447Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder has changed
2024-06-14T23:01:26.838463Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-bcdf65946-6spv7
2024-06-14T23:01:26.838469Z DEBUG status::Controller: linkerd_policy_controller_k8s_status::index: Lease holder has changed
2024-06-14T23:01:28.308137Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x6A26 (27174 bytes)
2024-06-14T23:01:28.320945Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x6ACE (27342 bytes)
2024-06-14T23:01:28.321977Z DEBUG pods:apply{ns=keela name=keela-form-meteord-558d89749f-gs4s2}: linkerd_policy_controller_k8s_index::inbound::index: No changes pod=keela-form-meteord-558d89749f-gs4s2
2024-06-14T23:01:28.351834Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x72F6 (29430 bytes)
2024-06-14T23:01:28.353502Z DEBUG pods:apply{ns=keela name=keela-form-meteord-558d89749f-gs4s2}: linkerd_policy_controller_k8s_index::inbound::index: No changes pod=keela-form-meteord-558d89749f-gs4s2
2024-06-14T23:01:30.024660Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x7458 (29784 bytes)
2024-06-14T23:01:30.025992Z DEBUG pods:apply{ns=keela name=keela-form-meteord-558d89749f-gs4s2}: linkerd_policy_controller_k8s_index::inbound::index: No changes pod=keela-form-meteord-558d89749f-gs4s2
2024-06-14T23:01:31.033086Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x75C4 (30148 bytes)
2024-06-14T23:01:31.034993Z DEBUG pods:apply{ns=keela name=keela-form-meteord-558d89749f-gs4s2}: linkerd_policy_controller_k8s_index::inbound::index: No changes pod=keela-form-meteord-558d89749f-gs4s2
2024-06-14T23:01:31.040732Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:01:31.040792Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:01:31.040799Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:31.041012Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:01:31.041404Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:01:31.041420Z DEBUG hyper::proto::h1::io: parsed 3 headers
2024-06-14T23:01:31.041425Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:31.041478Z DEBUG hyper::proto::h1::io: flushed 107 bytes
2024-06-14T23:01:31.046476Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:01:31.681586Z DEBUG h2::codec::framed_write: send frame=Settings { flags: (0x0), initial_window_size: 1048576, max_frame_size: 16384, max_header_list_size: 16777216 }
2024-06-14T23:01:31.681770Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Settings { flags: (0x0), enable_push: 0, initial_window_size: 65535, max_frame_size: 16384 }
2024-06-14T23:01:31.681810Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Settings { flags: (0x1: ACK) }
2024-06-14T23:01:31.681822Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=WindowUpdate { stream_id: StreamId(0), size_increment: 983041 }
2024-06-14T23:01:31.681878Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(1), flags: (0x4: END_HEADERS) }
2024-06-14T23:01:31.681975Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(1) }
2024-06-14T23:01:31.681997Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(1), flags: (0x1: END_STREAM) }
2024-06-14T23:01:31.682012Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=WindowUpdate { stream_id: StreamId(0), size_increment: 983041 }
2024-06-14T23:01:31.682212Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(1), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:01:31.682291Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Settings { flags: (0x1: ACK) }
2024-06-14T23:01:31.682336Z DEBUG Connection{peer=Server}: h2::proto::settings: received settings ACK; applying Settings { flags: (0x0), initial_window_size: 1048576, max_frame_size: 16384, max_header_list_size: 16777216 }
2024-06-14T23:01:31.722715Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(3), flags: (0x4: END_HEADERS) }
2024-06-14T23:01:31.722762Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(3) }
2024-06-14T23:01:31.722773Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(3), flags: (0x1: END_STREAM) }
2024-06-14T23:01:31.722924Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(3), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:01:32.033139Z DEBUG hyper::proto::h1::decode: incoming chunked header: 0x75A7 (30119 bytes)
2024-06-14T23:01:32.035383Z DEBUG pods:apply{ns=keela name=keela-form-meteord-558d89749f-gs4s2}: linkerd_policy_controller_k8s_index::inbound::index: No changes pod=keela-form-meteord-558d89749f-gs4s2
2024-06-14T23:01:36.034510Z DEBUG hyper::proto::h1::role: setting h1 header read timeout timer
2024-06-14T23:01:36.034556Z DEBUG hyper::proto::h1::io: parsed 6 headers
2024-06-14T23:01:36.034563Z DEBUG hyper::proto::h1::conn: incoming body is empty
2024-06-14T23:01:36.037433Z DEBUG hyper::proto::h1::io: flushed 1220 bytes
2024-06-14T23:01:36.041954Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:01:36.042120Z DEBUG hyper::proto::h1::conn: read eof
2024-06-14T23:01:36.098797Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(37), flags: (0x4: END_HEADERS) }
2024-06-14T23:01:36.098872Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(37) }
2024-06-14T23:01:36.098892Z DEBUG Connection{peer=Server}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(37), flags: (0x1: END_STREAM) }
2024-06-14T23:01:36.099070Z DEBUG Connection{peer=Server}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(37), flags: (0x5: END_HEADERS | END_STREAM) }
2024-06-14T23:01:36.840186Z DEBUG status::Index: linkerd_policy_controller_k8s_status::index: Lease holder reconciling cluster index.name=linkerd-destination-bcdf65946-6spv7
2024-06-14T23:01:39.458677Z DEBUG hyper::proto::h1::conn: incoming body decode error: Connection reset by peer (os error 104)
2024-06-14T23:01:39.458754Z DEBUG rustls::common_state: Sending warning alert CloseNotify    
2024-06-14T23:01:39.458781Z DEBUG hyper::proto::h1::conn: error shutting down IO: Broken pipe (os error 32)
2024-06-14T23:01:39.458903Z DEBUG hyper::client::client: client connection error: error shutting down connection: Broken pipe (os error 32)
2024-06-14T23:01:39.458914Z DEBUG servers: kube_runtime::watcher: watcher error: ReadEvents(Custom { kind: Other, error: hyper::Error(Body, hyper::Error(Body, Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })) })
2024-06-14T23:01:39.458950Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)

The next block was a cilium issue that we fixed by downgrading to the previous version. The above still persists

Every linkerd-proxy is login this constantly:

[  7147.456806s]  WARN ThreadId(01) dst:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.0.1.156:8086}: linkerd_reconnect: Failed to connect error=endpoint 10.0.1.156:8086: received corrupt message of type InvalidContentType error.sources=[received corrupt message of type InvalidContentType]
[  7147.509818s]  WARN ThreadId(01) policy:controller{addr=linkerd-policy.linkerd.svc.cluster.local:8090}:endpoint{addr=10.0.4.94:8090}: linkerd_reconnect: Failed to connect error=endpoint 10.0.4.94:8090: received corrupt message of type InvalidContentType error.sources=[received corrupt message of type InvalidContentType]

Running one workload with the linkerd debugger I can see an TLS error:

Transport Layer Security
    TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Decode Error)
        Content Type: Alert (21)
        Version: TLS 1.2 (0x0303)
        Length: 2
        Alert Message
            Level: Fatal (2)
            Description: Decode Error (50)

I verified with hubble and cillium that the pod can communicate with the API Server without any issues.

I'm using:

$ kubectl version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

$ cilium version
cilium-cli: v0.13.2 compiled with go1.20.2 on darwin/arm64
cilium image (default): v1.13.1
cilium image (stable): v1.15.6
cilium image (running): v1.15.6

$ linkerd version
Client version: edge-24.6.1
Server version: edge-24.6.1

This is running in AKS. Linkerd was deployed with terraform using this code:

resource "kubernetes_namespace" "linkerd_namespace" {
  metadata {
    name = "linkerd"
  }
}
resource "kubernetes_namespace" "linkerd_viz_namespace" {
  metadata {
    name = "linkerd-viz"
    labels = {
      "linkerd.io/extension" = "viz"
    }
  }
}

resource "kubernetes_namespace" "linkerd_cni_namespace" {
  metadata {
    name = "linkerd-cni"
    labels = {
      "config.linkerd.io/admission-webhooks" = "disabled"
      "linkerd.io/cni-resource"              = "true"
    }
  }
}

resource "helm_release" "linkerd_cni" {
  depends_on = [kubernetes_namespace.linkerd_namespace]
  name       = "linkerd-cni"
  repository = "https://helm.linkerd.io/edge"
  chart      = "linkerd2-cni"
  namespace  = "linkerd-cni"
  version    = "2024.6.1"
  wait       = true
}

resource "helm_release" "linkerd_crds" {
  depends_on = [kubernetes_namespace.linkerd_namespace, helm_release.linkerd_cni]
  name       = "linkerd-crds"
  repository = "https://helm.linkerd.io/edge"
  chart      = "linkerd-crds"
  namespace  = "linkerd"
  version    = "2024.6.1"
  wait       = true

  set {
    name  = "cniEnabled"
    value = true
  }
}

resource "helm_release" "linkerd" {
  depends_on = [helm_release.linkerd_crds]
  name       = "linkerd-control-plane"
  repository = "https://helm.linkerd.io/edge"
  chart      = "linkerd-control-plane"
  namespace  = "linkerd"
  version    = "2024.6.1"
  wait       = true

  values = [
    "${file("${path.module}/values-ha.yaml")}"
  ]

  set {
    name  = "controllerReplicas"
    value = var.replicas
  }

  set {
    name  = "cniEnabled"
    value = true
  }

  set {
    name  = "identity.externalCA"
    value = true
  }

  set {
    name  = "identity.issuer.scheme"
    value = "kubernetes.io/tls"
  }

  set {
    name  = "disableHeartBeat"
    value = true
  }

  set {
    name  = "webhookFailurePolicy"
    value = "Fail"
  }

  set {
    name  = "networkValidator.connectAddr"
    value = "0.0.0.0:4140"
  }
}

resource "helm_release" "linker-viz" {
  depends_on = [helm_release.linkerd_crds, module.linkerd_viz_policy]
  name       = "linkerd-viz"
  repository = "https://helm.linkerd.io/edge"
  chart      = "linkerd-viz"
  namespace  = "linkerd-viz"
  version    = "2024.6.1"
  wait       = true
}

[alpeb]

Closed as duplicate of #12755