Why does sentry-issue-occurrence-consumer can't connect to sentry-kafka-controller？

[ZLBillShaw]

I configured an external ClickHouse, and logs show it has connected successfully. However, it seems that the consumer cannot connect to Kafka. My Kafka setup is unchanged from the default in values.yaml.
Below are the logs from sentry-issue-occurrence-consumer pod

2024-11-06 04:25:35,434 Initializing Snuba...
2024-11-06 04:25:37,340 Snuba initialization took 1.9061147570610046s
2024-11-06 04:25:37,349 Consumer Starting
2024-11-06 04:25:37,350 Checking Clickhouse connections...
2024-11-06 04:25:37,482 Successfully connected to Clickhouse: cluster_name=None
2024-11-06 04:25:37,483 librdkafka log level: 6
2024-11-06 04:25:40,498 New partitions assigned: {Partition(topic=Topic(name='generic-events'), index=0): 0}
%6|1730871000.715|FAIL|rdkafka#consumer-3| [thrd:sentry-kafka-controller-1.sentry-kafka-controller-headless.defa]: sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Disconnected (after 3860218ms in state UP)
2024-11-06 05:30:00,716 Error callback from librdKafka -195, _TRANSPORT, sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Disconnected (after 3860218ms in state UP)
%3|1730871000.717|FAIL|rdkafka#consumer-3| [thrd:sentry-kafka-controller-1.sentry-kafka-controller-headless.defa]: sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Connect to ipv4#11.157.248.32:9092 failed: Connection refused (after 1ms in state CONNECT)
2024-11-06 05:30:00,718 Error callback from librdKafka -195, _TRANSPORT, sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Connect to ipv4#11.157.248.32:9092 failed: Connection refused (after 1ms in state CONNECT)
%3|1730871000.984|FAIL|rdkafka#consumer-3| [thrd:sentry-kafka-controller-1.sentry-kafka-controller-headless.defa]: sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Connect to ipv4#11.157.248.32:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2024-11-06 05:30:00,984 Error callback from librdKafka -195, _TRANSPORT, sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Connect to ipv4#11.157.248.32:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1730871001.501|FAIL|rdkafka#producer-1| [thrd:sentry-kafka-controller-1.sentry-kafka-controller-headless.defa]: sentry-kafka-controller-1.sentry-kafka-controller-headless.default.svc.cluster.local:9092/1: Connect to ipv4#11.157.248.32:9092 failed: Connection refused (after 0ms in state CONNECT)
%6|1730873082.680|FAIL|rdkafka#consumer-3| [thrd:GroupCoordinator]: GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Disconnected (after 5945189ms in state UP)
2024-11-06 06:04:42,681 Error callback from librdKafka -195, _TRANSPORT, GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Disconnected (after 5945189ms in state UP)
%3|1730873082.681|FAIL|rdkafka#consumer-3| [thrd:GroupCoordinator]: GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Connect to ipv4#11.157.248.126:9092 failed: Connection refused (after 0ms in state CONNECT)
2024-11-06 06:04:42,683 Error callback from librdKafka -195, _TRANSPORT, GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Connect to ipv4#11.157.248.126:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1730873082.871|FAIL|rdkafka#consumer-3| [thrd:GroupCoordinator]: GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Connect to ipv4#11.157.248.126:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2024-11-06 06:04:42,871 Error callback from librdKafka -195, _TRANSPORT, GroupCoordinator: sentry-kafka-controller-0.sentry-kafka-controller-headless.default.svc.cluster.local:9092: Connect to ipv4#11.157.248.126:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)

[avznog]

Hello, any news on this issue ?

[ZLBillShaw]

I have already resolved it, it was my network issue