[Bug]: Failed test cluster cases

[JinHai-CN]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Actual behavior and How to reproduce it

https://github.com/infiniflow/infinity/actions/runs/12192349442/job/34012833132

python/test_cluster/test_basic.py::test_mock[mock_cluster0] PING 17.0.0.1 (17.0.0.1) 56(84) bytes of data.
64 bytes from 17.0.0.1: icmp_seq=1 ttl=64 time=0.036 ms

--- 17.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.036/0.036/0.036/0.000 ms
Launch node1 successfully. pid: 1610
Cannot remove namespace file "/var/run/netns/ns_node2": No such file or directory
Cannot find device "ns_node2_v"
Cannot find device "ns_node2_vbr"
ns_name: ns_node2, mock_ip: 17.0.0.3/24
PING 17.0.0.1 (17.0.0.1) 56(84) bytes of data.
64 bytes from 17.0.0.1: icmp_seq=1 ttl=64 time=0.048 ms

--- 17.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.048/0.048/0.048/0.000 ms
Launch node2 successfully. pid: 1830
Set node node1 as leader.
Thrift: Fri Dec  6 11:40:34 2024 TSocket::open() timed out <Host: 17.0.0.2 Port: 23851>
<mocked_infinity_cluster.http_result object at 0x7b85921bacb0>
Uninit node node2
Uninit node node1
Exception: 
FAILED

=================================== FAILURES ===================================
___________________________ test_mock[mock_cluster0] ___________________________

mock_cluster = <mocked_infinity_cluster.MockInfinityCluster object at 0x7b859c0b1450>

    def test_mock(mock_cluster: MockInfinityCluster):
        cluster = mock_cluster
        with cluster:
            cluster.add_node("node1", "conf/leader.toml")
            cluster.add_node("node2", "conf/follower.toml")
    
            cluster.set_leader("node1")
>           cluster.set_follower("node2")

python/test_cluster/test_basic.py:52: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/test_cluster/infinity_cluster.py:302: in set_follower
    follower_runner.client.set_role_follower(
python/infinity_http.py:180: in set_role_follower
    r = self.net.request(url, "post", h, d)
python/infinity_http.py:80: in request
    raise e
python/infinity_http.py:75: in request
    self.raise_exception(response)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <mocked_infinity_cluster.mocked_http_network object at 0x7b85921bae60>
resp = <mocked_infinity_cluster.http_result object at 0x7b85921bacb0>

    def raise_exception(self, resp):
        if resp.error_code != 0:
            self.logger.error(resp)
>           raise InfinityException(
                error_code=resp.error_code, error_message=resp.json()["error_message"]
            )
E           infinity.common.InfinityException

python/test_cluster/mocked_infinity_cluster.py:74: InfinityException
------------------------------ Captured log call -------------------------------
DEBUG    test_mock:infinity_cluster.py:254 Minio container minio_host already exists.
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link set br0 down
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link delete br0
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link add br0 type bridge
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip addr add 17.0.0.1/24 dev br0
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link set br0 up
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: iptables -A FORWARD -i br0 -j ACCEPT
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns delete ns_node1
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns add ns_node1
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link delete ns_node1_v
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link delete ns_node1_vbr
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link add ns_node1_v type veth peer name ns_node1_vbr
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link set ns_node1_v netns ns_node1
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns exec ns_node1 ip link set lo up
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link set ns_node1_vbr master br0
INFO     test_mock:mocked_infinity_cluster.py:276 ns_name: ns_node1, mock_ip: 17.0.0.2/24
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns exec ns_node1 ip addr add 17.0.0.2/24 dev ns_node1_v
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns exec ns_node1 ip link set ns_node1_v up
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link set ns_node1_vbr up
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns exec ns_node1 ping -c 1 17.0.0.1
INFO     test_mock:infinity_cluster.py:133 Launch node1 successfully. pid: 1610
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns delete ns_node2
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns add ns_node2
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link delete ns_node2_v
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link delete ns_node2_vbr
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link add ns_node2_v type veth peer name ns_node2_vbr
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link set ns_node2_v netns ns_node2
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns exec ns_node2 ip link set lo up
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link set ns_node2_vbr master br0
INFO     test_mock:mocked_infinity_cluster.py:276 ns_name: ns_node2, mock_ip: 17.0.0.3/24
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns exec ns_node2 ip addr add 17.0.0.3/24 dev ns_node2_v
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns exec ns_node2 ip link set ns_node2_v up
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link set ns_node2_vbr up
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns exec ns_node2 ping -c 1 17.0.0.1
INFO     test_mock:infinity_cluster.py:133 Launch node2 successfully. pid: 1830
DEBUG    test_mock:mocked_infinity_cluster.py:52 cmd: ip netns exec ns_node1 curl -sS --request POST --url http://17.0.0.2:23821/admin/node/current --header 'accept:application/json' --header 'content-type:application/json' --data '{"role": "leader", "name": "node1"}'
INFO     test_mock:infinity_cluster.py:295 Set node node1 as leader.
DEBUG    test_mock:mocked_infinity_cluster.py:52 cmd: ip netns exec ns_node2 curl -sS --request POST --url http://17.0.0.3:23822/admin/node/current --header 'accept:application/json' --header 'content-type:application/json' --data '{"role": "follower", "name": "node2", "address": "17.0.0.2:23851"}'
DEBUG    test_mock:mocked_infinity_cluster.py:52 cmd: ip netns exec ns_node2 curl -sS --request POST --url http://17.0.0.3:23822/admin/node/current --header 'accept:application/json' --header 'content-type:application/json' --data '{"role": "follower", "name": "node2", "address": "17.0.0.2:23851"}'
ERROR    test_mock:mocked_infinity_cluster.py:73 <mocked_infinity_cluster.http_result object at 0x7b85921bacb0>
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link set br0 down
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link delete br0
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns exec ns_node1 ip link set ns_node1_v down
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link set ns_node1_vbr down
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link delete ns_node1_vbr
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns delete ns_node1
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns exec ns_node2 ip link set ns_node2_v down
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link set ns_node2_vbr down
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip link delete ns_node2_vbr
DEBUG    test_mock:mocked_infinity_cluster.py:298 cmd: ip netns delete ns_node2
INFO     test_mock:infinity_cluster.py:138 Uninit node node2
DEBUG    test_mock:timeout_kill.py:56 process 1830 exited with code -15
INFO     test_mock:infinity_cluster.py:138 Uninit node node1
DEBUG    test_mock:timeout_kill.py:56 process 1610 exited with code -15
ERROR    test_mock:infinity_cluster.py:198 Exception:
=========================== short test summary info ============================
FAILED python/test_cluster/test_basic.py::test_mock[mock_cluster0] - infinity...
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
========================= 1 failed, 5 passed in 13.88s =========================
An error occurred: None