bug: node won't start with RLN in on-chain dynamic mode 2

[romanzac]

Problem

During "test_publish_with_valid_payloads_dynamic_at_slow_rate" execution, first container running Node1 won't start even within the timeout of 10 minutes. Static mode is working fine. Please have a look.

I've built docker image from latest master version: wakunode2-v0.34.0-24-g650a94

This is second occurrence of the #2662

I've tested with Infura and Alchemy RPC endpoints to Sepolia. Wallets had sufficient amount of SepoliaETH. Registration was successful.

Impact

High occurrence, medium severity. RLN in dynamic mode not functional.

To reproduce

1. Please checkout https://github.com/waku-org/waku-interop-tests.git
2. cd waku-interop-tests
3. vim tests/relay/test_rln.py
4. Comment out lines with @pytest.mark.skip(reason="Waiting for issue resolution https://github.com/waku-org/nwaku/issues/3208")
5. python -m venv .venv
6. source .venv/bin/activate
7. pip install -r requirements.txt
8. pre-commit install
9. pytest tests/relay/test_rln.py -k 'test_publish_with_valid_payloads_dynamic_at_slow_rate'

Expected behavior

RLN Relay in on-chain mode working.

Screenshots/logs

node2_2024-12-10_17-58-59__b4ac17e8-087d-4b11-9fa7-d15bbf97f149__wakuorg_nwaku_latest.log
node1_2024-12-10_17-58-59__b4ac17e8-087d-4b11-9fa7-d15bbf97f149__wakuorg_nwaku_latest.log
node_2024-12-10_17-59-13__61fe3c65-0ab5-4da0-94a0-0d9c5d9000e2__wakuorg_nwaku_latest.log
node_2024-12-10_17-58-59__b3d2cc8f-0788-434b-a49a-f058f2e28bb9__wakuorg_nwaku_latest.log
test.log

nwaku version/commit hash

d464ca06e569ee134dfd8d61ff430d314aef5359

[fryorcraken]

@Ivansete-status @darshankabariya @jm-clius

Do we care about static vs dynamic modes?

[jm-clius]

@fryorcraken afaict static mode is really only suitable for testing. However, the next step after #3312 would be to remove most of the "dynamic" part (i.e. on-chain syncing) from dynamic mode, which should fix the issues associated with it. cc @darshankabariya