IMA: Message processing time can be slow down in case unstable network

[EvgeniyZZ]

Description
Message processing between schains can be slow down for a while in case unstable network OR network connection
Version: 2.1.0-beta.2
Skaled: any

Steps to reproduce

• Turn off 1-5 skaled core containers
• Observe logs of IMA Agent

Actual result
In case of unstable network IMA Agent may stuck for some period of time after unsuccessful getIncomingMessagesCounter() call
from two minutes up to 7, depends on amount of connected chains.

Expected result
Add expected result.
Logs
For grep use: transfer loop from skale-welcome-that-diphda to skale-swift-quixotic-marfic, S2S
imaS2S_stuck.txt
imaLog.txt

@DmytroNazarenko

2024-1-30 16:13:43.243: S2S/#17 transfer loop from skale-welcome-that-diphda to skale-swift-quixotic-marfic: Using externally provided signing function
2024-1-30 16:13:43.243: S2S/#17 transfer loop from skale-welcome-that-diphda to skale-swift-quixotic-marfic: Will call src-chain.MessageProxy.getOutgoingMessagesCounter()...
2024-1-30 16:21:22.445: S2S/#17 transfer loop from skale-welcome-that-diphda to skale-swift-quixotic-marfic: Will call dst-chain.MessageProxy.getIncomingMessagesCounter()...