Do we want a counter to detect client not recovered or gateway peer down

[davidkel]

At the moment if the client exits when it just so happens that the last set of stats recorded that all failed then it will exit with 2 saying the client/network didn't recover. We mitigate this by ensuring the chaos engine has shutdown for a period of time to allow for successful transactions. Would a counter for number of sequential failures be useful to output as stats ?

We could do something similar for no transactions being submitted/evaluated but this time we could also have a threshold value as well. If at termination we have exceeded that threshold then we will exit with 3 to indicate that the gateway peer is likely to be down. This threshold value would have to match up with when the chaos engine terminated and how long the client was left running for after to ensure that if the final scenario in chaos kills the gateway peer permanently that this will be detected.