Issue with NATS Cluster Health Check and Jetstream on OpenShift: Recommendations Needed [v2.10.19]

[mohamedsaleem18]

Observed behavior

A NATS cluster with 3 nodes (Jetstream enabled) is installed on Red Hat OpenShift. Due to an issue with Jetstream, the NATS cluster shut down and was no longer accepting Jetstream connections. When we restarted the NATS pods one by one, the cluster formed with only two pods. One pod failed to start, throwing a "health check failed" error. As a result, the NATS cluster is up and running with only 2 NATS servers. Please find the NATS pod logs below. Could you please provide your recommendations to resolve this issue?

Expected behavior

NATS cluster should be up and running with 3 nodes.

Server and client version

NATS server --version : 2.10.19

Host environment

Redhat OpenShift
nats.txt

Steps to reproduce

No response

[derekcollison]

We would need quite a bit more information.

Might be good to consider a support agreement with Synadia.

[wallyqs]

hi @mohamedsaleem18, what you can try is to change the readiness probe to be done instead on /varz or change the readinessprobe to be /healthz?js-server-enabled=true instead, that way the service will not detach whenever you stop one of the pods. We are changing from this default settings in the next version.