You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Expose Sentinel events—such as SentinelTopologyRefreshEvent—in the public API, so that irrecoverable events during failover (e.g. "READONLY You can't write against a read only replica") can be captured and acted upon without relying on reflection or custom workarounds. This is currently due to the fact that the events are package protected.
Context:
We use Lettuce (version 6.3.0.RELEASE) in a Redis Sentinel setup. During master failover events, we encounter irrecoverable errors such as receiving READONLY You can't write against a read only replica errors on connections. We are testing some mitigations such as #2082 however as the connection issues are very hard to reproduce we'd like to increase the visibility in order to produce alerts on certain Sentinel events.
Alternatives Tried:
Reflection:
We attempted to access internal fields for topology refresh data.
Custom Pub/Sub Listener:
We implemented a listener that subscribes to Sentinel’s +switch-master channel. For example:
StatefulRedisPubSubConnection<String, String> pubSubConnection = redisClient.connectPubSub();
pubSubConnection.addListener(newRedisPubSubListener<String, String>() {
@Overridepublicvoidmessage(Stringchannel, Stringmessage) {
if ("+switch-master".equals(channel)) {
// Capture irrecoverable events, e.g. cannot write to a read-only instance.LOG.warn("Sentinel detected a master failover: {}", message);
}
}
@Overridepublicvoidsubscribed(Stringchannel, longcount) {
LOG.info("Subscribed to Sentinel notifications on channel: {}", channel);
}
@Overridepublicvoidunsubscribed(Stringchannel, longcount) {}
@Overridepublicvoidmessage(Stringpattern, Stringchannel, Stringmessage) {}
});
pubSubConnection.sync().subscribe("+switch-master");
Benefits:
Enables direct integration with external monitoring or recovery logic.
Eliminates reliance on fragile internal APIs or reflection.
Provides a more robust mechanism for detecting irrecoverable failover events (like inability to write to a read-only instance) and triggering corrective actions.
Request:
Please consider exposing Sentinel topology events (e.g. SentinelTopologyRefreshEvent) in the public API or providing a documented mechanism for subscribing to these events. This would allow external components to react to failover events, capture irrecoverable errors, and implement custom recovery logic without resorting to workarounds.
The text was updated successfully, but these errors were encountered:
I would like to dig a bit deeper to understand better the problem you are having.
Specifically about the alternatives, I can understand why using reflection was not an option, but I am curious why does the pub/sub approach not work for you? Is it that you would like to not map the message type / payload yourself and would like to have a better abstraction?
Generally if there is a way to do this right now - even if it is not ideal - this would reduce the priority as we have some heavy backlog with missing features and bugs.
We would be happy to receive a PR with it though, if you or somebody else are considering a contribution.
Summary:
Expose Sentinel events—such as SentinelTopologyRefreshEvent—in the public API, so that irrecoverable events during failover (e.g. "READONLY You can't write against a read only replica") can be captured and acted upon without relying on reflection or custom workarounds. This is currently due to the fact that the events are package protected.
Context:
We use Lettuce (version 6.3.0.RELEASE) in a Redis Sentinel setup. During master failover events, we encounter irrecoverable errors such as receiving
READONLY You can't write against a read only replica
errors on connections. We are testing some mitigations such as #2082 however as the connection issues are very hard to reproduce we'd like to increase the visibility in order to produce alerts on certain Sentinel events.Alternatives Tried:
Reflection:
We attempted to access internal fields for topology refresh data.
Custom Pub/Sub Listener:
We implemented a listener that subscribes to Sentinel’s +switch-master channel. For example:
Benefits:
Enables direct integration with external monitoring or recovery logic.
Eliminates reliance on fragile internal APIs or reflection.
Provides a more robust mechanism for detecting irrecoverable failover events (like inability to write to a read-only instance) and triggering corrective actions.
Request:
Please consider exposing Sentinel topology events (e.g. SentinelTopologyRefreshEvent) in the public API or providing a documented mechanism for subscribing to these events. This would allow external components to react to failover events, capture irrecoverable errors, and implement custom recovery logic without resorting to workarounds.
The text was updated successfully, but these errors were encountered: