Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Public API Exposure for Sentinel Topology Events #3215

Open
jszczesnyoa opened this issue Mar 12, 2025 · 1 comment
Open
Labels
status: waiting-for-feedback We need additional information before we can continue

Comments

@jszczesnyoa
Copy link

jszczesnyoa commented Mar 12, 2025

Summary:

Expose Sentinel events—such as SentinelTopologyRefreshEvent—in the public API, so that irrecoverable events during failover (e.g. "READONLY You can't write against a read only replica") can be captured and acted upon without relying on reflection or custom workarounds. This is currently due to the fact that the events are package protected.

Context:

We use Lettuce (version 6.3.0.RELEASE) in a Redis Sentinel setup. During master failover events, we encounter irrecoverable errors such as receiving READONLY You can't write against a read only replica errors on connections. We are testing some mitigations such as #2082 however as the connection issues are very hard to reproduce we'd like to increase the visibility in order to produce alerts on certain Sentinel events.

Alternatives Tried:

Reflection:
We attempted to access internal fields for topology refresh data.

Custom Pub/Sub Listener:
We implemented a listener that subscribes to Sentinel’s +switch-master channel. For example:

StatefulRedisPubSubConnection<String, String> pubSubConnection = redisClient.connectPubSub();
pubSubConnection.addListener(new RedisPubSubListener<String, String>() {
    @Override
    public void message(String channel, String message) {
        if ("+switch-master".equals(channel)) {
            // Capture irrecoverable events, e.g. cannot write to a read-only instance.
            LOG.warn("Sentinel detected a master failover: {}", message);
        }
    }
    @Override public void subscribed(String channel, long count) {
        LOG.info("Subscribed to Sentinel notifications on channel: {}", channel);
    }
    @Override public void unsubscribed(String channel, long count) {}
    @Override public void message(String pattern, String channel, String message) {}
});
pubSubConnection.sync().subscribe("+switch-master");

Benefits:

Enables direct integration with external monitoring or recovery logic.
Eliminates reliance on fragile internal APIs or reflection.
Provides a more robust mechanism for detecting irrecoverable failover events (like inability to write to a read-only instance) and triggering corrective actions.

Request:

Please consider exposing Sentinel topology events (e.g. SentinelTopologyRefreshEvent) in the public API or providing a documented mechanism for subscribing to these events. This would allow external components to react to failover events, capture irrecoverable errors, and implement custom recovery logic without resorting to workarounds.

@tishun
Copy link
Collaborator

tishun commented Mar 17, 2025

Hey @jszczesnyoa ,

thanks for filing this request.

I would like to dig a bit deeper to understand better the problem you are having.

Specifically about the alternatives, I can understand why using reflection was not an option, but I am curious why does the pub/sub approach not work for you? Is it that you would like to not map the message type / payload yourself and would like to have a better abstraction?

Generally if there is a way to do this right now - even if it is not ideal - this would reduce the priority as we have some heavy backlog with missing features and bugs.

We would be happy to receive a PR with it though, if you or somebody else are considering a contribution.

@tishun tishun added status: waiting-for-feedback We need additional information before we can continue and removed status: waiting-for-triage labels Mar 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: waiting-for-feedback We need additional information before we can continue
Projects
None yet
Development

No branches or pull requests

2 participants