[Bug] Redelivery (replay) queue reads don't get deduplicated, limited by managedLedgerMaxReadsInFlightSizeInMB or cached

[lhotari]

Search before asking

• I searched in the issues and found nothing similar.

Read release policy

• I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

Version

any released version

Minimal reproduce step

problem description:
This problem applies to Shared and Key_Shared subscriptions.
When a consumer disconnects and while there are other active consumers, the unacknowledged (pending) message ids get added to the redelivery (replay) queue in the dispatcher. When these messages get dispatched on the next round in the Shared or Key_Shared subscription dispatcher, the reads for the messages don't get deduplicated, limited by managedLedgerMaxReadsInFlightSizeInMB or cached. This is a problem for high fanout scenarios for Shared subscriptions, when a large amount of consumers disconnect and reconnect while there's at least one active consumer so that the dispatcher doesn't discard the replay queue and "rewind" in between.

What did you expect to see?

That redelivery reads get deduplicate, limited by managedLedgerMaxReadsInFlightSizeInMB and are eligible for caching

What did you see instead?

PendingReadsManager isn't used, so all reads will be made multiple times to bookkeeper (in fanout with multiple consumers). The reads won't be limited by managedLedgerMaxReadsInFlightSizeInMB and broker cache is by passed.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!