Dapr PubSub API: Trim Redis stream when TTL is set

[alicejgibbons]

Describe the feature

Dapr should trim the pubsubs in redis streams honoring TTL.

When using redis streams, redis data is not purged once TTL expires. It's needed to write a custom cleaner, but this cleaner does not have visibility to the requested TTL for the message (and the cleaner would break once implementation details from dapr change).

Release Note

RELEASE NOTE: ADD Redis streams TTL deletes messages.

[elena-kolevska]

dapr/dapr#6948

[elena-kolevska]

The problem users reported here was high memory usage for Redis topics.

Currently, when a message can't be processed, or expires based on its TTL, Dapr copies it to a Dead Letter Queue (DLQ), if one is configured, but it doesn't delete it from the original topic.

Redis streams work as an append-only-log and even though users have the possibility to delete messages from it, removal is not done automatically when a message is acknowledged. For reference, Kafka and AWS SQS behave the same, while RabbitMQ, ASB or Pulsar for example, do remove messagea after acknowledged.

I see two possible options:

• We add a component-level config option (specifically for Redis) to delete messages when acknowledged (deleteOnACK)
• We add a component-level config option specifically for Redis to continuously trim both topics and the DLQ based on number of entries or time (docs)

[alicejgibbons]

We add a component-level config option (specifically for Redis) to delete messages when acknowledged (deleteOnACK)
We add a component-level config option specifically for Redis to continuously trim both topics and the DLQ based on number of entries or time (docs)

QQ on deleteOnACK:

• Moving to DLQ is considered an "ACK" correct?
• This method would allow a case where no consumer ever connects to the broker but if the TTL was set, still wouldn't fill up the memory, correct?

[elena-kolevska]

Moving to DLQ is considered an "ACK" correct?

It's actually the opposite, we move messages to the DLQ when they can't be processed (and thus acknowledged).

This method would allow a case where no consumer ever connects to the broker but if the TTL was set, still wouldn't fill up the memory, correct?

Yes, if the trim option would be enabled, the trimming will happen every time a message is added to the queue.

[elena-kolevska]

I'm gonna move forward implementing option number two above: "We add a component-level config option specifically for Redis to continuously trim both topics and the DLQ based on number of entries or time (docs)"

[alicejgibbons]

We add a component-level config option specifically for Redis to continuously trim both topics and the DLQ based on number of entries or time (docs)

Okay perfect, as we discussed that the second option is more "redis-like" I think it makes sense to go this route. Plus it would give the option for when no consumers ever connect to the pubsub broker, it still wouldn't fill up.

As an aside i do like the deleteOnACK option for an option across ALL stream-like message brokers that don't do this by default.

[elena-kolevska]

So we already supported maxLenApprox (keeps the stream capped at X elements). I added the option to cap by time (entries older than X will be deleted).

[fsedano]

Thanks @alicejgibbons and @elena-kolevska - Going the route of delete older than X looks great to me. I'm not sure how 'deleteonAck' would work since it could be many subscribers consuming from the same message.

However, if we're thinking on setting max age per component this won't really work well for our use case. I was expecting per-message or at least per topic. I.e honor the metadata.ttlInSeconds data on the message.

Thanks,

[alicejgibbons]

@fsedano in this case could you use a separate Dapr component per topic? Then if there was only one topic per component, the maxLenApprox and minIDApprox would trim those streams at the topic level

[elena-kolevska]

I think we can also consider using something like the maxLenApprox and minIDApprox fields in the metadata of the PublishRequest. Because the trimming in Redis happens every time we add a message to the queue, this would be a great way to do it on a topic level.