Skip to content

Serious performance issue with redis connection pooling due to CaseInsensitiveDict #3624

Open
@aldem

Description

@aldem

I was doing performance tests in my app, and noticed that if I serve a large number of requests simultaneously, doing something like this:

async with redis.asyncio.Redis(connection_pool=redis_blocking) as conn:
    await conn.set(...)

... the number of requests per second is significantly lower (around 45-50%) compared to the case where I take a connection only once.

Since every request (as the app provides an API) requires taking a connection from the pool, I cannot do it once and then use multiple times - it will be even slower due to serialization (and will actually defeat the purpose of the pool).

Profiling revealed that the slowdown is primarily caused by the use of CaseInsensitiveDict in the Redis.__init__() method:

redis-py/redis/client.py

Lines 382 to 387 in a4df6b2

self.response_callbacks = CaseInsensitiveDict(_RedisCallbacks)
if self.connection_pool.connection_kwargs.get("protocol") in ["3", 3]:
self.response_callbacks.update(_RedisCallbacksRESP3)
else:
self.response_callbacks.update(_RedisCallbacksRESP2)

Especially, update() is expensive, since there are many keys to update; the profiler shows that __setitem__() alone consumes approximately 50% of the time spent, most of which is attributed to the str.upper() method.

Replacing the dictionary with a simple dict reduced the time needed to acquire the client (not a connection) by a factor of 12, resulting in a performance boost of approximately 45% in the aforementioned use case, and its performance became nearly comparable to that of acquiring a connection only once.

Now the question - is there a reason to use CaseInsensitiveDict? I could not spot anything obvious in the code that requires case-insensitive keys, while the performance gain is significant.

Additionally, moving the initialization of self.response_callbacks to the pool (as they are constants anyway) could potentially lead to further performance improvements.

Thank you!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions