support redis cluster transport

[zs-neo]

Attempt to address #1021

Thank you very much for your code, it helps us a lot. @auvipy

We use redis-py instead of redis-py-cluster because redis-py-cluster has been merged into redis-py.
Celery works fine on our cluster with multi producers and multi consumers, when a node goes down, it can automatically switch.

celery==5.4.0
Django==4.1
django-celery-beat==2.7.0
django-celery-results==2.5.1
django-filter==24.3
django-redis==5.4.0
django-split-settings==0.3.0
django-timezone-field==5.1
djangorestframework==3.15.2
gunicorn==22.0.0
pyOpenSSL==21.0.0
redis==5.0.0
requests==2.22.0
uWSGI==2.0.18

app = Celery('celery_test', broker='rediscluster://:[email protected]:7024')

[codecov]

Codecov Report

Attention: Patch coverage is 86.76123% with 56 lines in your changes missing coverage. Please review.

Project coverage is 81.80%. Comparing base (5d95b64) to head (4a59e72).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
kombu/transport/rediscluster.py	86.76%	34 Missing and 22 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2204      +/-   ##
==========================================
+ Coverage   81.51%   81.80%   +0.29%     
==========================================
  Files          77       78       +1     
  Lines        9526     9949     +423     
  Branches     1153     1229      +76     
==========================================
+ Hits         7765     8139     +374     
- Misses       1569     1597      +28     
- Partials      192      213      +21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[auvipy]

lets see if the CI passes

[auvipy]

Only one test is failing, due to connection failure

[zs-neo]

@auvipy @Nusnus ping :)
I fixed the issues I found recently and now this PR is ready.
I'm not a kombu expert, any suggestions are welcome

[Nusnus]

Check linter error :)

[zs-neo]

Sorry for the failure, I will fix it and improve the test coverage.

[auvipy]

I am following it and already had reviewed it twice. Will have an in depth review again tomorrow. No worries. Thanks for picking my work

[Nusnus]

https://github.com/celery/kombu/actions/runs/12492830091/job/34860983164?pr=2204

[zs-neo]

https://github.com/celery/kombu/actions/runs/12492830091/job/34860983164?pr=2204

Added docs for kombu.transport.rediscluster, now I'm sure there won't be problems anymore

[Nusnus]

Finally that everything passes on the CI, we can properly review the code.
That being said, we’re currently in a release phase so we can’t merge it until we complete the release, after the new year holidays.

Good work and thank you for fixing everything.

[lan17]

Would this also support Redis cluster for backend?

[jnu]

@zs-neo thanks for all of your work on this! I'm testing out this PR right now and noticed that TLS support is broken. it looks as though it might be as simple as using the rediss scheme here with the ssl logic above?