How does the queue of requests works in proxy? #8848

Unanswered

hnykda asked this question in Q&A

hnykda
Feb 26, 2025

I am confused if there even is a queue, as I did find things like use_proxy at https://github.com/BerriAI/litellm/blob/main/litellm/proxy/proxy_server.py#L753 , but I don't see there being anything about e.g. distributing it via redis, setting TTL etc.

Is this even planned?

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment