TRFC-2: Multiqueue support

[s3rius]

Abstract

Currently our brokers only leverage one queue and all tasks are sent in a single queue. The idea is to make it possible to assign different queues to tasks and being able to send tasks and read from multiple queues.

Solution

Currently there's a PR #418 that adds ability to add new methods to broker.task. We can make use of this feature in order to achieve the following API:

@broker.task.queue(...)
async def task(): ...

The question is how to specify queues, because queue API is different to all brokers.

One of possible solutions is to add function extra_queue method to broker, that returns a generic ExtraQueue object which can be used by the same broker to change task's destination.

Here's a usage example:

second_queue = broker.extra_queue(...)

# And then we can use these objects to assign tasks to these queues.

@broker.task.queue(second_queue)
async def task() :..

Questions

• Define ExtraQueue class. (or return strings)
• How to declare queues that brokers need to read from?
• Shall we split extra_queue to extra_read_queue and extra_write_queue

[vonsteer]

Before anything, thanks for continuing to improve this library! I'm particularly interested in this feature as I got a use case for it at work that is currently being handled (poorly) with celery that I would love to replace.

At least from my perspective working with SQS and on taskiq_aio_sqs:

• this kind of feature for SQS would require an extendable ExtraQueue class rather than strings as there's multiple configuration options for each queue (e.g. fifo, MessageVisibilityTimeout, MaxNumberOfMessages...).
• I personally wouldn't split between extra_read_queue/extra_write_queue, in the end the broker will be setting up a single connection to each queue so should be simple enough to just support extra_queue for both read/write internally. This is the way that celery handles it with broker_transport_options, defining the predefined queues once and then the task usage determines what is done with the queues.
• Maybe Queue object might be better naming, then you could also standardize the way the queue options are defined for each broker by each requiring at least one Queue object on broker __init__ that they can extend with more options depending on the broker.

Also, I was wondering with this feature, would we be able to define a single task that reads from a number of queues like so?

queues = [broker.extra_queue(...) i for range(1, 10)]

@broker.task.queues(*queues)
async def task() :..

it could simplify the definition for having a task consuming from a number of queues a lot.