Skip to content

Commit

Permalink
Merge branch 'release/0.3.5'
Browse files Browse the repository at this point in the history
  • Loading branch information
s3rius committed Mar 31, 2023
2 parents 04dd68d + 5a12a35 commit 318b25b
Show file tree
Hide file tree
Showing 12 changed files with 2,076 additions and 1,372 deletions.
Binary file modified docs/.vuepress/public/favicon.ico
Binary file not shown.
12 changes: 8 additions & 4 deletions docs/.vuepress/styles/index.scss
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,17 @@
visibility: hidden;
}

.hero{
img{
max-width: 55% !important;
padding: 2rem;
.hero-info-wrapper {
img {
max-width: 60% !important;
padding: 1rem;
}
}

.actions {
align-items: flex-end;
}

span.site-name {
visibility: hidden;
}
1 change: 1 addition & 0 deletions docs/examples/introduction/inmemory_run.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ async def add_one(value: int) -> int:


async def main() -> None:
# Never forget to call startup in the beginning.
await broker.startup()
# Send the task to the broker.
task = await add_one.kiq(1)
Expand Down
10 changes: 10 additions & 0 deletions docs/examples/testing/main_file.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import os

from taskiq import AsyncBroker, InMemoryBroker, ZeroMQBroker

env = os.environ.get("ENVIRONMENT")

broker: AsyncBroker = ZeroMQBroker()

if env and env == "pytest":
broker = InMemoryBroker()
9 changes: 4 additions & 5 deletions docs/guide/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,14 @@ Taskiq is a library that helps you send and process python functions in a distri
For example, you have many heavy to calculate functions you want to execute on another server.
You can implement interservice communication by yourself, or you can use Taskiq to make the job done easily.

The core library has only two brokers. It provides CLI, basic functionality for creating tasks, and abstractions to extend functionality. The main idea is to make taskiq modular with a lot of
replaceable parts, such as brokers, result backends, and middlewares.
The core library doesn't have much functionality. It provides two built-in brokers, CLI, basic functionality for creating distributed tasks, and abstractions to extend the taskiq. The main idea of taskiq is to make it modular and easy to extend. We have libraries for many
possible use cases, but if you lack something, you can adopt taskiq to fit your needs.

## Why not use existing libraries?

We couldn't find a solution like Celery or Dramatiq that can run async code and send tasks asynchronously.
It was the main reason we created this project.
We created this project because we couldn't find any project that can execute and send async functions using distributed queues like RabbitMQ.

You might have seen projects built on top of asyncio that solve similar problem, but here's a comparasion table of taskiq and other projects.
You might have seen projects built on top of asyncio that solve a similar problem, but here's a comparison table of the taskiq and other projects.

| Feature name | Taskiq | Arq | AioTasks |
| --------------------------: | :----: | :---: | :------: |
Expand Down
8 changes: 4 additions & 4 deletions docs/guide/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ After installation of the core library, you need to find the broker that fits yo

::: info Cool tip!

We highly recommend [taskiq-aio-pika](https://pypi.org/project/taskiq-aio-pika/) as the broker and [taskiq-redis](https://pypi.org/project/taskiq-redis/) as the result backend for production use.
We highly recommend [taskiq-aio-pika](https://pypi.org/project/taskiq-aio-pika/) or [taskiq-nats](https://pypi.org/project/taskiq-nats/) as the broker and [taskiq-redis](https://pypi.org/project/taskiq-redis/) as the result backend for production use.

:::

Expand Down Expand Up @@ -57,9 +57,9 @@ And that's it. Now let's add some tasks and the main function. You can add tasks

@[code python](../examples/introduction/inmemory_run.py)

::: tip Cool tip!
::: warning Cool warning!

Calling the `startup` method is not required, but we strongly recommend you do so.
Calling the `startup` method is necessary. If you don't call it, you may get an undefined behaviour.

:::

Expand All @@ -74,7 +74,7 @@ Returned value: 2
Ok, the code of the task execution is a little bit fancier than an ordinary function call, but it's still relatively simple to understand. To send a task to the broker,
you need to call the `.kiq` method on the function,
it returns the `TaskiqTask` object that can check whether the result is ready
or it can wait for it to become available.
or not. Also it has methods to wait for the result to become available.

You can get more information about taskiq types, CLI and internal structure in the "[Architecture overview](./architecture-overview.md)" section.

Expand Down
4 changes: 2 additions & 2 deletions docs/guide/scheduling-tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ order: 8
# Scheduling tasks

Sometimes you may want to execute some tasks according to some schedule.
For example, you want to call a function every day at 2 pm.
For example, you maybe want to call a function every day at 2 pm.

It's easy to do with taskiq. We have primitives that can help you to solve your problems.
That's not a problem if you use taskiq. We have primitives that can help you to solve your problems.

Let's imagine we have a module, as shown below, and we want to execute the `heavy_task` every 5 minutes.
What should we do?
Expand Down
87 changes: 87 additions & 0 deletions docs/guide/taskiq-with-fastapi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
order: 10
---

# Taskiq + FastAPI

FastAPI is one of the most popular async web frameworks in python. It has gained it's popularity because of two things:
1. It's easy to use;
2. It has a cool dependency injection.

In taskiq we try to make our libraries easy to use and we too have a depenndency injection. But our dependencies
are not compatible with FastAPI's dependencies by default. That is why we have created a library "[taskiq-fastapi](https://pypi.org/project/taskiq-fastapi/)" to make integration with
FastAPI as smooth as possible.

Let's see what we got here. In this library, we provide you with only one public function called `init`. It takes a broker and a string path (as in uvicorn) to the fastapi application (or factory function). This function must be called in your main broker file.


```python
from taskiq import ZeroMQBroker
import taskiq_fastapi

broker = ZeroMQBroker()

taskiq_fastapi.init(broker, "my_package.application:app")

```

There are two rules to make everything work as you expect:
1. Add `TaskiqDepends` as a default value for every parameter with `Request` or `HTTPConnection` types in base dependencies.
2. Use only `TaskiqDepends` in tasks.


::: tip Cool and important note!

The Request or HTTPConnection that you'll get injected in your task is not the same request or connection you have had in your handler when you were sending the task!

:::

Many fastapi dependency functions are depend on `fastapi.Request`. We provide a mocked request to such dependencies. But taskiq cannot resolve dependencies until you explicitly specify that this parameter must be injected.

As an example. If you previously had a dependency like this:

```python
from fastapi import Request
from typing import Any

def get_redis_pool(request: Request) -> Any:
return request.app.state.redis_pool

```

To make it resolvable in taskiq, you need to make it clear that Request object must be injected. Like this:

```python
from fastapi import Request
from taskiq import TaskiqDepends


async def get_redis_pool(request: Request = TaskiqDepends()):
return request.app.state.redis_pool

```


Also you want to call startup of your brokers somewhere.

```python
from fastapi import FastAPI
from your_project.taskiq import broker

app = FastAPI()


@app.on_event("startup")
async def app_startup():
if not broker.is_worker_process:
await broker.startup()


@app.on_event("shutdown")
async def app_shutdown():
if not broker.is_worker_process:
await broker.shutdown()

```

And that's it. Now you can use your taskiq tasks with functions and classes that depend on FastAPI dependenices. You can find bigger examples in the [taskiq-fastapi repo](https://github.com/taskiq-python/taskiq-fastapi).
205 changes: 205 additions & 0 deletions docs/guide/testing-taskiq.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
---
order: 9
---

# Testing with taskiq

Everytime we write programs, we want them to be correct. To achieve this, we use tests.
Taskiq allows you to write tests easily as if tasks were normal functions.

Let's dive into examples.

## Preparations

### Environment setup
For testing you maybe don't want to use actual distributed broker. But still you want to validate your logic.
Since python is an interpreted language, you can easily replace you broker with another one if the expression is correct.

We can set an environment variable, that indicates that currently we're running in testing environment.

::: tabs

@tab linux|macos


```bash
export ENVIRONMENT="pytest"
pytest -vv
```

@tab windows

```powershell
$env:ENVIRONMENT = 'pytest'
pytest -vv
```

:::


Or we can even tell pytest to set this environment for us, just before executing tests using [pytest-env](https://pypi.org/project/pytest-env/) plugin.

::: tabs

@tab pytest.ini

```ini
[pytest]
env =
ENVIRONMENT=pytest
```

@tab pyproject.toml

```toml
[tool.pytest.ini_options]
env = [
"ENVIRONMENT=pytest",
]
```

:::

### Async tests

Since taskiq is fully async, we suggest using [anyio](https://anyio.readthedocs.io/en/stable/testing.html) to run async functions in pytest. Install the [lib](https://pypi.org/project/anyio/) and place this fixture somewhere in your root `conftest.py` file.

```python
@pytest.fixture
def anyio_backend():
return 'asyncio'
```

After the preparations are done, we need to modify the broker's file in your project.

@[code python](../examples/testing/main_file.py)

As you can see, we added an `if` statement. If the expression is true, we replace our broker with an imemory broker.
The main point here is to not have an actual connection during testing. It's useful because inmemory broker has
the same interface as a real broker, but it doesn't send tasks acutally.

## Testing tasks

Let's define a task.

```python
from your_project.taskiq import broker

@broker.task
async def parse_int(val: str) -> int:
return int(val)
```

This simple task may be defined anywhere in your project. If you want to test it,
just import it and call as a normal function.

```python
import pytest
from your_project.tasks import parse_int

@pytest.mark.anyio
async def test_task():
assert await parse_int("11") == 11
```

And that's it. Test should pass.

What if you want to test a function that uses task. Let's define such function.

```python
from your_project.taskiq import broker

@broker.task
async def parse_int(val: str) -> int:
return int(val)


async def parse_and_add_one(val: str) -> int:
task = await parse_int.kiq(val)
result = await task.wait_result()
return result.return_value + 1
```

And since we replaced our broker with `InMemoryBroker`, we can just call it.
It would work as you expect and tests should pass.

```python
@pytest.mark.anyio
async def test_add_one():
assert await parse_and_add_one("11") == 12
```

## Dependency injection

If you use dependencies in your tasks, you may think that this can become a problem. But it's not.
Here's what we came up with. We added a method called `add_dependency_context` to the broker.
It sets base dependencies for dependency resolution. You can use it for tests.

Let's add a task that depends on `Path`. I guess this example is not meant to be used in production code bases, but it's suitable for illustration purposes.

```python
from pathlib import Path
from taskiq import TaskiqDepends

from your_project.taskiq import broker


@broker.task
async def modify_path(some_path: Path = TaskiqDepends()):
return some_path.parent / "taskiq.py"

```

To test the task itself, it's not different to the example without dependencies, but we jsut need to pass all
expected dependencies manually as function's arguments or key-word arguments.

```python
import pytest
from your_project.taskiq import broker

from pathlib import Path

@pytest.mark.anyio
async def test_modify_path():
modified = await modify_path(Path.cwd())
assert str(modified).endswith("taskiq.py")

```

But what if we want to test task execution? Well, you don't need to provide dependencies manually, you
must mutate dependency_context before calling a task. We suggest to do it in fixtures.

```python
import pytest
from your_project.taskiq import broker
from pathlib import Path


# We use autouse, so this fixture
# is called automatically before all tests.
@pytest.fixture(scope="function", autouse=True)
async def init_taskiq_dependencies():
# Here we use Path, but you can use other
# pytest fixtures here. E.G. FastAPI app.
broker.add_dependency_context({Path: Path.cwd()})

yield

# After the test we clear all custom dependencies.
broker.custom_dependency_context = {}

```

This fixture will update dependency context for our broker before
every test. Now tasks with dependencies can be used. Let's try it out.

```python
@pytest.mark.anyio
async def test_modify_path():
task = await modify_path.kiq()
result = await task.wait_result()
assert str(result.return_value).endswith("taskiq.py")

```

This should pass. And that's it for now.
Loading

0 comments on commit 318b25b

Please sign in to comment.