test: Add tests cancelling BLS decoupled request in Python backend

[richardhuo-nv]

What does the PR do?

Add tests cancelling BLS decoupled request in Python backend.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

• build
• ci
• docs
• feat
• fix
• perf
• refactor
• revert
• style
• test

Related PRs:

triton-inference-server/python_backend#398

Where should the reviewer start?

Check the response_sender_until_cancelled python decoupled model added and then check the two added testing models.
The tests are based on the two added testing models sending BLS requests to the response_sender_until_cancelled python decoupled model.

Test plan:

• CI Pipeline ID: 25879687

Caveats:

NA

Background

NA

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

https://jirasw.nvidia.com/browse/DLIS-7831

[richardhuo-nv]

To fix the problem, we need to remove the unused import statement for the asyncio module. This will eliminate the unnecessary dependency and make the code cleaner and easier to read. The change should be made in the file qa/L0_backend_python/decoupled/models/decoupled_bls_async_cancel/1/model.py by deleting the line that imports asyncio.

[kthui]

Nice work adding tests for both async and regular decoupled BLS request cancellation!

Request cancellation is best effort in Triton, meaning the cancellation only "happens" if the backend implements it. Can we add two new tests, in addition to the two tests, for checking the behavior of BLS cancellation when the inner model does not honor request cancellation, which the inner model will return all the responses and final flag normally despite cancellation is requested?

Edit: Can we also add a small test checking what will happen if the model is trying to cancel a completed/finialed BLS request?

[richardhuo-nv]

Nice work adding tests for both async and regular decoupled BLS request cancellation!

Request cancellation is best effort in Triton, meaning the cancellation only "happens" if the backend implements it. Can we add two new tests, in addition to the two tests, for checking the behavior of BLS cancellation when the inner model does not honor request cancellation, which the inner model will return all the responses and final flag normally despite cancellation is requested?

Edit: Can we also add a small test checking what will happen if the model is trying to cancel a completed/finialed BLS request?

Thanks! Tests added.

[richardhuo-nv]

To fix the problem, we need to remove the unused import statement for the asyncio module. This will clean up the code and remove the unnecessary dependency, making the code easier to read and maintain.

• Locate the import statement for asyncio on line 26.
• Remove the line import asyncio from the file qa/L0_backend_python/decoupled/models/decoupled_bls_cancel_after_complete/1/model.py.

[yinggeh]

Please avoid force pushing commits as it makes it harder for reviewers to track your changes.

[richardhuo-nv]

Please avoid force pushing commits as it makes it harder for reviewers to track your changes.

Sorry about that, I will not do it next time.

All changes from the last review are in the second commit of this PR, you can check the commit history and click the second commit and find all the changes. I had a habit of squash changes into each one commit for each round of review, but next time I will not squash commits. Thanks!

[yinggeh]

Nice work! Make sure new tests pass the CI pipeline.