the report of the failed subtest is not generated if when='call'

[anb76ru]

Hello! I use the pytest_runtest_logreport method to get subtest reports. A report is generated for when='call', but if the test failed, the report is not generated. I think this is because the log_passing_tests=True attribute is used in the elif report.failed: block. If the test failed, then if when == 'call' we don't step into the code block:

if not self.log_passing_tests:        
    reporter.write_captured_output(report)

and the report is not generated. I don't understand why self.log_passing_test using in report.failed. Maybe this is a mistake. If you remove the 'if not self.log_passing_tests: ', then the report for the failed tests for when='call' is generated without problems.

OS: win10
pytest_ver: 7.1.1

I suggest making reporter.write_captured_output(report) unconditional, but I don't have the rights to create my own branch.

[Zac-HD]

Hi @anb76ru - I've edited your issue a little to clarify the report and use code formatting rather than mixed bold and italics. For future reference, we strongly prefer markdown code blocks over screenshots of code too.

I can't comment directly on this code, but you don't need permissions to create a branch in this repo in order to contribute: the standard open-source workflow is to create your own copy ("fork"), make a branch in that, and then open an pull request ("PR"). This quickstart guide is pretty good, and Pytest maintainers are generally happy to have further discussion about proposed changes on the pull request.

[anb76ru]

Hi @anb76ru - I've edited your issue a little to clarify the report and use code formatting rather than mixed bold and italics. For future reference, we strongly prefer markdown code blocks over screenshots of code too.

I can't comment directly on this code, but you don't need permissions to create a branch in this repo in order to contribute: the standard open-source workflow is to create your own copy ("fork"), make a branch in that, and then open an pull request ("PR"). This quickstart guide is pretty good, and Pytest maintainers are generally happy to have further discussion about proposed changes on the pull request.

Hello Zac-HD, thank you very much. I figured out how to do PR

[nicoddemus]

@anb76ru can you explain exactly, from a users' perspective, what is the current behavior, and what you want to achieve? I would like to have a better use case description (you post what changes you want in pytest's code, but not the behavior you want to see exactly).

I ask because I thought I interpreted the use case before, but now I'm not so sure after rereading the issue.

[anb76ru]

@nicoddemus, OK. I'll try to explain.
I have some structure test

As a rule, in subtests i checking layout.
Test do several steps. Thereafter i making a screenshot and compare this screenshot with reference image.
layout test - this is subtests.
All tests running in job in Jenkins.
In Jenkins we can see stdout
But, stdout for subtests is not displayed. Only Traceback

I want see stdout for subtests. However, if subtests is failed, report is not generated.
A report for subtests is generated for when='call'.
if we deleted condition if not self.log_passing_tests, when report will be generated, and I can see stdout in subtests

Like as

If I will be generated subtests report for when='call', I will get the desired result, but i think what this solution is not the best

[nicoddemus]

Glad I asked. 😅

I had understood somewhat differently, you did mention "subtests" and "report", and pointed to the code in junitxml.py in pytest, for the latter reason I assumed you actually meant "xml reports" by "report".

Really sorry for wasting your time working on the PR, but this is a duplicate of pytest-dev/pytest-subtests#9. While it seems like your PR might fix it, I'm hesitant of applying this change into pytest: I think the junitxml's hook needs to be refactored to account for multiple calls of pytest_runtest_logreport, handling the possible SubTestReport correctly.

I would prefer to fix the issue once we integrate pytest-subtests into pytest itself (many of the long standing issues in pytest-subtests are due that it is hard to integrate tightly with some internal implementation details).

So unfortunately I think we will need to close this, along with #10496, and get to this once we integrate pytest-subtests.

Again, sorry for not realizing/asking for the real use case sooner, hope you understand.

Thanks!

[anb76ru]

@nicoddemus, Thanks for feedback