Handle pytest failure

[kasium]

Feature request type

• Suppress false positives from existing pylint warnings
• Add new pytest-specific warning
• Other

Is your feature request related to a problem? Please describe.
This plugin executes pytest in the background which obviously loads modules. If loading of a module fails for whatever reasons (e.g. init imports an non-existing type), pytest will fail. However, pylint-pytest just ignores the failure and continue as nothing happens. Since no fixtures could be collected, all previously suppressed errors will be shown.
Of course, the issue can be found with pylint, etc. But e.g. in my project I suppress >50 errors which makes it hard to find the root cause.

Describe the solution you'd like
I would be nice, if pylint-pytest could fail on an pytest error or at least shown an indicator. Maybe a configuration can help :-)

Sample code to demonstrate the current imperfect behavior

import pytest
import this_is_invalid  # makes pytest fail

@pytest.fixture
def fix():
  pass

def test_a(fix):
  pass

With the above code, no supression will happen, but you will not see, that this was caused by an issue.

[reverbc]

@iakum this is implemented and released with v1.1.0

Can you help to verify if it's working as expected on your end?

[crown-bdee]

Unfortunately this change has resulted in the following error whenever I lint setup.py when using the command pylint setup.py in the root of my projects.

setup.py:setup:1: [F6401(cannot-enumerate-pytest-fixtures), ] pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only` and resolve any potential syntax error or package dependency issues

Running pytest --fixtures --collect-only did not produce any errors to fix.

[reverbc]

Thanks for reporting the issue @crown-bdee! Apparently pytest is not happy when I run the collection on setup.py. I reproduced the error:

setup.py:13: in <module>
    setup(
../_venv_/pylint-pytest-Q6jowuRR/lib/python3.8/site-packages/setuptools/__init__.py:145: in setup
    return distutils.core.setup(**attrs)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/core.py:134: in setup
    ok = dist.parse_command_line()
../_venv_/pylint-pytest-Q6jowuRR/lib/python3.8/site-packages/setuptools/dist.py:707: in parse_command_line
    result = _Distribution.parse_command_line(self)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py:483: in parse_command_line
    args = self._parse_command_opts(parser, args)
../_venv_/pylint-pytest-Q6jowuRR/lib/python3.8/site-packages/setuptools/dist.py:1014: in _parse_command_opts
    nargs = _Distribution._parse_command_opts(self, parser, args)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/distutils/dist.py:532: in _parse_command_opts
    raise SystemExit("invalid command name '%s'" % command)
E   SystemExit: invalid command name 'setup.py'
=========================== short test summary info ===========================
ERROR setup.py - SystemExit: invalid command name 'setup.py'

I just pushed a fix 5804967 can you help to test that out? If that works in your setup then I'll publish a hotfix release soon.

[crown-bdee]

That was quick!
I can confirm that works well in my testing :-)

[kasium]

Also works fine for me. Thanks a lot!

[crown-bdee]

Looks like around 50% of our projects failed on their nightly builds with this warning - this time not on setup.py, but rather on (some of) the actual test files.
Still working on slimming down a minimal reproduction to give you, but so far I've observed that if I run pylint src/* it produces the warning, but if I run pylint src/my_package/tests/test_my_module.py then it doesn't produce the warning. Running collect on the failing module doesn't show any errors.
Also, wiping out the contents of that test file with the failure didn't remove the error.

[reverbc]

@crown-bdee can you provide more info? More specifically the output from pytest --collect-only --fixtures src/my_package/tests/test_my_module.py

It could probably relate to how the folders/modules were structured.

[crown-bdee]

I have narrowed it down (in one project at least, haven't checked through others yet) to it producing the error if a module with in the package has either a raise or sys.exit() call at the module scope. The module that has it doesn't need to be imported to a test file to cause the failure, it just needs to be present in the package. I've attached a minimal example.
After setting up the venv and activating, running pylint src/mypackage/ should produce the error.
sample.zip

Output of pylint command:

F  src/mypackage/tests/test_failure.py:1: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only path/to/current/module.py` and resolve any potential syntax error or package dependency issues (cannot-enumerate-pytest-fixtures | F6401)

Output of collect:

pytest --fixtures --collect-only src/mypackage/tests/test_failure.py
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.8.5, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /tmp/work/sample
collected 1 item                                                                                                                                                                                                  

<Package tests>
  <Module test_failure.py>
    <Function test_me>
...[trimmed for brevity]

Thanks for your work on this!

[crown-bdee]

Out of interest, @reverbc did that reproduce the error on your end?

[reverbc]

Out of interest, @reverbc did that reproduce the error on your end?

#22 should fix this issue when the raise/sys.exit(1) occurs in an unrelated module

[crown-bdee]

Thanks