Warnings misuse

[webknjaz]

Summary

I've faced something that I believe is improper use of the warnings mechanism in Python.
Warnings are supposed to signal about something deprecated or called incorrectly. However, I see that it's being used to duplicate the reported information during regular operation (emitting a warning in addition to printing out the same and setting the test session outcome).

So in this instance, the end-user uses pytest-cov correctly, but still gets warnings.

I tend to surface warnings as errors so they're caught early, which causes it to show up as an internal pytest error. I'm convinced this shouldn't be happening.

Expected vs actual result

When I'm running python -Werror -Im pytest -- some/test.py, while having fail_under = 100, this causes a traceback.

Since it just runs a subset of tests, it's expected that coverage wouldn't be at 100%. And I'd expect it to mark the test session as failed.
What I didn't expect, though, was that it additionally issues a warning, which -Werror turns into an error resulting in a traceback being printed out.

Reproducer

Versions

$ .tox/py/bin/pip show pytest-cov
Name: pytest-cov
Version: 6.0.0
Summary: Pytest plugin for measuring coverage.
Home-page: https://github.com/pytest-dev/pytest-cov
Author: Marc Schlaich
Author-email: [email protected]
License: MIT
Location: ~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages
Requires: coverage, pytest
Required-by: 

$ .tox/py/bin/pip show pytest    
Name: pytest
Version: 8.3.4
Summary: pytest: simple powerful testing with Python
Home-page: https://docs.pytest.org/en/latest/
Author: Holger Krekel, Bruno Oliveira, Ronny Pfannschmidt, Floris Bruynooghe, Brianna Laugher, Florian Bruhin, Others (See AUTHORS)
Author-email: 
License: MIT
Location: ~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages
Requires: iniconfig, packaging, pluggy
Required-by: pytest-cov, pytest-mock, pytest-xdist

$ .tox/py/bin/pip show coverage  
Name: coverage
Version: 7.6.10
Summary: Code coverage measurement for Python
Home-page: https://github.com/nedbat/coveragepy
Author: Ned Batchelder and 235 others
Author-email: [email protected]
License: Apache-2.0
Location: ~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages
Requires: 
Required-by: covdefaults, pytest-cov

Config

The covdefaults plugin of coveragepy sets fail_under = 100

#.coveragerc
[run]
plugins =
  covdefaults

Code

This is the repo version without the workaround: https://github.com/antonbabenko/pre-commit-terraform/tree/4b0ad90

tox -qq -- tests/pytest/_cli_test.py::test_app_exit -qq will do:

$ tox -qq -- tests/pytest/_cli_test.py::test_app_exit -qq
.
ERROR: Coverage failure: total of 61 is less than fail-under=100
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pluggy/_callers.py", line 156, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pytest_cov/plugin.py", line 355, in pytest_runtestloop
INTERNALERROR>     warnings.warn(CovFailUnderWarning(message), stacklevel=1)
INTERNALERROR> pytest_cov.CovFailUnderWarning: Coverage failure: total of 61 is less than fail-under=100
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pluggy/_callers.py", line 160, in _multicall
INTERNALERROR>     _warn_teardown_exception(hook_name, teardown[1], e)
INTERNALERROR>   File "~/src/github/antonbabenko/pre-commit-terraform/.tox/py/lib/python3.12/site-packages/pluggy/_callers.py", line 50, in _warn_teardown_exception
INTERNALERROR>     warnings.warn(PluggyTeardownRaisedWarning(msg), stacklevel=5)
INTERNALERROR> pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
INTERNALERROR> Plugin: _cov, Hook: pytest_runtestloop
INTERNALERROR> CovFailUnderWarning: Coverage failure: total of 61 is less than fail-under=100
INTERNALERROR> For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning

py: exit 3 (0.30 seconds) ~/src/github/antonbabenko/pre-commit-terraform> .tox/py/bin/python -bb -E -s -I -Werror -m pytest --color=yes tests/pytest/_cli_test.py::test_app_exit -qq pid=2113830

A workaround is to pass -W 'ignore:Coverage failure::pytest_cov.plugin' but it's rather a hack.