ModuleNotFoundError: No module named 'distutils.util'

[mdaffad]

Describe the bug
I tried to run pants generate-lockfiles, and the following error message appeared:

pip:   File "/home/***/.cache/pants/named_caches/pex_root/venvs/***/lib/python3.10/site-packages/pip/_vendor/packaging/tags.py", line 7, in <module>
pip:     import distutils.util
pip: ModuleNotFoundError: No module named 'distutils.util'

After debugging, I found that it uses Python 3.10 from the /usr/bin path because I installed a Python interpreter that fulfills the interpreter constraints. I tried to resolve this by updating the search path to search_path = ["<PYENV>", "<PYENV_LOCAL>"] and setting interpreter_constraints = ['>=3.10-dev,<3.12-dev']. My goal is to use the Python interpreter installed via pyenv, which includes the distutils package. I installed the interpreter with the 3.10-dev version.

My question:

1. Is the interpreter selection determined by the lowest fulfilled version available in the search_path? For example: intepreter_constraints = ['>=3.10,<3.12'] search_path = ["<PYENV_LOCAL>", "/usr/bin"]
2. How can I properly define interpreter_constraints for a <major>.<minor>-dev version? When I use interpreter_constraints = ['>=3.10-dev,<3.12-dev'], it works without error, but there is no documentation explaining how to specify constraints for <major>.<minor>-dev.

Note:

1. I verified the constraint by adding version 3.10 via pyenv (without the -dev suffix), but it still properly selects the interpreter with -dev.
2. I checked if the distutils import works by running import distutils.util on the interpreter that was exported using pants export.

Pants version
2.23.0

OS
Ubuntu 20.04

Additional info
May relevant information on python and python-bootstrap configuration

Old pants.toml

...
[python]
interpreter_constraints = ['>=3.10,<3.12']
...
[python-bootstrap]
search_path = ["<PYENV_LOCAL>", "/usr/bin"]

New pants.toml

...
[python]
interpreter_constraints = ['>=3.10-dev,<3.12-dev']
enable_resolves = true

[python-bootstrap]
search_path = ["<PYENV>", "<PYENV_LOCAL>"]

[jsirois]

@mdaffad is it correct to say you're only using 3.10-dev as a hack to ensure /usr/bin Pythons are never found or do you actually mean to use a dev version of 3.10 which is now up to a 3.10.15 production release? If this is just a hack, which I suspect it must be, you should focus instead on whether search_path = ["<PYENV>", "<PYENV_LOCAL>"] is enough to filter out /usr/bin Pythons - it should be. Alternatively, the /usr/bin Python not coming equipped with distutils is a "well known" problem with Debian Python packaging. For example, see here: pex-tool/pex#1027. Perhaps you just want to fix the /usr/bin Python by installing python3-distutils or whatever the particular package may be for your OS flavor. If you still really mean you really want 3.10-dev, the documentation simply won't be found easily unless Pants decides to add its own documentation of this corner case. The closest documentation lives here: https://packaging.pypa.io/en/stable/specifiers.html#packaging.specifiers.Specifier. In this case Pex does not pass the prereleases parameter; so it defaults to autodetect. In other words, unless you have at least one pre-release version in your specifier, pre-releases are ignored. More simply, >=3.10-dev,<3.12 is enough to admit pre-releases since one component of the specifier is a pre-release version (3.10-dev).

[mdaffad]

Thank you for your answer and related docs, @jsirois.

After checking again, the python from pyenv without -dev suffix comes with distutils, i already updated my description above.

For your question

is it correct to say you're only using 3.10-dev as a hack to ensure /usr/bin Pythons are never found...?

Yes, I want to filter out the /usr/bin because it doesnt come with distutils. After install disutils for python on /usr/bin using sudo apt-get install python3.10-distutils and update my toml to only select on that path it works.

I think there should be a dedicated documentation or note on pants regarding distutils requirement since debian is a major distribution.

[jsirois]

I am not a Pants maintainer; so I'll let one of them chime in on docs. Suffice it to say I disagree with your stance; moderate googling would have told you about the Debian issue.

[jsirois]

@mdaffad perhaps you don't understand - distutils is not a requirement. It's part of the Python stdlib. It should always be available. It's the Debian packaging here that is odd / broken, depending on your stance, for leaving it out.

[mdaffad]

Thank you for your help, hope this discussion help others search result as well. For this case, i will use pyenv.