Poetry 1.2 or Hatch?

[br3ndonland]

Background

Poetry has been helpful by putting dependency management and packaging together in one tool using pyproject.toml. However, Poetry has also created many problems for maintainers, and many of these problems have been introduced in patch releases.

Background from this project:

• Add Python 3.10 support #36 (couldn't support Python 3.10 for many months because of problems with Poetry)
• Pin and test Poetry version #44 (fixed some problems with Poetry by pinning the version and not updating)
• Install Poetry with pipx #47 (fixed more problems by installing Poetry with pipx instead of their install script)

Poetry 1.2 introduces an enormous number of changes for a minor version, and upgrading from 1.1 to 1.2 may involve as much work as switching to a different tool. Hatch is one such alternative tool. Here, we can discuss the pros and cons of Poetry, Hatch, and other related tools.

Poetry

Poetry for project metadata

• Poetry does not align with Python standards for declaring project metadata in pyproject.toml.
  • Python has a standard for declaring project metadata in pyproject.toml.
  • Poetry does not align with the standard.
  • "The discussion about how the dependencies should be defined within PEP-621 was quite said and frustrating for us as poetry maintainers." (Use the [project] section in pyproject.toml according to PEP-621 python-poetry/poetry#3332 (comment)).
  • Poetry might comply with the Python standard in Poetry 2.0 (How can I use poetry with PEP-621 style `pyproject.toml`? python-poetry/poetry#5833).

Poetry for dependency management

• Poetry's dependency resolver is not as important anymore.
  • When Poetry was first released, pip did not have a dependency resolver, so Poetry's was helpful.
  • However, pip 20.3 added a dependency resolver in 2020. See the pip docs for further details on how it works.
• Poetry does not align with Python standards for version specifiers.
  • Python has a standard for declaring package versions, defined in PEP 440.
  • Poetry supports both Python version specifiers as well as some SemVer-style specifiers like the caret (^).
  • While SemVer is common, the usefulness of diverging from the Python spec has been questioned (can poetry be used by library developers / maintainers, e.g. to declare abstract dependencies? python-poetry/poetry#3757).
• Poetry 1.2 introduced circular dependencies between poetry and poetry-plugin-export.
  • Poetry 1.2.0 conda-forge/poetry-feedstock#70 (the circular dependency makes Poetry 1.2 incompatible with conda and conda-forge)
  • Cyclic dependency between poetry and poetry-plugin-export python-poetry/poetry#6441 (note that the original poster is also now working with Hatch - Poetry 1.2.0 conda-forge/poetry-feedstock#70 (comment))
• Poetry 1.2 handles setuptools differently than Poetry 1.1. This is one reason why updating to Poetry 1.2 may be a breaking change for many projects.
  • No module named 'pkg_resources' during sdist build of dependency when mixing Poetry 1.2 and 1.1 python-poetry/poetry#6328
  • Update poetry to 1.2.2 and regenerate lockfile #55
• Poetry accesses the system keyring during non-publishing operations.
  • Keyring errors during non-publishing operations python-poetry/poetry#1917

Poetry for packaging

• Poetry packaging has been generally reliable. The poetry publish --build command has been a convenient way to build and publish in one step.
• Poetry can add classifiers to package builds automatically, but there have been some issues with this.
  • Poetry was not adding classifiers for Python 3.10 (Include Python 3.10 classifier when running poetry build on 3.10 python-poetry/poetry#4581). This was fixed in Poetry 1.1.11.
  • Poetry has to keep its own list of "available Pythons" (versions of Python that have been released, like 3.11), and their associated classifiers. The list of available Pythons is not always up to date (fix: added 3.11 to list of available pythons python-poetry/poetry-core#477, fix: fix failing tests due to updated list of available pythons python-poetry/poetry#6561)
• Poetry moved its package build backend to a separate package, poetry-core. There have been some issues with poetry-core.
  • poetry-core 1.5.0 introduced a breaking change in a minor release by altering parsing of extras metadata (validate extras against dependencies and in schema python-poetry/poetry-core#542), which broke package builds from source for many projects including isort (Build from source (e.g. via pre-commit) is broken by new poetry version PyCQA/isort#2077).

Poetry for environments

• Poetry doesn't work well without virtualenvs. Complications are observed when Poetry itself is installed without a virtualenv, as well as when project dependencies are installed without a virtualenv.
  • Document why virtual environments should (nearly) always be used, and system site-packages is considered harmful python-poetry/poetry#6398
  • Poetry with virtualenvs.create false installs packages into the wrong place on Ubuntu python-poetry/poetry#6459
  • As explained in Document docker poetry best practices python-poetry/poetry#1879 (comment), using Poetry with Docker requires some additional considerations. Poetry works best with a virtualenv, but the convention in Docker is to not use a virtualenv, because containers themselves provide sufficient isolation. Developers need to adjust their Docker configurations to accommodate virtualenvs.
• Poetry moved their install script twice, requiring every installation environment (CI/CD jobs, Dockerfiles, development environments) to be updated each time, creating unnecessary work for maintainers.
  • Removal of get-poetry.py python-poetry/poetry#6377 (get-poetry.py -> install-poetry.py)
  • Removal of install-poetry.py python-poetry/poetry#6676 (install-poetry.py -> install.python-poetry.org)
  • Install Poetry with pipx #47 (to avoid this problem, inboard started installing Poetry with pipx instead of their install script)

Hatch as an alternative to Poetry

General points

• Hatch aligns with Python standards, meaning that interoperability is high and risk of lock-in is low.
• Notable packages are adopting Hatch. Related to this project:
  • FastAPI (🔧 Update package metadata, drop support for Python 3.6, move build internals from Flit to Hatch fastapi/fastapi#5240)
  • HTTPX (Update package metadata encode/httpx#2334)
  • pydantic
  • Starlette (Update package metadata encode/starlette#1819)
  • Uvicorn (Update package metadata encode/uvicorn#1612)

Hatch for project metadata

Hatch does align with the Python standard for declaring project metadata in pyproject.toml. Hatch does not have Poetry in its migration FAQ, but a Poetry pyproject.toml could be refactored as described in the Hatch docs on project metadata:

• Most of the project metadata in the [tool.poetry] section moves to the [project] section.
• The dynamic version specifier is a useful feature that Poetry does not have (Make version in pyproject.toml optional, or at least validate it *after* the plugins are run python-poetry/poetry#4299). The version number moves to __init__.py or __about__.py, and is then directly accessible from Python code.

Hatch for dependency management

Hatch dependencies can be installed with pip, and can therefore benefit from the pip dependency resolver and pip keyring support.

Refactoring dependencies in pyproject.toml is fairly straightforward, as described in the Hatch docs on project metadata:

• [tool.poetry.dependencies] move to the dependencies list in the [project] section
• [tool.poetry.dev-dependencies] move to a group in [project.optional-dependencies] (these groups are called "extras" according to Python packaging standards)
• [tool.poetry.extras] would also move to [project.optional-dependencies]
• [tool.poetry.urls] would move to [project.urls]

Hatch currently has some limitations around managing dependencies:

• No support for locking (Environment dependency locking pypa/hatch#355) - lack of lockfile support could be considered a drawback, though it may be supported in the future as described in discussion 355 and the FAQ
• No support for adding/removing dependencies (Command to add a dependency pypa/hatch#437) - dependencies have to be written into the pyproject.toml file manually, which is maybe a minor drawback, but not a big deal

Hatch for packaging

The [build-system] section of pyproject.toml could be easily refactored as described in the docs on build configuration:

• requires = ["poetry-core>=1.0.0"] -> requires = ["hatchling"]
• build-backend = "poetry.core.masonry.api" -> build-backend = "hatchling.build"

Hatch for environments

• Hatch supports environments
• Environment commands can be easily migrated from Poetry to Hatch (poetry run -> hatch run)
• Environment directory paths can be customized to match the poetry config virtualenvs.in-project true setting (in project_directory/.venv)
  • Project-specific configuration goes in pyproject.toml in a section like [tool.hatch.envs.<ENV_NAME>]
  • Within a [tool.hatch.envs.<ENV_NAME>] section, setting path = ".venv" will create an in-project virtualenv in the same location as a Poetry in-project virtualenv. A similar effect can be achieved by setting the environment variable HATCH_ENV_TYPE_VIRTUAL_PATH=.venv.
  • The appropriate environment can be selected in GitHub Actions CI/CD or Docker images by setting the HATCH_ENV environment variable or specifying the environment name in commands, like hatch env create ci or hatch run ci:coverage run.

[ofek]

Wow, this is a fantastic write-up! (Hatch creator btw)

[br3ndonland]

Wow, this is a fantastic write-up! (Hatch creator btw)

@ofek I was able to migrate the project to Hatch (#58). Hatch has been great so far, and I will look for ways I can contribute to Hatch in the future.

Thanks and Happy New Year!

[ofek]

I am very happy that you find value in using Hatch, Happy New Year to you as well!

[bodograumann]

One other thing that could be mentioned: Poetry tries to access the keyring, even when that it not actually necessary: python-poetry/poetry#1917
It causes problems for many people from what I can tell.
I know you have decided to switch to Hatch, but this overview is so amazing, it would be cool to maintain it just a bit.

[br3ndonland]

Thanks 🙏! I agree, would be great to keep the info here updated. I have added your suggestion to the "Poetry for dependency management" section, and the corresponding info on the pip keyring integration to the "Hatch for dependency management" section.

[br3ndonland]

Added some info about the breaking change in poetry-core==1.5.0 (python-poetry/poetry-core#542). Lots of projects getting hit by this right now, like isort (PyCQA/isort#2077).

[bodograumann]

To be fair the failures actually indicate broken extra dependency configs.
So instead of a missing dependency with possible production implication you now get a much earlier error.

[spookyuser]

Thank for this repo, i feel like i finally understand hatch bc of it!!, honestly this pyproject.toml is way better than the one generated by hatch new, any way, since you're using hatch for dependencies and builds, just wondering how you add dependencies, bc rn i'm just copying and pasting them to the dependencies array, which feels weird

[br3ndonland]

Awesome! Yeah, Hatch doesn't add dependencies on its own (pypa/hatch#437). I just write the dependencies into pyproject.toml directly. No big deal.

[spookyuser]

I tried that for a while but it gets really annoying, like, how do you handle version numbers?

[spookyuser]

I feel like i try every python new python dependency manager for a few minutes but it rolls down the mountain and when i pick it back up it's a different dependency manager and have to start all over again and halfway up it falls back down :(

[br3ndonland]

I was able to migrate the project from pre-commit to Hatch scripts today (#89). It was easy to do and works great.