How to handle dependecies?

[anentropic]

I have seen the Polylith concept before but have not used it

Seeing you had this Poetry plugin and I had a new monorepo to structure I decided to give it a go

I am a bit confused about how to handle dependencies

I can see there is a root pyproject.toml and then individual ones for each project

Should I poetry add all deps at the root level and then use poetry poly sync to let the plugin apply only relevant deps to the project pyproject.toml files? i.e. I shouldn't manage the project files manually myself at all?

[anentropic]

I guess I do manage the project pyproject.tomls manually, as in I should manually add bases and components to them (?)

so I tried e.g.

• poetry add django at the root level
• import something from django in one of my bases' core.py
• add that base to a project's pyproject.toml packages directive
• poetry poly sync

but this didn't update the project dependencies

does this mean I have to keep these in sync manually and poetry add everything twice? am I missing something?

it'd be nice if I can add packages to a project from the root without cd into its subdir first

[DavidVujic]

The poetry poly sync would need an entry point defined in the project-specific toml files. You could add your base in the project-specific toml and then run the command. If the base is importing a component, it will add it to the project toml if it is missing.

poetry add is for the third-party dependencies, such as django or requests. You can add the dependencies to the root workspace pyproject.toml for development, and add the needed third-party dependencies to each project in the workspace. The poetry poly check and poetry poly libs commands will highlight any missing libraries, helping developers catch any missing dependencies.

All commands support the --directory option (coming from Poetry). This means that you can run the commands from the workspace root, but for a specific project: poetry poly check --directory projects/my-project

[anentropic]

yes I was primarily talking about 3rd party dependencies

ah, I missed poetry poly check command when I was looking through

so I see if I do:

poetry lock ---directory myproject
poetry poly check

it will tell me e.g. "django is missing in myproject" and then I know to go and manually add it

[anentropic]

another question...

I can see that poly create base generates a structure like:

bases/
  myworkspace/
    mybase/
      __init__.py
      core.py

and __init__.py looks like:

from reporting.infra import core

__all__ = ["core"]

This is quite a nice default, but my question is: is this structure something that the plugin relies on for being able to parse the code structure?

Can I deviate from this if I wanted to and omit the __all__ from __init__.py ?

[DavidVujic]

The __init__.py and the core.py can both be renamed or removed as you like. The thing that is expected is the actual namespace path. Polylith expects to find code in the bases and components folders, under a top namespace. Other than that, you can structure modules and subfolders as you like.

[DavidVujic]

(converting this into a GitHub discussion)

[DavidVujic]

I have seen the Polylith concept before but have not used it

Seeing you had this Poetry plugin and I had a new monorepo to structure I decided to give it a go

I am a bit confused about how to handle dependencies

I can see there is a root pyproject.toml and then individual ones for each project

Should I poetry add all deps at the root level and then use poetry poly sync to let the plugin apply only relevant deps to the project pyproject.toml files? i.e. I shouldn't manage the project files manually myself at all?

Hi!

The pyproject.toml at the workspace root is the one that is recommended to use during development. This is where I would recommend to create a virtual environment when developing. Yes, I would add the dependencies there!

The poetry poly sync is there to make it a little bit more convenient to include the bricks (your code, not the third-party dependencies). That command will analyze your source code and figure out what's missing in the project-specific toml files. But it will need to have a starting point defined in there already (the "base", where the entry point is of your app or service).

The sync command is totally optional, you can add the packages manually if you like!

Polylith for Python separates the bricks (the code in the components and bases folder) from third-party dependencies. The bricks is defined in packages of the [tool.poetry] section, and the third-party dependencies (libraries) are added as usual in the [tool.poetry.dependencies] section.

[DavidVujic]

The Clojure tool does it somewhere in-between of being a self-contained package and what we have here in Python 😄 There's the possibility to define the third-party libraries within a brick, but the projects (and development) still would pick all the needed bricks even if an individual brick would use another one internally.

[DavidVujic]

Maybe having a really simple pyproject.toml in a brick, or even a requrements.txt, as optionally defining the dependencies there would be a way to go? If each brick would be a real package, then they would need also to define the other bricks used in it (a brick can be a combination of several other bricks). 😅

[anentropic]

my gut feel is that would be a good idea

I think the bricks need to be defined enough in python packaging terms that you can specify one as a dependency in another brick and the tooling i.e. poetry is able to resolve the dependency graph automatically

[DavidVujic]

Good feedback, thank you! ⭐

I will think about this more and how to do this long-term. What I don't like is to have the bricks as all-in packages because they are meant to be small, a lot smaller than the traditional PyPI package (that is more of an entire feature packaged for reuse). A brick could be a single function, or wrapping some third-party library. But I think the Python tool should align more with the Clojure one about this thing, finding some way that would make sense in a Python context.

[BeRT2me]

This was a rather interesting discussion to read!

I agree with @anentropic that further guards against conflicting dependencies would be especially helpful to a system like this.

I'm trying, unsuccessfully, to think of a situation where I'd want dependencies defined in my development pyproject.toml to differ in version constraint from those in individual projects upon release. A big purpose is to make sure they all work together and don't conflict, no matter which pieces I'm using! Yet still have the ability to not have to include every possible dependency.

To me, poetry poly sync should be more of a true synchronization with the "main" development pyproject.toml as the source of truth. This means that project specific pyproject.toml should essentially be hands off besides adding their primary base package.

• If there are tool.poetry.source sections in the main pyproject.toml those get mirrored to projects.
• If a project's pyproject.toml is missing a dependency, try to auto-fill it in from the main pyproject.toml.
• If a project's pyproject.toml has a different dependency version than the main pyproject.toml, raise an error, or allow a --force flag to make it conform with the main pyproject.toml
• Shouldn't happen in this scenario, but if a project's pyproject.toml has a dependency not defined in the main pyproject.toml, it should have a similar error/--force response.

As another layer to this, having sync and/or check also look for unnecessary dependencies would be a nice feature.

• E.g, whenever we move on the py3.11 and tomli can be replaced with tomllib through a simple find/replace, having tomli be auto-removed would be quite helpful.