Reduce burden of package management in pyproject

[billcrook]

Hi David, I've been thinking about how this would work with larger monorepos and many components/bases. Instead of explicitly adding each component to the pyproject file, I'm wondering if the multiproject plugin could be enhanced to support globbing. For instance, instead of this:

[tool.poetry]
name = "my_example_project"
packages = [
         {include = "my_namespace/my_base", from = "../../bases"},
         {include = "my_namespace/another_base", from = "../../bases"},
         {include = "my_namespace/my_component", from = "../../components"},
         {include = "my_namespace/my_other_component", from = "../../components"}
         {include = "my_namespace/and_other_component", from = "../../components"}
         {include = "my_namespace/yet_other_component", from = "../../components"}
]

Something like the following would take the management burden out of pyproject:

[tool.poetry]
name = "my_example_project"
packages = [
         {include = "my_namespace/*", from = "../../bases"},
         {include = "my_namespace/*", from = "../../components"},
]

Taking it a bit further, I suspect this entire packages entry could be removed completely if the standard polylith structure was followed - no special configuration required.

Thoughts?

[DavidVujic]

It is possible to do something like that today, but with leaving out the "*" part. The tool should pick up components and bases from there when running the poetry build-project command. I haven't tested that part out, other than quickly creating a new project, add include = "my-namespace", from = "../../components" and inspect the built wheel.

I hesitate if this is a good thing, though, since all components and all bases will be included in a project, even if only few of them are used in the actual project. There's also the poly diff feature, that will help out determining which of the projects in a Polylith workspace need to be re-deployed: that is done by keeping track of changed components or bricks, and inspect where they are used. I believe including all of them will make it more difficult to automate the project build process.

I haven't run into any issues with manually adding dependencies for projects (other than occasionally forgetting about doing that), and was hoping that the tooling would be useful (with the libs, info and check commands). Do you think it is a problem, maybe for larger monorepos with many projects?

[billcrook]

I hesitate if this is a good thing, though, since all components and all bases will be included in a project, even if only few of them are used in the actual project

Yes, I agree at the polylith projects level this would be a bad practice. At the top-level the goal is to pull in all packages, so I think it makes sense there.

It is possible to do something like that today, but with leaving out the "*" part.

Neat, I'll give that a go when I finally get polylith set up on my project 😆 I'm doing a bit of repo merging and refactoring before I make the jump.

Do you think it is a problem, maybe for larger monorepos with many projects?

Potentially yes, at least for the top level. I'm looking at our common/utils package at work and I count ~50. That is without any further decomposition of code that should be split into components. On the flip side, maybe some would be rolled up together, hard to say. The codebase is ~4 years old but we are only a 2 person team. At a larger python shop I imagine it would get tedious to manage.

At the project level, for the reasons you mentioned, being explicit about each component is a good thing. If someone has hundreds/thousands of components in a project, maybe it's time for them to rethink the granularity of components.

thanks again

[DavidVujic]

Sounds like a big software project and it would be very interesting to hear your feedback how it will work out with Polylith! In general, those type of projects should fit well with the architecture. The Clojure community has some examples of very large scale projects: https://polylith.gitbook.io/polylith/conclusion/production-systems

I'm currently working on automation to add components and bases in the tooling for Python. I know that the team working on the Clojure version prefer manually adding it, their design choice is to have as little "magic" as possible. But since the Python version that I develop live in the Poetry context, where poetry add is modifying the pyproject.toml, I think it will make sense if a similar feature would be available here.

[billcrook]

Fantastic, looking forward to the tooling. Thanks for the link and I'll let you know how it goes.

cheers

[DavidVujic]

There's a new command: poetry poly sync and is introduced in v1.6.0. The command aims to help out keeping the workspace projects (including development) up to date with the used bricks. Here's the docs: https://davidvujic.github.io/python-polylith-docs/commands/#sync

I will also record a video shortly describing this command and how it can be used with the already existing ones for a nice developer experience 😄

[billcrook]

Great work