Distributed system project structure: multi-base project? Nested projects?

[balladeer]

Hi there! I came across this body of work after searching "rye monolith" and am enjoying my exploration of it so far.

I'm exploring how to apply the architecture to a larger system / project and wondered where I could find the state of the art: how would you structure the bases + projects for a system w/ many deployable pieces that are assembled together with docker-compose?

---

Assume a small distributed system:

• Reverse proxy (nginx | traefik)
• "Main" web app (fastapi)
• Authentication server (standalone go executable)
• Authentication self-serve UI (fastapi)
• Prometheus (pre-built docker image)
• Grafana (pre-build docker image)

My current Polylith knowledge suggests that:

• I put the fastapi pieces in a Polylith project and deploy via Docker
• I put the standalone, non-python pieces in Polylith projects with just Dockerfiles to define their artifacts
• I make an "assembly" project that operates on the build artifacts of the other projects and layers docker-compose on top.

However, I could also imagine:

• each 'instance' of this distributed system gets a single Polylith project
  • that project contains references to both fastapi bases, as well as the non-python Dockerfiles and the docker-compose.yml

Does either approach fit better with the intended usage of Polylith? Is the inclusion of non-Python 'projects' likely to cause issues?

[balladeer]

Ok, so:

• the official Polylith docs FAQ suggest mixing programming languages is not recommended: better to have one workspace per language (presumably with language-specific tooling).
• Using this repo, a project must have at least one brick or the build fails with Unable to determine which files to ship inside the wheel, which makes a ton of sense: can't build a python package w/o telling hatchling what to build.

Taken together, it seems that to get the most out of the Polylith architecture in a multi-language environment, you'd really want to have Polylith tooling for each language. (In my case, I'm only developing in Python, but there is a good amount of non-python config / packaging to do, and that work doesn't feel very congruent with Polylith.)

[DavidVujic]

Hi @balladeer, I'm happy that you have found Polylith!

I'm curious about the non-python things: how would you package them in a non-Polylith context?

[balladeer]

I prepare the non-Python things with their own Dockerfiles, typically to combine an executable binary with some config or initialization info stored as yaml / json files. So everything (python + non-python) ends up as its own Docker image, ready to get orchestrated by docker-compose.

In the example above, I'd have six Dockerfiles that each make a sub-system docker image (e.g. one for the main web app, one for the reverse proxy, etc) and then use docker-compose to make sure they all talk to each other as intended.

[DavidVujic]

I think that would work well in a Polylith setup too! You can put the non-Python things where you want and run the setup scripts for them as today, and for the Python code you can build wheels using Rye, Hatch, PDM or Poetry. If I understand it correctly, the non-python things aren't included in the Python wheels, right?

[balladeer]

You are correct re: non-python stuff staying out of wheels.

I'll give this a go and see what comes of it -- thanks for the responses!