poly info command does not list components under lower level namespaces #95
Replies: 11 comments 6 replies
-
|
Thank you, very interesting ideas @hugobettmach! Today the |
Beta Was this translation helpful? Give feedback.
-
|
This could possibly be a new How are the bricks included in the pyproject.toml files? |
Beta Was this translation helpful? Give feedback.
-
|
For now we are just testing the new setup so we follow your recommendation: But could also be nice to have some thing like include all bricks from a sub-namespace, e.g.: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! Edit, similar discussion here: #89 (comment) |
Beta Was this translation helpful? Give feedback.
-
|
I'm going to give the theming a try, but I would recommend you to reconsider the sub-namespacing into a flat structure as suggested in the If you still want the sub-namespacing, with the current tooling (while waiting for a future version to support it), you could do something like this: put all your bricks in the Example: create a new "component" (i.e. your sub-namespace): poetry poly create component --name my_namespace_1Import the bricks that should belong to this sub-namespace in the # the __init__.py file
from org_name import component1, component2
__all__ = ["component1", "component2"]When using the bricks in a sub-namespace, you could import it like this: # some other module using the bricks in a sub-namespace
from org_name.my_namespace_1 import component1
...The pyproject.toml would need to add all components needed + the new namespace: packages = [
{include = "org_name/component1", from = "components"},
{include = "org_name/component2", from = "components"},
{include = "org_name/my_namespace_1", from = "components"},
...
] |
Beta Was this translation helpful? Give feedback.
-
|
I understand that you might want bricks with the same name, but in different in sub-namespaces. This could be accomplished in the suggestion above, but with an alias: # the __init__.py file of namespace_1
from org_name import some_brick_doing_X as component1
from org_name import some_brick_doing_Y as component2
__all__ = ["component1", "component2"]# the __init__.py file of namespace_2
from org_name import some_other_brick_doing_X as component1
from org_name import some_other_brick_doing_Y as component2
__all__ = ["component1", "component2"]When importing the namespace, it expose the alias: # import the component via the namespace, that has been "renamed" (alias)
from org_name.namespace_1 import component1 |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @DavidVujic for your suggestions. This may be all we need :) |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your excellent work. Although using alias is a good workaround, is there a plan to support namespace in future? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @lhttjdr! Currently, I'm not focusing on that part. The general recommendation for Polylith is to have a fairly flat structure: a shared top namespace, and one sub-namespace (that itself can have any number of sub-namespaces/folders). The combination of a top namespace and the upper most sub-namespace becomes a brick, top_ns/brick. You can have it namespaced with several sub-namespaces, but Polylith will only recognize the upper most one as a brick. From a functionality aspect it shouldn't be any issues with it. |
Beta Was this translation helpful? Give feedback.
-
|
Here's another usage for sub-namespaces. Our small team maintains 100+ python repos. We have a few different types of repos:
As you can guess, it has gotten difficult to maintain the code base. We want to move everything to a single monorepo. There are many ways to do it but we like Polylith. The only "problem" we see with Polylith is that we will have 100+ components (and counting). We would like to nest the components so people don't need to navigate 100+ subdirectories to do their work. Some people on the team work primarily on data science bricks. Some focus on reusable libraries. Others focus on microservices and Dash apps. Rather than overwhelming everyone with 100 bricks every time they look at the workspace, it would be nice if components could be organized into sub-namespaces, such as: Sub-namespaces can be superficial - visual candy that makes it easier to navigate the code base. We already ensure that each component (i.e. repo) uses a unique namespace. We will continue to do that in a monorepo regardless of how the components are organized on disk. Different people on the team tend to focus on different types of components. Data scientists focus on I think I understand the previous comments: the multi-project poetry plugin will function properly but some features of the polylith plugin won't work because it won't find components in subnamespaces - I ran into that a few days ago. We'd really like to take advantage of the full capabilities of the polylith plugin. It would be helpful to visualize the bricks and sync dependencies. Any advice or guidance would be appreciated. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
@DavidVujic It seems to me that polylith can only scale so far without support for sub-namespaces because workspaces with dozens or hundreds of components will get more and more difficult to work with. It would be helpful to see an example. Do you know of any polylith repos with many components? |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your feedback, much appreciated!I think I understand the reasoning about sub-namespaces, and that they should be represented as grouped polylith components. The grouping sounds much like domains, or entry points for the artifacts to build. The way Polylith views code is that it should be possible to share code across all domains and artifacts. That’s why the architecture promotes a flat component structure.
Otherwise it might not be that straightforward to share code between the “flask” domain and the “analytics” domain. It’s at least a risk that the code within the domain is hidden within the domain itself.
Along with components, there’s also bases in Polyllith. Maybe the groups in your example would fit as bases, with the code behind it as components?I know of some monorepos with hundreds of components that work really well with Polylith. It’s built for those kind of scenarios.
Here’s some examples of production systems:Production systems — polylith/clj-poly 0.2.19cljdoc.org
(Writing this from my phone, because I’m away from keyboard for a couple of days)
/David
|
Beta Was this translation helpful? Give feedback.
-
|
Another try, link to Production systems:
https://cljdoc.org/d/polylith/clj-poly/0.2.19/doc/production-systems
(Sorry about ha poorly formatted response)
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks @DavidVujic! |
Beta Was this translation helpful? Give feedback.
-
|
@hstravis When I look at your examples again, it seems very much like some of those would be Polylith bases (i.e. entry points), and I would suggest that you have a look at that. I’ll gladly help out if you have any questions or more feedback!
The source code you would have behind the bases would very much fit as small reusable components. I don’t think 100+ components will be a problem for the developers. Usually it is the opposite: quickly identifying features to use.
The link to production systems I posted before includes a couple of huge monorepos with a lot of bricks (bases and components) and a lot of projects (the artifacts to deploy).
(I realized replying from the email client doesn’t seem to handle responses in threads)
|
Beta Was this translation helpful? Give feedback.
-
I spent more time re-organizing the code base. I realized the code can be grouped into shared buckets such as utility functions, data schemas, database repositories, web clients, and service layers (biz logic). Each bucket corresponds to a single component. We also need an app-specific component for each app because there are multiple bases per app - need a place for code shared between app-specific bases. Now there are ~10 components instead of ~100. Relatively easy to navigate and it should encourage contributors to follow the established pattern. This is better but it is a more disruptive change. Almost all of the code must be refactored. But we are just changing namespaces, there should be no functional changes, and we have adequate test coverage. We will probably run into problems where poor design choices were made - it might be difficult to decouple some things that should not be coupled. |
Beta Was this translation helpful? Give feedback.
-
|
I would love to hear more about your progress, feedback and decisions along the way. Let’s keep in touch!
|
Beta Was this translation helpful? Give feedback.
-
Is your feature request related to a problem? Please describe.
Hi @DavidVujic, first of all this is such a great project, this way of organising python code makes so much sense in many situations.
The issue I came across is related to using multiple namespaces, under the top level namespace (or org_name) as mentioned in #45. This is something I think could be very useful for my use case.
But, when creating something like the example below...
Using
poetry poly create component --name=namespace_1/component_1... And then runningpoetry poly info, I get:Describe the solution you'd like
Would be nice
poly infocould list all the components detected under "sub namespaces". Same would apply to bases as well. For example, would like a summary something like the one below. Hopefully this makes sense?Additional context
Following on #45.
Beta Was this translation helpful? Give feedback.
All reactions