How to manage complexity in the project?

[markfinger]

Caveat: this largely a philosophical question that doesn't have any one correct answer

There are a couple of points that I think it's worth considering in general.

Should a project's complexity live in core, or should the core provide a flexible set of hooks that enable complexity to be externalised (plugins, ecosystem, etc)?

Should dependent packages live in the repo of the project that generated them (eg: the monorepo approach), or should they be distinct projects once they've matured and can be externalised?

[markfinger]

A common issue with build tools is the need to cover a lot of edge-cases. These cases accumulate, codebases become bloated, and development slows down. The usual solution is a 'plugin' system that enables complexity to exist externally, but still contribute to a tool's functionality.

---

Examples

If you look at webpack, it's a minimal core with a massive battery of plugins that perform most of the business logic. Personally, I think it's an approach that has more downsides than advantages, most pointedly: it's nightmarish trying to learn how anything works in webpack.

Browserify is a mostly opinionated core that covers the common use-case, but exposes a stream interface to intercept and change outputs. Implementation details aside, this approach is reasonably easy to understand.

---

In the context of this project, I'm very much inclined towards the most simple and basic solution, which is keeping everything in core, but ensuring that most behaviours can be changed or overridden.

[markfinger]

Unfort used to be a fairly large monorepo project that was broken up once it had stabilised. This was done with the intention that boundaries would be formed and cognitive overheads would be reduced.

In practice, some benefit has emerged. Specifically, externalising the packages does help to treat them as black boxes. However, a lot of negatives have also emerged:

• Cognitive overheads have increased as boundaries between packages reduces clarity and slows down development.
• Project maintenance has more overheads. Changes to infrastructure require the same change to be applied across multiple sites, rather than one. Ex: everytime a dependency breaks something, greenkeeper goes mental and sends me ~15 emails about the same problem.
• Refactoring code becomes more difficult as the boundaries reduce the ease of making improvements.

For a bit of broader context, this is one of many projects I've worked on that has been split up into many repos. These same issues have been repeated in every single case.

At this stage, it's probably easiest to leave things as they are. An alternative would be to bring each of the projects back into this repo. Another alternative would be to incrementally bring each across as needed.

[spalger]

I am a big fan of the monorepo approach. It's one I've been practicing in personal projects, but a core project at work has adopted it now and it's fantastic. We previously had module separation at the repo level and that resulted in the same issues you listed, but the worst issues were created by having multiple issue trackers, pull request queues, and wikis.

We still have module separation but simply implement it in the style of a lerna project. We self-police the separation of modules, only use the API documented in each module's README, and reference all top level modules directly by their name, not relative paths. Each module maintains it's own dependencies, and has it's own test suite. At the root of the repo we can execute lerna run test to run all of the tests, or just cd package/{name} and focus your efforts on a single module.

The best benefits, though, come from sharing a single git history. Each release results in a version bump across all packages and is represented by a single git tag. Pull requests and feature branches are developed against a specific version of the entire stack, rather than requring a special commit/branch from this or that dependency in order to test out functionality, and we have a single issue tracker, pull request queue, and wiki for the entire stack.

I'm not sure how this works with greenkeeper though, in order for some of this to work each module needs to have it's own package.json and dependencies.

[spalger]

Not sure how to format this response, so I'm just going to get listy

Thoughts about plugins in webpack:

• I have thought a lot about better ways to do loaders, but the general idea makes more then anything I was able to come up with
• While simple to understand from the outside, properly implementing a loader is tricky work and I think that is the reason sorka is/must be so involved in plugin maintenance
• Non-loader plugins for webpack are able to change the webpack compiler in so many ways it's insane, but I think the interface could be improved a lot if desired.
• It's almost impossible to know when you write a webpack plugin that is correct

More general thoughts

• Streams are great, and people love (and importantly, understand) their gulpfiles, but I feel like that would get in the way of Thread-based workers #110
• Error handling and debugging in streams is super gnarly
• Powerful/Raw job extensions and overloading seems like a good first step

[markfinger]

monorepo + greenkeeper

Yeah, monorepo sounds like the way to go.

greenkeeperio/greenkeeper#139 is still pending. In the short term, it's probably easiest to simply turn greenkeeper off and go back to manual updates.

loaders/plugins

Yeah, all your points are pretty accurate.

Powerful/Raw job extensions and overloading seems like a good first step

Definitely, agreed.

[rarkins]

@markfinger I've just released an open source tool for maintaining package.json versions via Pull Requests, which could be of help? https://github.com/singapore/renovate/

It supports specifying multiple package.json locations per-repository (i.e. monorepo architecture).