Make site generation use multiprocessing #1709

bartfeenstra · 2024-07-14T17:49:07Z

This fixes #1516

To do

Add betty.typing.processsafe() which implies betty.typing.threadsafe()
Allow reduced (pickleable) values to implement a factory interface so they can he rehydrated after unpickling
Do all pickling in one go so we don't risk pickling the same things twice
Remove caches from job contexts (because memory caches cannot be multiprocess-safe), but introduce a new lightweight alternative, some kind of registry, just to do bookkeeping? This is also partly why performance right now is sub-optimal, as the image filters use this duplicates memory cache and therefore end up duplicating work between processes.
ACTUALLY caches were designed so cache item values MUST be pickleable.... This should be documented.
We should allow extensions to be 'pickled' as well, in order to share services between processes. For the Wikipedia extension, this would be its Wikipedia API rate limiter. Document this on the Extension class.
Allow KeyboardInterrupt to gracefully shutdown the executor
Should Jinja2 filters use the job pool?
do we still need App's process pool then?
Can we extract the pool functionality into a separate API/submodule? Even if we don't reuse this now, splitting the pool code from site generation reduces complexity and makes for easier testing. Update: we've now run into this in practice, where we want to mock something (testing third-party entity types), but obviously those mocks don't carry over to the processes in the pool. Separating the pool from the generation API means that we can test the pool fully (not needing any Project or Ancestry related mocking), site generation fully (by running the relevant tasks in the main process), and integration (by fully testing integration points and nothing more, because this is the part where conflicts occur)
if we embed the pool in the job context, the context can do bookkeeping (prevent duplicates) for every task if needed (opt-in?)

betty/cache/_base.py

betty/generate/pool.py

betty/generate/__init__.py

bartfeenstra · 2024-07-14T19:52:25Z

betty/concurrent.py

@@ -81,6 +82,13 @@ async def acquire(self, *, wait: bool = True) -> bool:
    def release(self) -> None:
        self._lock.release()

+    @classmethod
+    def multiprocessing(cls) -> Self:


betty/generate/__init__.py

bartfeenstra · 2024-08-13T18:09:08Z

betty/generate/pool.py

+
+class _GenerationProcessPool:
+    """
+    Set up a worker process, before the worker starts performing tasks.


Incorrect doc?

betty/__init__.py

betty/_patch.py

betty/cache/_base.py

betty/concurrent.py

bartfeenstra · 2024-10-01T11:17:44Z

betty/model/collections.py

@@ -41,9 +41,6 @@ class EntityCollection(Generic[_TargetT], ABC):

    __slots__ = ()

-    def __init__(self):


Clean this and similar initializers in a separate PR

bartfeenstra added enhancement New feature or request python Pull requests that update Python code performance labels Jul 14, 2024

bartfeenstra added this to the 0.4.x milestone Jul 14, 2024

bartfeenstra force-pushed the generate-multiprocessing branch from 429b33f to 5925ef9 Compare July 14, 2024 18:29