Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not share FeedContext between Harvesting Utils and Broker Microservice #19

Open
lukehesluke opened this issue Feb 4, 2025 · 0 comments

Comments

@lukehesluke
Copy link
Contributor

A huge amount of work was done to decouple Harvesting Utils and Broker Microservice in #9.

However this issue still remains.

Do not share FeedContexts

Harvesting Utils has a FeedContext object, which it uses to track harvesting progress (perhaps it should even be renamed HarvestProgress?)

Broker also wants to tracks harvesting progress — so that it can render this information in its /status route — as well as other bits of data (like a multibar progress bar, etc) for each feed. It does this by:

  1. Having its own FeedContext type, which is the same as Harvesting-Utils', and also adds extra fields
  2. Before initiating the harvest of a feed, Broker initiates its own context and then passes that context to Harvesting-Utils using the overrideContext arg.

The result is that this one object is updated by both Harvesting-Utils and Broker. This is overly coupled design and means that changes to Harvesting-Utils could break Broker and vice versa.

Suggested Fix

  1. Harvesting-Utils: Removes the overrideContext arg (which is already marked as deprecated); AND no longer exposes the createFeedContext(..) function.
  2. Harvesting-Utils: Include a copy of its internal context into the various callbacks (e.g. processPage(..))
  3. Broker: Maintains its own FeedContext type, which is no longer derived from Harvesting Utils
  4. Broker: In the callbacks that it sends to harvestRPDELossless(..), it should use the context object that Harvesting-Utils shares (see step 2) to update its own context with the progress info that it needs for its progressbar and its /status endpoint.

ALSO:

  • Move totalItemsQueuedForValidation, validatedItems, sleepMode, timeToHarvestCompletion out of Harvesting-Utils' FeedContext definition, and into Broker's. These fields are only relevant to Broker.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant