Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Importer restructure #5624

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Conversation

semohr
Copy link
Contributor

@semohr semohr commented Feb 9, 2025

Description

Hello y'all, when working on the importer.py file in a previous PR I noticed that this file grew quite large and badly needs a restructuring. Restructuring should improve our ability to apply changes to it in the future and isolate sub-functionalities within the importer.

Overview

For now I only changed the structure keeping the code (mostly) unchanged.

I split the functions and classes in the importer.py into the following responsibilities:

  • importer/session.py : Includes the ImportSession class.
  • importer/stages.py : Includes all stage functions, I prefixed the helper functions with a _ to allow distinguishing between stages and helper functions more easily.
  • importer/state.py : Includes the logic for the ImportState handling i.e. the resume feat.
  • importer/tasks.py : Includes the ImportTask class and all derived classes. Also includes the Action enum which I have renamed from action.
  • importer/__init__.py : Identified all public facing classes and functions and added them to __all__

Potential future changes

I don't want to add this to this PR but there are some places here where I see possible improvements for our code:

  • There are quite some config parsing related functions in the ImportSession which could be isolated (see e.g. set_config, want_resume). Maybe a mixin class which handles the config operations could be useful?
  • The ImportSession should be abstract if it is not used directly (I think it shouldn't). The function definitions which raise NotImplemented errors are quite weird imo and could be avoided by making the class abstract.
  • For me it was difficult to understand the flow of the importer as stages call session function and it is not clear which function is called by which stage and when. Maybe a naming convention for the stage functions in conjunction with the session methods could help here. Not sure how this will look in practice but right now it is quite hard to follow imo. Alternatively splitting the session into a outfacing session and a session context which is passed to the stages could help.
  • The use of the stage decorator is highly inconsistent. Maybe a better way to handle the stages could be found. This is more of a pipeline related issue and not directly related to the restructuring but I think it is worth mentioning.
  • Similar to the ImportSession, I think the ImportTask should be abstract as well, maybe we can put a bit more thought into the task hierarchy. This might also automatically improve the flow of the importer pipeline.

Am happy to tackle some of these issues in future PRs if you also think they are worth it.

Best,
Sebastian

Note: This PR is based on #5611 and can only be merged once the typing additions are accepted.

@snejus snejus requested a review from Copilot March 20, 2025 21:30
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR restructures the importer code by splitting the large importer.py into modular files to isolate different responsibilities and simplify maintenance.

  • Split core importer functionalities into dedicated modules: state, session, stages, and tasks.
  • Updates the public API in the init.py to reflect the new structure.
  • Updates tests to reference the relocated modules.

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

File Description
beets/importer/state.py Introduces ImportState with state persistence using pickle.
beets/importer/init.py Re-exports public classes and functions, updating API exposure.
beets/importer/session.py Implements ImportSession with pipeline setup and session logic.
test/test_importer.py Updates import paths to reflect module restructuring.
Comments suppressed due to low confidence (3)

beets/importer/state.py:92

  • Consider using proper logging formatting (e.g., %s) or passing 'exc_info=True' to ensure exceptions are logged correctly.
            log.debug("state file could not be read: {0}", exc)

beets/importer/session.py:260

  • Creating a new ImportState instance during each element check may lead to performance inefficiencies or inconsistent state; consider reusing an existing instance instead.
            [ImportState().progress_has_element(toppath, p) for p in paths]

beets/importer/session.py:182

  • [nitpick] Using 'is' for enum comparisons may lead to unexpected behavior; consider using '==' when checking for equality.
            elif task.choice_flag is (Action.SKIP):

Comment on lines +105 to +106
log.error("state file could not be written: {0}", exc)

Copy link
Preview

Copilot AI Mar 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider using proper logging placeholders (e.g., %s) or including 'exc_info=True' to log exception details more effectively.

Suggested change
log.error("state file could not be written: {0}", exc)
log.error("state file could not be written: %s", exc, exc_info=True)

Copilot is powered by AI, so mistakes are possible. Review output carefully before use.

Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant