Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use new narrow requirement selection resolvelib api to reduce cost of resolution #13253

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

notatallshaw
Copy link
Member

@notatallshaw notatallshaw commented Mar 1, 2025

resolvelib 1.1 introduced a new API for providers: narrow_requirement_selection.

narrow_requirement_selection is very similar to get_preference but it is only called once per round whereas get_preference is called for the number of unsatisfied requirements there are in that round, which can be arbitrarily large.

In short narrow_requirement_selection allows the provider to narrow what requirements, by their identifiers, are considered for get_preference, reducing the cost of getting a preference, if it only returns 1 identifier then calling get_preference is skipped altogether.

Pip has two obvious preferences in it's current get_preference that can be moved to narrow_requirement_selection:

  • REQUIRES_PYTHON_IDENTIFIER - Should always be considered immediately, this skips checking all other preferences if this identifier is present
  • Backtrack Cause - Currently if there are a large number of unsatisfied names (n), and a large number backtrack causes (b) then getting a preference costs O(n*b), there have been reports that this can dominate the resolution time, by moving this to narrow_requirement_selection identifying backtrack causes costs O(n+b) and the cost is completely removed in getting the preference

@notatallshaw notatallshaw force-pushed the Use-new-`narrow_requirement_selection`-resolvelib-API-to-speed-up-resolution branch from 6040f02 to 0f6c25f Compare March 1, 2025 16:38
@notatallshaw notatallshaw added this to the 25.1 milestone Mar 1, 2025
Copy link
Member

@pfmoore pfmoore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One typo in a variable name.

Note - I've only desk checked this, I don't have any useful examples of resolutions to test it against.

@notatallshaw
Copy link
Member Author

Note - I've only desk checked this, I don't have any useful examples of resolutions to test it against.

Unfortunately the user who raised #10621 never provided a reproducible example, which was the initial report that led to making this API.

But, I will test this branch against all scenarios in Pip-Resolution-Scenarios-and-Benchmarks, which includes popular resolutions and known difficult resolutions, before merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants