⚡ migrate to uv for packaging #23814
Conversation
danielgafni
force-pushed
the
uv-migration
branch
from
36090c1 to
026877d
Compare
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @danielgafni and the rest of your teammates on |
|
Feel free to ping me if you need anything. We're very interested in the workspace concept working for Dagster. |
|
Deploy preview for dagster-docs ready! Preview available at https://dagster-docs-23g2vsazm-elementl.vercel.app Direct link to changed pages: |
|
Made some progress with this: migrated all packages to Current problem: |
|
Ah yes the resolver needs to solve for all extras in the workspace. Let me check-in with the team to see if we have a recommended workaround. |
|
Hey @zanieb, just checking, are there any updates on this? I understand that mutually exclusive dependency groups is not a trivial feature to implement. Do you have plans for this at all? If not, is there a recommended workaround? |
|
@danielgafni we've been talking about this quite a bit internally, especially in the context of PEP 735 — I think we'll try to implement something for this in the near future but yeah it's hard. I don't think we have a good workaround at the moment, but I'll check with the rest of the team. |
|
There's a tracking issue at astral-sh/uv#6981 |
This PR adds support for conflicting extras. For example, consider
some optional dependencies like this:
```toml
[project.optional-dependencies]
project1 = ["numpy==1.26.3"]
project2 = ["numpy==1.26.4"]
```
These dependency specifications are not compatible with one another.
And if you ask uv to lock these, you'll get an unresolvable error.
With this PR, you can now add this to your `pyproject.toml` to get
around this:
```toml
[tool.uv]
conflicting-groups = [
[
{ package = "project", extra = "project1" },
{ package = "project", extra = "project2" },
],
]
```
This will make the universal resolver create additional forks
internally that keep the dependencies from the `project1` and
`project2` extras separate. And we make all of this work by reporting
an error at **install** time if one tries to install with two or more
extras that have been declared as conflicting. (If we didn't do this,
it would be possible to try and install two different versions of the
same package into the same environment.)
This PR does *not* add support for conflicting **groups**, but it is
intended to add support in a follow-up PR.
Closes #6981
Fixes #8024
Ref #6729, Ref #6830
This should also hopefully unblock
dagster-io/dagster#23814, but in my testing, I
did run into other problems (specifically, with `pywin`). But it does
resolve the problem with incompatible dependencies in two different
extras once you declare `test-airflow-1` and `test-airflow-2` as
conflicting for `dagster-airflow`.
NOTE: This PR doesn't make `conflicting-groups` public yet. And in a
follow-up PR, I plan to switch the name to `conflicts` instead of
`conflicting-groups`, since it will be able to accept conflicting extras
_and_ conflicting groups.
|
@danielgafni heads up — we just released support for declaring conflicting extras. |
|
You guys rock! |
Summary & Motivation
Internal discussion: https://github.com/dagster-io/internal/discussions/11123
How I Tested These Changes
Changelog [New | Bug | Docs]