Skip to content

refactor: Move view and stream from datasource to catalog, deprecate View::try_new #15260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Mar 19, 2025

Conversation

logan-keede
Copy link
Contributor

Which issue does this PR close?

Rationale for this change

view and stream are TableProviders, catalog seems like the proper place for them considering their Table Providers and their dependency.

What changes are included in this PR?

just refactor of both.
I left test view at its original place for now.

Are these changes tested?

Testing by Github CI.

Are there any user-facing changes?

Nope, I have kept backward compatibility.

@github-actions github-actions bot added core Core DataFusion crate catalog Related to the catalog crate datasource Changes to the datasource crate labels Mar 16, 2025
@github-actions github-actions bot removed the datasource Changes to the datasource crate label Mar 16, 2025
@logan-keede logan-keede marked this pull request as ready for review March 16, 2025 20:53
Copy link
Contributor

@m09526 m09526 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks all good!

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @logan-keede -- this looks really nice

I have a few questions / comments about avoiding the datafusion-optimizer dependency -- let me know what you think

datafusion-execution = { workspace = true }
datafusion-expr = { workspace = true }
datafusion-optimizer = { workspace = true }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be really nice to avoid the new dependency on the datafusion-optimizer crate here --

However, it seems like it is related to the fact that coercion is part of the optimizer crate 🤔

I can think of two ways to avoid this:

  1. Make the caller of ViewTable::try_new call coerce_types directly
  2. Move the coercion into a new crate (seems overkill)

But the more I think about this the less of a problem I think it is

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have moved it to the only caller SessionContext, though I have a concern if there is a use case where someone might not want to use SessionContext but use ViewTable directly. since they will not get default Coercion now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can perhaps add a note to the upgrade guide

The other thing we could do is add a new function and deprecate the old one, so the compiler would tell all users when they had upgrade 🤔 I will try and give this a shot

@github-actions github-actions bot added the physical-expr Changes to the physical-expr crates label Mar 18, 2025
@alamb alamb changed the title refactor: Move view and stream from datasource to catalog refactor: Move view and stream from datasource to catalog, deprecate View::try_new Mar 18, 2025
@github-actions github-actions bot added the proto Related to proto crate label Mar 18, 2025
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @logan-keede -- I pushed a commit to deprecate View::try_new and renamed the apply_required_rule

Copy link
Member

@xudong963 xudong963 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you

@xudong963 xudong963 merged commit 3dd3d73 into apache:main Mar 19, 2025
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
catalog Related to the catalog crate core Core DataFusion crate physical-expr Changes to the physical-expr crates proto Related to proto crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants