Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KYUUBI #6979] Support check paimon system producers #6980

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

davidyuan1223
Copy link
Contributor

@davidyuan1223 davidyuan1223 commented Mar 12, 2025

Why are the changes needed?

Cuurently, ranger check missing paimom system producers command, need to support these command

  1. create_tag
  2. delete_tag
  3. rollback

#6979

PS: There has a question about paimon, paimon'sparkCatalog need the currentCatalog Env is the paimon's catalog, use default spark_catalog will throw exception, maybe we should add this hint to the documentation.
such as
If you wanna support producers check with paimon, you need use sql use $paimon_catalog to ensure the session currentCatalog is paimon_catalog

PS: paimon-spark-3.3:0.8.2 has some compaitable question, suggest upgrade the paimon version

How was this patch tested?

producers test cases

  1. create_tag
  2. delete_tag
  3. rollback

Was this patch authored or co-authored using generative AI tooling?

No

@codecov-commenter
Copy link

codecov-commenter commented Mar 12, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (d0d24cd) to head (57aac60).

Additional details and impacted files
@@          Coverage Diff           @@
##           master   #6980   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         693     693           
  Lines       42723   42723           
  Branches     5813    5813           
======================================
  Misses      42723   42723           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pan3793 pan3793 added this to the v1.11.0 milestone Mar 12, 2025
@pan3793
Copy link
Member

pan3793 commented Mar 14, 2025

@davidyuan1223 the failure looks related, could you please fix it?

- Producers *** FAILED ***
  Expected exception org.apache.kyuubi.plugin.spark.authz.AccessControlException to be thrown, but java.lang.NoSuchMethodError was thrown (PaimonCatalogRangerSparkExtensionSuite.scala:553)

@davidyuan1223
Copy link
Contributor Author

@davidyuan1223 the failure looks related, could you please fix it?

- Producers *** FAILED ***
  Expected exception org.apache.kyuubi.plugin.spark.authz.AccessControlException to be thrown, but java.lang.NoSuchMethodError was thrown (PaimonCatalogRangerSparkExtensionSuite.scala:553)

Ok

@pan3793
Copy link
Member

pan3793 commented Mar 17, 2025

paimon-spark-3.3:0.8.2 has some compaitable question, suggest upgrade the paimon version

@davidyuan1223 so we can upgrade Paimon to bring Spark 3.3 support back? If so, let's upgrade Paimon first

@davidyuan1223
Copy link
Contributor Author

paimon-spark-3.3:0.8.2 has some compaitable question, suggest upgrade the paimon version

@davidyuan1223 so we can upgrade Paimon to bring Spark 3.3 support back? If so, let's upgrade Paimon first

apache/paimon#3465

https://github.com/apache/paimon/blob/release-0.9/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/analysis/PaimonProcedureResolver.scala

i think we could upgrade the version to 0.9, from this branch, paimon has support the case compatibile

@pan3793
Copy link
Member

pan3793 commented Mar 18, 2025

i think we could upgrade the version to 0.9, from this branch ...

@davidyuan1223 please go ahead

# Conflicts:
#	extensions/spark/kyuubi-spark-authz/src/test/scala/org/apache/kyuubi/plugin/spark/authz/ranger/PaimonCatalogRangerSparkExtensionSuite.scala
@davidyuan1223
Copy link
Contributor Author

i think we could upgrade the version to 0.9, from this branch ...

@davidyuan1223 please go ahead

hello, pan, when i try to update the paimon version, it will report error, i found same issue #6634
Maybe we sould keep the paimon version, i found its spark-extensions version > 3.5, which has some compitable exceptions with the spark3.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants