Skip to content

[core] (work in progress) Iceberg REST #5621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 11 commits into from

Conversation

nickdelnano
Copy link
Contributor

@nickdelnano nickdelnano commented May 19, 2025

Issue: #4394

EDIT:

This PR was created to investigate Iceberg REST and is incomplete. See the comments on #4394 for up to date information.

Design decisions

  1. Use Iceberg RESTCatalog - the APIs are different than Paimon's REST Catalog and it allows to use the existing auth providers in the Iceberg repo.

My personal motivation is to get Iceberg REST working with the REST catalog servers provided by Apache Polaris and AWS Glue but I also want to support the community's interests.

@nickdelnano nickdelnano marked this pull request as draft May 19, 2025 00:21
@ParameterizedTest
// TODO add other file types back
@ValueSource(strings = {"avro"})
public void testIcebergRESTCatalog(String format) throws Exception {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is an example. It does not pass.

tEnv.executeSql(
"CREATE CATALOG iceberg WITH (\n"
+ " 'type' = 'iceberg',\n"
+ " 'catalog-type' = 'rest',\n"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Iceberg rest catalog

@JingsongLi
Copy link
Contributor

Thanks @nickdelnano for the proposal! +1 to your analysis, implement an IcebergRESTMetadataCommitter looks good to me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants