Skip to content

[FLINK-34467] add lineage integration for jdbc connector #149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[FLINK-34467] add lineage integration for jdbc connector #149

wants to merge 1 commit into from

Conversation

HuangZhenQiu
Copy link
Contributor

Add native lineage support for JDBC connector. It is an ongoing effort. The PR is not ready for review.

@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-34467-jdbc-lineage branch 7 times, most recently from f1f5587 to d468795 Compare December 3, 2024 04:03
@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-34467-jdbc-lineage branch from d468795 to 231ad7d Compare December 4, 2024 05:55
@davidradl davidradl mentioned this pull request Dec 9, 2024
Closed
@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-34467-jdbc-lineage branch from 231ad7d to 17c6ad8 Compare December 10, 2024 03:54
@davidradl
Copy link
Contributor

@HuangZhenQiu I notice your comment "The PR is not ready for review.", I wonder if you could turn this PR into a draft until it is ready for review - thanks.

@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-34467-jdbc-lineage branch from 17c6ad8 to 4719928 Compare December 15, 2024 03:47
@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-34467-jdbc-lineage branch from 4719928 to 2d0ba42 Compare December 31, 2024 04:45
pawel-big-lebowski
pawel-big-lebowski reviewed Jan 3, 2025
View reviewed changes
...jdbc-core/src/main/java/org/apache/flink/connector/jdbc/lineage/DefaultTypeDatasetFacet.java

/** Default implementation of {@link TypeDatasetFacet}. */
@PublicEvolving
public class DefaultTypeDatasetFacet implements TypeDatasetFacet {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At some point this should be moved to flink-core but I understand why this hasn't been done prior to this PR.

...src/main/java/org/apache/flink/connector/jdbc/db2/database/lineage/Db2LocationExtractor.java Outdated
public class Db2LocationExtractor implements JdbcLocationExtractor {

private JdbcLocationExtractor delegate() {
return new OverrideJdbcLocationExtractor("db2");
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wouldn't it be better to have delegate as class member instead of creating it each call?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense.

...a/org/apache/flink/connector/jdbc/cratedb/database/lineage/CrateDBLocationExtractorTest.java Outdated
* limitations under the License.
*/

package org.apache.flink.connector.jdbc.cratedb.database.lineage;
Copy link

@pawel-big-lebowski pawel-big-lebowski Jan 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it make more sense to implement those classes within openlineage-sql-java package?
It's kind of a namespace convention thing, and implementing this a generic way, which is reusable across the integrations, can be a better solution. what do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. Added extractors to OpenLineage.

...-connector-jdbc-core/src/main/java/org/apache/flink/connector/jdbc/lineage/LineageUtils.java Outdated
return nameOf(simpleJdbcQueryStatement.query());
}

public static Optional<String> nameOf(String query) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please specify what name is this. Dataset name, job name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

table name.

@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-34467-jdbc-lineage branch from 2d0ba42 to 032eab5 Compare January 20, 2025 04:31
@davidradl
Copy link
Contributor

Add native lineage support for JDBC connector. It is an ongoing effort. The PR is not ready for review.

@HuangZhenQiu It would be worth changing this text in the PR, as I think the PR is ready for review.

@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-34467-jdbc-lineage branch 4 times, most recently from 87215fe to e81d3fc Compare April 20, 2025 04:17
@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-34467-jdbc-lineage branch 4 times, most recently from 0d09333 to c1bfe4b Compare April 27, 2025 03:50
@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-34467-jdbc-lineage branch 3 times, most recently from aa889f0 to 996b8f5 Compare April 28, 2025 04:07
@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-34467-jdbc-lineage branch 7 times, most recently from 781af4f to f9a2c92 Compare May 5, 2025 05:12
@HuangZhenQiu HuangZhenQiu force-pushed the FLINK-34467-jdbc-lineage branch from f9a2c92 to 95b92a3 Compare May 27, 2025 03:18
@lvyanquan lvyanquan self-assigned this May 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pawel-big-lebowski pawel-big-lebowski pawel-big-lebowski approved these changes

Assignees

@lvyanquan lvyanquan

Labels
component=BuildSystem component=JDBC/Core component=JDBC/CrateDB component=JDBC/DB2 component=JDBC/MySql component=JDBC/OceanBase component=JDBC/Oracle component=JDBC/Postgres component=JDBC/Shaded component=JDBC/SqlServer
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@HuangZhenQiu @davidradl @pawel-big-lebowski @lvyanquan