Trino 473 Unable to Query Hudi 1.0.x Tables (COW and MOR) - GENERIC_INTERNAL_ERROR

[NICK-T-D]

Hi Trino community,

I am encountering an issue with Trino 473 when attempting to query Hudi tables (both COW and MOR types) after upgrading Hudi from 0.14.0 to 1.0.1. The same tables can be queried successfully in Apache Flink with Hudi 1.0.1 and in Trino with Hudi 0.14.0, but they fail in Trino 473 with Hudi 1.0.1, resulting in a GENERIC_INTERNAL_ERROR. I would appreciate your assistance in diagnosing and resolving this issue.

Environment

• Trino Version: 473

• Hudi Version: 1.0.1 (previously 0.14.0)

• Hive Metastore Version: 3.1.3

• Flink Version: 1.19.1

• Deployment: Docker containers

Steps to Reproduce

1. Create a Hudi catalog and table in Flink using Hudi 1.0.1. Below is the Flink SQL used :

   sql

   -- Create Hudi catalog
   CREATE CATALOG hudi_catalog_3 WITH (
      'type' = 'hudi',
      'catalog.path' = 's3a://catalog_name/data_pool2/hudi_catalog_3',
      'hive.conf.dir' = 's3a://catalog_name/data_pool/conf_hms_8084',
      'table.external' = 'true',
      'mode' = 'hms'
   );
   
   -- Create COW table
   CREATE TABLE IF NOT EXISTS hudi_catalog_3.data_pool2.file_metadata_cow_1_4 (
    -- Primary key
    uuid STRING,
    object_id STRING,
    -- Base fields
    tenant_id STRING,
    store_type STRING,
    store_object_key STRING,
    file_name STRING,
    file_path STRING,
    file_format STRING,
    source_channel STRING,
    description STRING,
    tags MAP<STRING,STRING>,
    -- High-frequency search fields
    file_width INT,
    file_height INT,
    duration DOUBLE,
    alarm_id STRING,
    device_id STRING,
    skill_name STRING,
    -- Skill attributes
    skill_attrs ROW<
      labels ROW<
        alertname STRING,
        business STRING,
        orgUuid STRING,
        requestID STRING,
        roiID STRING,
        skillID STRING,
        trackerID STRING
      >,
      annotations ROW<
        callback STRING,
        desc STRING,
        imageTimestamp STRING,
        isMeter STRING,
        predictions STRING
      >
    >,
    created_at TIMESTAMP(3),
    updated_at TIMESTAMP(3),
    deleted_at TIMESTAMP(3),
    delete_status TINYINT,
    -- Processing time
    proc_time AS PROCTIME(),
    PRIMARY KEY (uuid) NOT ENFORCED
   ) WITH (
    'connector' = 'hudi',
    'path' = 's3a://catalog_name/data_pool2/data/file_metadata_cow_1_4',
    'table.type' = 'COPY_ON_WRITE',
    'hoodie.datasource.write.recordkey.field' = 'uuid',
    'hoodie.datasource.write.precombine.field' = 'updated_at',
    -- Partition configuration
    'hoodie.datasource.write.partitionpath.field' = 'tenant_id,file_format,source_channel',
    -- Lock configuration
    'hoodie.write.lock.provider' = 'org.apache.hudi.client.transaction.lock.InProcessLockProvider'
   );

2. Run a simple query in Trino:

   sql

   SELECT * FROM hudi_catalog_2.data_pool2.file_metadata_cow_1_4;

3. The query fails with the following error:

   Query 20250320_073504_00043_kf9gm failed: Internal error
   

4. The only relevant log from the Trino Docker container is:

   2025-03-20T07:35:04.474Z  INFO  dispatcher-query-81  io.trino.event.QueryMonitor
   TIMELINE: Query 20250320_073504_00043_kf9gm :: FAILED (GENERIC_INTERNAL_ERROR) :: elapsed 395ms :: planning 395ms :: waiting 0ms :: scheduling 0ms :: running 0ms :: finishing 0ms :: begin 2025-03-20T07:35:04.078Z :: end 2025-03-20T07:35:04.473Z
   

Observations

• The issue affects both COW and MOR tables in Hudi 1.0.x.

• Queries succeed in Trino with Hudi 0.14.0 and in Flink with Hudi 1.0.1, indicating a compatibility issue between Trino 473 and Hudi 1.0.1.

• The failure occurs during the query planning phase, suggesting potential issues with metadata parsing or query planning.

• Hive Metastore metadata is intact, as confirmed by successful queries in Flink.

Suspected Cause

Hudi 1.0.1 may have introduced changes (e.g., in metadata format or table structure) that are not yet supported by Trino’s Hudi connector in version 473.

Questions

• Does Trino 473 support Hudi 1.0.1 for COW and MOR tables?

• If not, is there a planned update to add support, or are there any known workarounds to enable querying these tables?

• Could you provide any guidance on debugging or resolving the GENERIC_INTERNAL_ERROR?

Thank you for your time and support!

[ebyhr]

Please share the full stacktrace of step 3. You can get it by CLI --debug option or Web UI.

[NICK-T-D]

Please share the full stacktrace of step 3. You can get it by CLI --debug option or Web UI.

trino> select * from hudi.data_pool.file_metadata_mor_now8_rt;
Query 20250324_062027_00024_kiek4 failed: Internal error
java.lang.IllegalArgumentException
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:129)
        at io.trino.plugin.hudi.timeline.TimelineLayoutVersion.<init>(TimelineLayoutVersion.java:29)
        at java.base/java.util.Optional.map(Optional.java:260)
        at io.trino.plugin.hudi.config.HudiTableConfig.getTimelineLayoutVersion(HudiTableConfig.java:103)
        at io.trino.plugin.hudi.table.HudiTableMetaClient.<init>(HudiTableMetaClient.java:70)
        at io.trino.plugin.hudi.table.HudiTableMetaClient.newMetaClient(HudiTableMetaClient.java:140)
        at io.trino.plugin.hudi.table.HudiTableMetaClient$Builder.build(HudiTableMetaClient.java:196)
        at io.trino.plugin.hudi.HudiUtil.buildTableMetaClient(HudiUtil.java:135)
        at io.trino.plugin.hudi.HudiSplitSource.<init>(HudiSplitSource.java:76)
        at io.trino.plugin.hudi.HudiSplitManager.getSplits(HudiSplitManager.java:101)
        at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorSplitManager.getSplits(ClassLoaderSafeConnectorSplitManager.java:51)
        at io.trino.split.SplitManager.getSplits(SplitManager.java:89)
        at io.trino.sql.planner.SplitSourceFactory$Visitor.createSplitSource(SplitSourceFactory.java:183)
        at io.trino.sql.planner.SplitSourceFactory$Visitor.visitTableScan(SplitSourceFactory.java:156)
        at io.trino.sql.planner.SplitSourceFactory$Visitor.visitTableScan(SplitSourceFactory.java:130)
        at io.trino.sql.planner.plan.TableScanNode.accept(TableScanNode.java:219)
        at io.trino.sql.planner.SplitSourceFactory$Visitor.visitOutput(SplitSourceFactory.java:360)
        at io.trino.sql.planner.SplitSourceFactory$Visitor.visitOutput(SplitSourceFactory.java:130)
        at io.trino.sql.planner.plan.OutputNode.accept(OutputNode.java:82)
        at io.trino.sql.planner.SplitSourceFactory.createSplitSources(SplitSourceFactory.java:110)
        at io.trino.execution.scheduler.PipelinedQueryScheduler$DistributedStagesScheduler.createStageScheduler(PipelinedQueryScheduler.java:1060)
        at io.trino.execution.scheduler.PipelinedQueryScheduler$DistributedStagesScheduler.create(PipelinedQueryScheduler.java:935)
        at io.trino.execution.scheduler.PipelinedQueryScheduler.createDistributedStagesScheduler(PipelinedQueryScheduler.java:324)
        at io.trino.execution.scheduler.PipelinedQueryScheduler.start(PipelinedQueryScheduler.java:307)
        at io.trino.execution.SqlQueryExecution.start(SqlQueryExecution.java:446)
        at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:272)
        at io.trino.dispatcher.LocalDispatchQuery.startExecution(LocalDispatchQuery.java:150)
        at io.trino.dispatcher.LocalDispatchQuery.lambda$waitForMinimumWorkers$2(LocalDispatchQuery.java:134)
        at io.airlift.concurrent.MoreFutures.lambda$addSuccessCallback$12(MoreFutures.java:570)
        at io.airlift.concurrent.MoreFutures$3.onSuccess(MoreFutures.java:545)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1139)
        at io.trino.$gen.Trino_473____20250320_093236_2.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1575)

[ebyhr]

What's the value of hoodie.timeline.layout.version?

[NICK-T-D]

What's the value of hoodie.timeline.layout.version?

I checked the hoodie.properties file. Here's what I found:

hoodie.timeline.layout.version=2

To provide full context, below is the relevant section of the hoodie.properties file I reviewed:

#Updated at 2025-03-18T08:08:37.505Z
#Tue Mar 18 16:08:37 CST 2025
hoodie.table.precombine.field=updated_at
hoodie.table.version=8
hoodie.table.initial.version=8
hoodie.datasource.write.hive_style_partitioning=false
hoodie.table.metadata.partitions.inflight=
hoodie.table.checksum=3137094075
hoodie.table.keygenerator.type=NON_PARTITION_AVRO
hoodie.table.create.schema=foo_xxx
hoodie.table.cdc.enabled=false
hoodie.archivelog.folder=history
hoodie.table.name=file_metadata_mor_now8
hoodie.record.merge.strategy.id=00000000-0000-0000-0000-000000000000
hoodie.timeline.history.path=history
hoodie.compaction.payload.class=org.apache.hudi.common.model.EventTimeAvroPayload
hoodie.table.type=MERGE_ON_READ
hoodie.datasource.write.partitionpath.urlencode=false
hoodie.datasource.write.drop.partition.columns=false
hoodie.table.metadata.partitions=files
hoodie.timeline.layout.version=2
hoodie.record.merge.mode=CUSTOM
hoodie.table.recordkey.fields=uuid
hoodie.table.partition.fields=tenant_id,file_format,source_channel
hoodie.timeline.path=timeline

[ebyhr]

Thanks, the connector supports only 0 and 1 hoodie.timeline.layout.version for now.

[NICK-T-D]

Thanks, the connector supports only 0 and 1 hoodie.timeline.layout.version for now.

Thank you so much for taking the time to help me troubleshoot this issue. Your assistance has deepened my understanding of the problem, and I’m truly grateful for that!

With your support, I’d like to learn more about the plans for addressing this issue and recommendations for the current situation. Specifically, I’m curious about the following:

1. Have any plans to address it in a future release?

2. For my current situation, do they have any suggestions or workarounds to offer?