How to control the final file size using Kyuubi Spark engine

[YiBingWork]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

使用kyuubi执行spark-sql 设置相关参数 执行结果文件数过大

Affects Version(s)

1.3.0

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;?#spark.yarn.queue=batch;spark.driver.memory=10g;spark.executor.cores=2;spark.num-executors=8;spark.executor.memory=10g;spark.executor-memory=10g;gspark.sql.shuffle.partitions=2;spark.sql.adaptive.enabled=true;spark.sql.adaptive.shuffle.targetPostShuffleInputSize=268435456;spark.sql.storeAssignmentPolicy=LEGACY;"

Additional context

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[github-actions[bot]]

Hello @18580725343,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi (Incubating).

[YiBingWork]

将结果插入目标表 目标表分区下的数据文件过大

[ulysses-you]

@18580725343 have you tried enable Kyuubi Spark extension ? some docs https://kyuubi.apache.org/docs/r1.3.0-incubating/sql/rules.html

[pan3793]

I convert it to a discussion because it's not a bug but a question

[pan3793]

I guess you think that the final output file size is controlled by spark.sql.adaptive.shuffle.targetPostShuffleInputSize?

[YiBingWork]

是的 想控制最终写出的文件大小

[pan3793]

It's not true.
spark.sql.adaptive.shuffle.targetPostShuffleInputSize is deprecated in Spark 3 and has been replaced by spark.sql.adaptive.advisoryPartitionSizeInBytes[1], it does NOT mean the final output file size.

In short, currently, for Hive table and DataSource V1 file table, there is no out-of-box solution in open-source Apache Spark to control the output file size by setting a magic configuration.

Kyuubi provides an extension[2] to handle the small file case, you can follow the document to set up and tune the configuration to see if it works for you. Again, there is NO magic configuration to control the final output file size directly and accurately in Spark.

[1] https://spark.apache.org/docs/latest/configuration.html#runtime-sql-configuration
[2] https://kyuubi.apache.org/docs/latest/sql/rules.html

[YiBingWork]

thanks very much