Weekly Plan: Andrew Lamb 2025-05-19

[alamb]

This is my plan this week for reviews, etc. I am putting it here to make it visible and keep myself organized

• Parquet Filter Pushdown Performance:
• DataFusion performance: help get per-file pruning with @adriangb merged Add late pruning of file based on file level statistics #16014
• DataFusion performance: review blocked aggregate PR from @Rachelint : Implement intermediate result blocked approach to aggregation memory management #15591 (comment)
• DataFusion performance: Projection Pushdown: review suggestion Support Push down expression evaluation in TableProviders #14993 (comment) from @adragomir
• Parquet Variant: get variant encoder/decoder into parquet-variant crate with @PinkCrow007
• Parquet Variant: Try and fix variant example files with @mapleFU primitive_int64.value maybe an int32 type parquet-testing#82
• DataFusion Feature: Update example of using multiple threadpools with object store Example for using a separate threadpool for CPU bound work (try 2) #14286 (comment)
• DataFusion Feature: async user defined functions with @goldmedal Introduce Async User Defined Functions  #14837
• Arrow Bug with concat'ing dictionaries from @davidhewitt: support merging primitive dictionaries in interleave and concat arrow-rs#7468
• DataFusion perf script from @logan-keede : Shell script to collect benchmarks for multiple versions #15144
• DataFusion perf script draft: feat(benchmark): collect benchmarks for last 5 versions in line protocol format #15846
• DataFusion PR about pruning ordering: pipe column orderings into pruning predicate creation #15821
• DataFusion Sorting: test: add fuzz test for doing aggregation with larger than memory groups and sorting with limited memory #15727
• Arrow Dictionary ID next steps from @brancz: Remove dict_id from schema and make it an IPC concern only arrow-rs#7467
• Arrow Variant: Expose tape decoder Add custom decoder in arrow-json arrow-rs#7442

Nice to have (really would be great to have someone help review):

• DataFusion: Aggregate UDFs in FFI: feat: Add Aggregate UDF to FFI crate #14775
• Arrow: Avro cleanup: Avro codec enhancements arrow-rs#6965
• Arrow: Avro Utf8View: Support Utf8View for Avro arrow-rs#7434