2024-12-12 kernel meeting notes #594

zachschuermann · 2024-12-12T19:23:52Z

zachschuermann
Dec 12, 2024
Maintainer

0.6.0 incoming

@zachschuermann on write FFI, user guides, parquet reader improvement (async), need a way to create engine data
@scovich data skipping improvement/nested cols merged!
@OussamaSaoudi-db CDF shipped!
@hntd187 has Map PR
@roeap delta-rs integration!

tyler: define MSRV policy. req: kernel rust should just follow datafusion? (1.80)
zach: introducing commit message requirements. thoughts?
tyler: don't make it required for every commit!
zach: 0.6.0 release upcoming (and asking to help upgrade delta-rs)
come up with a release expectation (e.g. if we maintain current 0.x and cherry pick back to 0.x-1 etc.)
zach: labels discussion labels for issues/PRs? #586
tyler: will robert bring log replay from kernel to delta-rs?
robert started on protocol action, but notice that after further integration there will be less reliance on actions in general
zach: delta-rs benchmarks? should we ensure we don't regress performance?
robert: third thing - don't do in delta-rs, try to implement datafusion engine? many operations are datafusion execution plans (merge etc.)
tyler: start discussion on integrating kernel. datafusion table provider - probably not? try a github codespace to work on branch together to integrate kernel more.
robert: rather than updating protocol - could implement within kernel. but requires using kernel in write path. note: delta-rs is all async, kernel is sync.
tyler: iterator vs stream
- kernel has iterators, consumers could make this stream
robert: integrate at halfway point of the scan? gonna try!