2025-01-02 kernel meeting notes

[OussamaSaoudi-db]

summary

• Our dependency on Reqwest pulls in native TLS, which doesn't work on AWS lambda. Delta-rs users are forced to use an older version of delta-rs, but they're missing out on new perf improvements.
• Had some discussion on error types in our crate. Tyler advocates for us looking into tracing as a complement to errors. Keep errors entirely user-facing, while tracing should be used for debugging.
• Also had discussion on benchmarking. Github actions aren't very useful for cpu perf since it's multi-tenant. He did suggest that memory usage benchmarks may be useful. Mentioned a duckdb test that reads a "zip-bomb" that expands to many GBs. For us, we'd want to perhaps check that when we read a large checkpoint, we're not reading the entire thing into memory, and instead streaming it.

pull requests/ongoing threads

• @zachschuermann on write FFI, user guides, parquet reader improvement (async), need a way to create engine data
• @OussamaSaoudi-db Introducing in-commit timestamp support
• @hntd187 has Map PR
• @roeap delta-rs integration!
• @nicklan Exposing expression for physical to logical transformation to engines.
• @timsaucer Expanding data skipping to use partition columns

attendees

@OussamaSaoudi-db @rtyler @dennyglee

leftovers

• zach: define MSRV policy (follow datafusion?)
• zach: define release expectation
• zach: labels discussion
• zach/robert: datafusion integration
• zach/oussama: Address the reqwest dependency that uses native TLS.

notes stream

• tyler: 0.2.3 of delta-rs is out
• Tyler: kernel usess reqwest with default features that uses native TLS that uses a vender SSL that causes issues with AWS lambda. cross compilation causes symbols to be incompatible with AWS lambda. PR is open, but it is needed atm. Users of delta-rs stuck on old, slower performing version. Pulling open-ssl is a big deal for embedded usecasse. Question: How does duckdb pull in kernel to handle this case?
• Tyler: Need to discuss about moving delta-rs to log replay powered by kernel. cc @nicklan @roeap. Goal: Delta-rs 1.0 should include kernel log replay. some kernel APIs in kernel are private, and delta-rs should identify those points and hit privacy parity.
• Oussama: we're looking at error handling.
• tyler: error handling is tricky. Bubbling up low level errors, we need to be aware of low level ABI structure. Ex: Object store could be included two times, and this may have a different incompatible ABI. Cargo makes it fine as long as you don't mix those structs. There's been trouble in delta-rs. You can either do Type Erasure but then you can't tell difference between error types. Other option is to have an enum with From impls that map object store errors into a delta error. Gives the caller information that bubbles up. Problem: you can have root level error and operation specific errors. For optional dependencies, we don't want to wrap their error types Ex deltaError(ObjectStoreError).
• delta-rs doesn't have heirarchy within their own enum variants. thiserror makes it easy to wrap an error.
• other issue with kernel error is that errors don't mean much over FFI. There needs to be some level of type erasure.
• Backtrace crate. One difficulty with kernel and arrow errors is that a lot of info about the error is lost. Backtrace helps, but getting the trace info out to the point where errors may be the wrong tool. Telemetry may be the better option using tracing. Could use different subscribers (ex: open telemetry and cloudwatch). Could use as a debugger endpoint. tracing spans makes it so that backtrace info is no longer necessary. It tracks all the trace info. Tyler argues that we may want more than the rust system can provide.
• Question: is tracing is sample or full trace. Answer: we can configure that to construct spans, and drop them if we don't need them. If we're emitting too many events, collector could be configured to do sampling. Tracing has been useful for performance info. Debug sucks for that, but tracing helps track perf info when you need it.
• Errors are a subset of observability. Solve observability to solve errors. Caller of APIs should be able to match on the error to influence their behaviour. ex: a write conflict demands a behaviour change. If there's no reason a user would match, the error isn't very useful.
• delta-rs mistake was having root error wrap external errors. We only need a serde error when parsing the log. There's no reason for a caller to match on a serde json error.
• question: does delta-rs use bench. answer: it exists, but it's not used a lot.
• github actions is multi-tenant, so it's hard to measure performance. Performance challenges mostly come from concurrency, which cargo bench doesn't help much with. Types of perf issues are typically in complex cases, not in contrived examples from bench. Another issue is memory usage, which bench doesn't help a lot with. Duckdb has a zipbomb test that extracts to GBs that is helpful for memory performance. There's no tool for closely monitoring memory usage or perf.
• Tyler: cargo bench is a runner. . But cpu tests aren't very useful.
• Duckdb has a test for a checkpoint of size n MB and asserts that the process doesn't use more than a*n MB of memory a < 1.