plannertest: criterion-rs based benchmark for both planning and execution

[yliang412]

Part of #263.

Problem

There is a lack of performance evaluation in optd. Without proper benchmarking, we will not know if some changes cause significant regression in performance. We should keep track of the two aspects of optimizer performance:

• plan generation time
• actual execution time

Summary of changes

• Add a Criterion-rs based benchmark to measure optd performance.
• Use the sqlplannertest yaml-based setup, bench if there is a bench task.
• Bump sqlplannertest to version 0.4.
  • better individual test matching added in improve individual test matching and output risinglightdb/sqlplannertest-rs#9

Planning:

• Input: SQL statement (aka datafusion_sql::parser::Statement).
• Measures the time it takes to generate a datafusion physical plan.

Execution:

• Input: datafusion physical plan outputted from optd (aka Arc<dyn ExecutionPlan>).
• Measures the time it takes to execute the entire query.

Next Steps

https://github.com/benchmark-action/github-action-benchmark can directly take cargo bench results as input. So it will be easy to set up CI.

[yliang412]

Examples

# benchmark all TPC-H queries with "bench" task enabled
cargo bench --bench planner_bench tpch/

# benchmark TPC-H Q1
$ cargo bench --bench planner_bench tpch/q1/
    Finished `bench` profile [optimized] target(s) in 0.21s
     Running benches/planner_bench.rs (target/release/deps/planner_bench-978004a60630bb99)
Gnuplot not found, using plotters backend
Benchmarking tpch/q1/planning: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.7s, enable flat sampling, or reduce sample count to 60.
tpch/q1/planning        time:   [292.65 µs 295.24 µs 300.24 µs]
Found 12 outliers among 100 measurements (12.00%)
  4 (4.00%) high mild
  8 (8.00%) high severe

tpch/q1/execution       time:   [534.62 µs 538.27 µs 544.50 µs]
Found 12 outliers among 100 measurements (12.00%)
  3 (3.00%) high mild
  9 (9.00%) high severe

# benchmark TPC-H Q1 planning
cargo bench --bench planner_bench tpch/q1/planning

# benchmark TPC-H Q1 execution
cargo bench --bench planner_bench tpch/q1/execution

[skyzh]

CI config not updated yet? I assume there will be a separate patch...

[yliang412]

CI config not updated yet? I assume there will be a separate patch...

CI will be done as a separate PR.