tests: make Workload more determinstic

[jcsp]

Problem

Previously, Workload was reconfiguring the compute before each run of writes, which was meant to be a no-op when nothing changed, but was actually writing extra data due to an issue being fixed in #10696.

The row counts in tests were too low in some cases, these tests were only working because of those extra writes that shouldn't have been happening, and moreover were relying on checkpoints happening.

Summary of changes

• Only reconfigure compute if the attached pageserver actually changed. If pageserver is set to None, that means controller is managing everything, so never reconfigure compute.
• Update tests that wrote too few rows.

[github-actions]

7425 tests run: 6944 passed, 122 failed, 359 skipped (full report)

---

Failures on Postgres 17

• test_location_conf_churn[3]: release-x86-64-with-lfc, release-x86-64-without-lfc, release-arm64-with-lfc, release-arm64-without-lfc, debug-x86-64-without-lfc
• test_location_conf_churn[1]: release-x86-64-with-lfc, release-x86-64-without-lfc, debug-x86-64-without-lfc, release-arm64-without-lfc, release-arm64-with-lfc
• test_location_conf_churn[2]: release-x86-64-without-lfc, debug-x86-64-without-lfc, release-arm64-with-lfc
• test_sharding_backpressure: release-x86-64-without-lfc, release-x86-64-with-lfc, release-arm64-without-lfc, debug-x86-64-without-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure7]: release-x86-64-with-lfc
• test_sharding_split_failures[failure2]: release-x86-64-with-lfc, release-x86-64-without-lfc, debug-x86-64-without-lfc, release-arm64-without-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure15]: release-x86-64-with-lfc, release-x86-64-without-lfc, debug-x86-64-without-lfc, release-arm64-without-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure11]: release-x86-64-with-lfc, release-arm64-without-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure16]: release-x86-64-with-lfc, release-x86-64-without-lfc, release-arm64-without-lfc, debug-x86-64-without-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure13]: release-x86-64-with-lfc
• test_sharding_split_failures[failure17]: release-x86-64-with-lfc, release-x86-64-without-lfc, release-arm64-without-lfc
• test_sharding_split_failures[failure8]: release-arm64-with-lfc
• test_sharding_split_failures[failure9]: release-arm64-with-lfc, release-arm64-without-lfc
• test_scrubber_physical_gc_ancestors[2]: release-arm64-without-lfc, debug-x86-64-without-lfc

Failures on Postgres 16

• test_location_conf_churn[3]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_location_conf_churn[2]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_location_conf_churn[1]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_smoke: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_backpressure: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure6]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure2]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure10]: release-x86-64-with-lfc
• test_sharding_split_failures[failure9]: release-x86-64-with-lfc
• test_sharding_split_failures[failure17]: release-x86-64-with-lfc
• test_sharding_split_failures[failure11]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure15]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure16]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_scrubber_physical_gc_ancestors[2]: release-arm64-with-lfc

Failures on Postgres 15

• test_location_conf_churn[3]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_location_conf_churn[1]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_location_conf_churn[2]: release-arm64-with-lfc
• test_sharding_smoke: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure5]: release-x86-64-with-lfc
• test_sharding_split_failures[failure4]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure6]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure7]: release-x86-64-with-lfc
• test_sharding_split_failures[failure10]: release-x86-64-with-lfc
• test_sharding_split_failures[failure17]: release-x86-64-with-lfc
• test_sharding_split_failures[failure15]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure2]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_backpressure: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure8]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure11]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure16]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_scrubber_physical_gc_ancestors[2]: release-arm64-with-lfc

Failures on Postgres 14

• test_location_conf_churn[3]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_location_conf_churn[2]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_location_conf_churn[1]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_smoke: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_backpressure: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure5]: release-x86-64-with-lfc
• test_sharding_split_failures[failure2]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure6]: release-x86-64-with-lfc
• test_sharding_split_failures[failure11]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure13]: release-x86-64-with-lfc
• test_sharding_split_failures[failure15]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure9]: release-x86-64-with-lfc
• test_sharding_split_failures[failure16]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure17]: release-x86-64-with-lfc
• test_scrubber_physical_gc_ancestors[2]: release-arm64-with-lfc

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_location_conf_churn[release-pg14-3] or test_location_conf_churn[release-pg14-3] or test_location_conf_churn[release-pg14-2] or test_location_conf_churn[release-pg14-2] or test_location_conf_churn[release-pg14-1] or test_location_conf_churn[release-pg14-1] or test_sharding_smoke[release-pg14] or test_sharding_smoke[release-pg14] or test_sharding_backpressure[release-pg14] or test_sharding_backpressure[release-pg14] or test_sharding_split_failures[release-pg14-failure5] or test_sharding_split_failures[release-pg14-failure2] or test_sharding_split_failures[release-pg14-failure2] or test_sharding_split_failures[release-pg14-failure6] or test_sharding_split_failures[release-pg14-failure11] or test_sharding_split_failures[release-pg14-failure11] or test_sharding_split_failures[release-pg14-failure13] or test_sharding_split_failures[release-pg14-failure15] or test_sharding_split_failures[release-pg14-failure15] or test_sharding_split_failures[release-pg14-failure9] or test_sharding_split_failures[release-pg14-failure16] or test_sharding_split_failures[release-pg14-failure16] or test_sharding_split_failures[release-pg14-failure17] or test_scrubber_physical_gc_ancestors[release-pg14-2] or test_location_conf_churn[release-pg15-3] or test_location_conf_churn[release-pg15-3] or test_location_conf_churn[release-pg15-1] or test_location_conf_churn[release-pg15-1] or test_location_conf_churn[release-pg15-2] or test_sharding_smoke[release-pg15] or test_sharding_smoke[release-pg15] or test_sharding_split_failures[release-pg15-failure5] or test_sharding_split_failures[release-pg15-failure4] or test_sharding_split_failures[release-pg15-failure4] or test_sharding_split_failures[release-pg15-failure6] or test_sharding_split_failures[release-pg15-failure6] or test_sharding_split_failures[release-pg15-failure7] or test_sharding_split_failures[release-pg15-failure10] or test_sharding_split_failures[release-pg15-failure17] or test_sharding_split_failures[release-pg15-failure15] or test_sharding_split_failures[release-pg15-failure15] or test_sharding_split_failures[release-pg15-failure2] or test_sharding_split_failures[release-pg15-failure2] or test_sharding_backpressure[release-pg15] or test_sharding_backpressure[release-pg15] or test_sharding_split_failures[release-pg15-failure8] or test_sharding_split_failures[release-pg15-failure8] or test_sharding_split_failures[release-pg15-failure11] or test_sharding_split_failures[release-pg15-failure11] or test_sharding_split_failures[release-pg15-failure16] or test_sharding_split_failures[release-pg15-failure16] or test_scrubber_physical_gc_ancestors[release-pg15-2] or test_location_conf_churn[release-pg16-3] or test_location_conf_churn[release-pg16-3] or test_location_conf_churn[release-pg16-2] or test_location_conf_churn[release-pg16-2] or test_location_conf_churn[release-pg16-1] or test_location_conf_churn[release-pg16-1] or test_sharding_smoke[release-pg16] or test_sharding_smoke[release-pg16] or test_sharding_backpressure[release-pg16] or test_sharding_backpressure[release-pg16] or test_sharding_split_failures[release-pg16-failure6] or test_sharding_split_failures[release-pg16-failure6] or test_sharding_split_failures[release-pg16-failure2] or test_sharding_split_failures[release-pg16-failure2] or test_sharding_split_failures[release-pg16-failure10] or test_sharding_split_failures[release-pg16-failure9] or test_sharding_split_failures[release-pg16-failure17] or test_sharding_split_failures[release-pg16-failure11] or test_sharding_split_failures[release-pg16-failure11] or test_sharding_split_failures[release-pg16-failure15] or test_sharding_split_failures[release-pg16-failure15] or test_sharding_split_failures[release-pg16-failure16] or test_sharding_split_failures[release-pg16-failure16] or test_scrubber_physical_gc_ancestors[release-pg16-2] or test_location_conf_churn[release-pg17-3] or test_location_conf_churn[release-pg17-3] or test_location_conf_churn[release-pg17-3] or test_location_conf_churn[release-pg17-3] or test_location_conf_churn[debug-pg17-3] or test_location_conf_churn[release-pg17-1] or test_location_conf_churn[release-pg17-1] or test_location_conf_churn[debug-pg17-1] or test_location_conf_churn[release-pg17-1] or test_location_conf_churn[release-pg17-1] or test_location_conf_churn[release-pg17-2] or test_location_conf_churn[debug-pg17-2] or test_location_conf_churn[release-pg17-2] or test_sharding_backpressure[release-pg17] or test_sharding_backpressure[release-pg17] or test_sharding_backpressure[release-pg17] or test_sharding_backpressure[debug-pg17] or test_sharding_backpressure[release-pg17] or test_sharding_split_failures[release-pg17-failure7] or test_sharding_split_failures[release-pg17-failure2] or test_sharding_split_failures[release-pg17-failure2] or test_sharding_split_failures[debug-pg17-failure2] or test_sharding_split_failures[release-pg17-failure2] or test_sharding_split_failures[release-pg17-failure2] or test_sharding_split_failures[release-pg17-failure15] or test_sharding_split_failures[release-pg17-failure15] or test_sharding_split_failures[debug-pg17-failure15] or test_sharding_split_failures[release-pg17-failure15] or test_sharding_split_failures[release-pg17-failure15] or test_sharding_split_failures[release-pg17-failure11] or test_sharding_split_failures[release-pg17-failure11] or test_sharding_split_failures[release-pg17-failure11] or test_sharding_split_failures[release-pg17-failure16] or test_sharding_split_failures[release-pg17-failure16] or test_sharding_split_failures[release-pg17-failure16] or test_sharding_split_failures[debug-pg17-failure16] or test_sharding_split_failures[release-pg17-failure16] or test_sharding_split_failures[release-pg17-failure13] or test_sharding_split_failures[release-pg17-failure17] or test_sharding_split_failures[release-pg17-failure17] or test_sharding_split_failures[release-pg17-failure17] or test_sharding_split_failures[release-pg17-failure8] or test_sharding_split_failures[release-pg17-failure9] or test_sharding_split_failures[release-pg17-failure9] or test_scrubber_physical_gc_ancestors[release-pg17-2] or test_scrubber_physical_gc_ancestors[debug-pg17-2]"

Flaky tests (44)

Postgres 17

• test_location_conf_churn[2]: release-x86-64-with-lfc, release-arm64-without-lfc
• test_sharding_smoke: release-x86-64-with-lfc
• test_sharding_gc: release-x86-64-with-lfc, release-arm64-without-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure11]: release-x86-64-without-lfc, debug-x86-64-without-lfc
• test_sharding_split_failures[failure12]: release-x86-64-with-lfc, debug-x86-64-without-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure9]: release-x86-64-with-lfc
• test_sharding_split_failures[failure8]: debug-x86-64-without-lfc, release-arm64-without-lfc
• test_sharding_split_failures[failure7]: debug-x86-64-without-lfc, release-arm64-without-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure4]: release-arm64-without-lfc
• test_sharding_split_failures[failure10]: debug-x86-64-without-lfc
• test_sharding_split_failures[failure6]: release-arm64-without-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure13]: release-arm64-with-lfc
• test_sharding_split_failures[failure17]: release-arm64-with-lfc
• test_scrubber_physical_gc_ancestors[2]: release-arm64-with-lfc

Postgres 16

• test_sharding_split_failures[failure7]: release-x86-64-with-lfc
• test_sharding_split_failures[failure5]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure12]: release-x86-64-with-lfc
• test_sharding_split_failures[failure4]: release-arm64-with-lfc
• test_sharding_split_failures[failure8]: release-arm64-with-lfc
• test_sharding_split_failures[failure9]: release-arm64-with-lfc
• test_sharding_split_failures[failure10]: release-arm64-with-lfc

Postgres 15

• test_sharding_split_failures[failure9]: release-x86-64-with-lfc, release-arm64-with-lfc
• test_sharding_split_failures[failure5]: release-arm64-with-lfc
• test_sharding_split_failures[failure13]: release-arm64-with-lfc
• test_sharding_split_failures[failure10]: release-arm64-with-lfc
• test_sharding_split_failures[failure7]: release-arm64-with-lfc
• test_scrubber_physical_gc_ancestors[None]: release-arm64-with-lfc

Postgres 14

• test_sharding_split_failures[failure8]: release-x86-64-with-lfc
• test_sharding_split_failures[failure5]: release-arm64-with-lfc
• test_sharding_split_failures[failure12]: release-arm64-with-lfc
• test_sharding_split_failures[failure13]: release-arm64-with-lfc
• test_sharding_split_failures[failure17]: release-arm64-with-lfc

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
5e6bf2d at 2025-02-10T13:36:44.979Z :recycle:}