release-7215

What's Changed

• pageserver: respect no_sync in VirtualFile by @jcsp in #9772
• Fix the regression test running against the staging instance by @a-masterov in #9773
• Use DATA_DIR instead of current workign directory in restore_from_wal script by @knizhnik in #9729
• adapt pgvector bench to minor version upgrades of PostgreSql by @Bodobolero in #9784
• tests: mitigate bug to stabilize test_storage_controller_many_tenants by @jcsp in #9771
• CI: remove separate check-build-tools-image workflow by @bayandin in #9708
• Bump default Postgres version for tests to v17 by @bayandin in #9777
• feat(compute/tests): Report successful replication test runs as well by @ololobus in #9787
• feat(pageserver): drop disposable keys during gc-compaction by @skyzh in #9765
• feat(page_service): timeout-based batching of requests by @VladLazar in #9321
• Remove at most one retain_lsn entry from (possibly offloaded) timelne's parent by @arpad-m in #9791
• Add the ability to configure GenericRemoteStorage for the scrubber by @arpad-m in #9652
• Move logic for ingest benchmark from GitHub workflow into python testcase by @Bodobolero in #9762
• Do not create DSM segment for wal_redo_postgres by @knizhnik in #9793
• chore(local_proxy): reduce some startup logging by @conradludgate in #9798
• fix(proxy/auth_broker): ignore -pooler suffix by @conradludgate in #9800
• PostgreSQL minor version updates (17.2, 16.6, 15.10, 14.15) by @hlinnaka in #9795
• re-acquire S3 OIDC token after long running tests for report upload to S3 by @Bodobolero in #9799
• build(deps): bump aiohttp from 3.10.2 to 3.10.11 by @dependabot in #9794
• fix(pageserver): queue stopped error should be ignored during create timeline by @skyzh in #9767
• scrubber: fix parsing issue with Azure by @arpad-m in #9797
• feat(pageserver): support key range for manual compaction trigger by @skyzh in #9723
• Register custom xlog reader callbacks for on-demand WAL download in StartupDecodingContext by @knizhnik in #9007
• CI(release): create reusable workflow for releases by @bayandin in #9806
• CODEOWNERS: add developer-productivity team by @bayandin in #9810
• compute_ctl: Streamline and Pipeline startup SQL by @MMeent in #9717
• Don't preload offloaded timelines by @arpad-m in #9646
• chore(proxy): demote a ton of logs for successful connection attempts by @conradludgate in #9803
• safekeeper: block deletion on protocol handler shutdown by @jcsp in #9364
• update macos version for runner by @fedordikarev in #9817
• CI(benchmarking): route test failures to on-call-qa-staging-stream by @bayandin in #9813
• scan_log_for_errors: check that regex is correct by @bayandin in #9815
• proxy: Rename RequestMonitoring to RequestContext by @cloneable in #9805
• tests: use fewer pageservers in test_sharding_split_smoke by @jcsp in #9804
• pageserver: avoid reading SLRU blocks for GC on shards >0 by @jcsp in #9423
• pageserver: enable compaction to proceed while live-migrating by @jcsp in #5397
• chore(proxy): demote more logs during successful connection attempts by @conradludgate in #9828
• pageserver: remove shard split hard link assertion by @VladLazar in #9829
• scrubber: allow restricting find_garbage to a partial tenant id prefix by @arpad-m in #9814
• [proxy] chore: allow bypassing empty params to /sql endpoint by @vadim2404 in #9827
• proxy: Refactor http conn pool by @awarus in #9785
• compute_ctl: print OpenTelemetry errors via tracing, not stdout by @arpad-m in #9830
• pageserver: revise metrics lifetime for SecondaryTenant by @jcsp in #9818
• Bump Postgres version by @knizhnik in #9808
• Python 3.11 by @bayandin in #9515
• fix(proxy): CancelKeyData display log masking by @conradludgate in #9838
• safekeeper,pageserver: add CPU profiling by @erikgrinaker in #9764
• Fix panic in compute_ctl metrics collection by @tristan957 in #9831
• minor PostgreSQL update in benchmarking by @Bodobolero in #9845
• Batch scrape workflows up to last 30 days and stop ad-hoc by @fedordikarev in #9846
• pageserver: permit reads behind GC cutoff during LSN grace period by @jcsp in #9833
• Avoid unnecessary send_replace calls in seqwait by @hlinnaka in #9852
• Write a newline after adding dynamic_shared_memory_type to PG conf by @tristan957 in #9843
• build(deps): bump mypy from 1.3.0 to 1.13.0 by @bayandin in #9670
• chore(proxy/jwks): reduce the rightward drift of jwks renewal by @conradludgate in #9853
• test_bulk_insert: fix typing for PgVersion by @bayandin in #9854
• proxy: Implement cancellation rate limiting by @awarus in #9739
• fix(pageserver): preempt and retry azure list operation by @skyzh in #9840
• safekeeper,pageserver: fix CPU profiling allowlists by @erikgrinaker in #9856
• test(pageserver): add detach ancestor smoke test by @skyzh in #9842
• fix(pageserver): ensure upload happens after delete by @skyzh in #9844
• Rename 'installed_extensions' metric to 'compute_installed_extensions' by @lubennikovaav in #9759
• fast imports: initial Importer and Storage changes by @problame in #9218
• test_runner: use LFC by default by @bayandin in #8613
• Bump futures-* crates, drop unused license, hide duplicate crate warnings by @cloneable in #9858
• Storage & Compute release 2024-11-25 by @vipvap in #9869
• chore: update rustls by @conradludgate in #9871
• Fixup Storage & Compute Release 2024-11-25 by @problame in #9881

Full Changelog: release-7145...release-7215

release-7145

What's Changed

• pageserver: refuse to load tenants with suspiciously old indices in old generations by @jcsp in #9719
• Improve error handling for NeonAPI fixture by @tristan957 in #9751
• safekeeper: fix atomicity of WAL truncation by @arssher in #9685
• cargo: use neon branch of rust-postgres by @VladLazar in #9757
• Correctly truncate VM by @knizhnik in #9342
• test: disable test_readonly_node_gc until proper fix by @yliang412 in #9755
• Remove the replication slot in test_snap_files at the end of the test by @tristan957 in #9752
• tests: add test_timeline_archival_chaos by @jcsp in #9609
• storcon: exclude non-Active tenants from shard autosplitting by @jcsp in #9743
• deny.toml: allow CDDL-1.0 license by @erikgrinaker in #9766
• tests: tolerate pageserver 500s in test_timeline_archival_chaos by @jcsp in #9769
• Correct mistakes in offloaded timeline retain_lsn management by @arpad-m in #9760
• Collect max_connections metric by @myrrc in #9770
• wal_decoder: make InterpretedWalRecord serde by @VladLazar in #9775
• Fix PG_MAJORVERSION_NUM typo by @tristan957 in #9774
• wal_decoder: rename end_lsn to next_record_lsn by @VladLazar in #9776
• safekeeper: send AppendResponse on segment flush by @erikgrinaker in #9692
• Cargo.toml: upgrade tikv-jemallocator to 0.6.0 by @erikgrinaker in #9779
• build(deps): bump smallvec to 1.13.2 to get UB fix by @problame in #9781
• Storage & Compute release 2024-11-18 by @vipvap in #9783

New Contributors

• @myrrc made their first contribution in #9770

Full Changelog: release-7125...release-7145

release-7125

What's Changed

• storcon: add peer token for peer to peer communication by @VladLazar in #9695
• CI: retry actions/github-script for 5XX errors by @bayandin in #9703
• fix(pageserver): increase frozen layer warning threshold; ignore in tests by @skyzh in #9705
• rfc: update aux file rfc to reflect latest optimizations by @skyzh in #9681
• safekeeper: add initialize_segment variant of safekeeper_wal_storage_operation_seconds by @erikgrinaker in #9691
• Remove checklist from the PR template by @bayandin in #9702
• new ingest benchmark by @Bodobolero in #9711
• test_runner: skip more tests using decorator instead of pytest.skip by @bayandin in #9704
• Clean up C.UTF-8 locale changes by @tristan957 in #9714
• feat(pageserver): support partial gc-compaction for delta layers by @skyzh in #9611
• use batch gh-workflow-stats-action with separate table by @fedordikarev in #9722
• Add a postgres_exporter config file by @tristan957 in #9720
• Add a retain_lsn test by @arpad-m in #9599
• Improve typing related to regress/test_logical_replication.py by @tristan957 in #9725
• Update pg_cron to 1.6.4 by @tristan957 in #9726
• safekeeper: don't flush control file on WAL ingest path by @erikgrinaker in #9698
• test(pageserver): add gc-compaction tests with delta will_init by @skyzh in #9724
• utils: remove unnecessary fsync in durable_rename() by @erikgrinaker in #9686
• Calculate compute_backpressure_throttling_seconds correctly by @tristan957 in #9732
• Add neon.logical_replication_max_logicalsnapdir_size by @tristan957 in #9467
• safekeeper: use WAL_SEGMENT_SIZE for empty timeline state by @erikgrinaker in #9734
• Fix autocommit footguns in performance tests by @tristan957 in #9735
• Do not wallog AUX files at replica by @knizhnik in #9457
• pageserver: add no_sync for use in regression tests (2/2) by @jcsp in #9678
• Add installed_extensions prometheus metric by @lubennikovaav in #9608
• tests: Increase timeout in test_create_churn_during_restart by @hlinnaka in #9736
• PostgreSQL minor version updates (17.1, 16.5, 15.9, 14.14) by @hlinnaka in #9727
• Compute release 2024-11-13 by @ololobus in #9745

Full Changelog: release-7097...release-7125

release-7097

What's Changed

• cherry-pick Clean up C.UTF-8 locale changes by @skyzh in #9721

Full Changelog: release-7095...release-7097

release-7095

What's Changed

• safekeeper: add walreceiver metrics by @erikgrinaker in #9450
• fix(test_regress): always use storcon virtual pageserver API to set tenant config by @problame in #9622
• quieten aws_config logs by @conradludgate in #9626
• proxy: Wrap JWT errors in separate AuthError variant by @cloneable in #9625
• proxy: Rewrite ControlPlaneEvent as enum by @cloneable in #9627
• Add tenant config option to allow timeline_offloading by @arpad-m in #9598
• Improve some typing related to performance testing for LR by @tristan957 in #9629
• Update psycopg2 to latest version for complete PG 17 support by @tristan957 in #9632
• pageserver: return 503 during tenant shutdown by @erikgrinaker in #9635
• Construct models::TenantConfig only once by @arpad-m in #9630
• scrubber: remove AWS region assumption, tolerate negative max_project_size by @jcsp in #9636
• proxy: Unify local and remote conn pool client structures by @awarus in #9604
• safekeeper: don't allocate send buffers on stack by @erikgrinaker in #9644
• [proxy] use the proxy protocol v2 command to silence some logs by @conradludgate in #9620
• proxy: Revert ControlPlaneEvent back to struct by @cloneable in #9649
• proxy: Refactor cplane types by @cloneable in #9643
• proxy: Read cplane JWT with clap arg by @cloneable in #9654
• proxy: Replace "web(auth)" with "console redirect" everywhere by @cloneable in #9655
• pageserver: ingest pre-serialized batches of values by @VladLazar in #9579
• storcon: respect tenant scheduling policy in drain/fill by @jcsp in #9657
• [proxy]: minor changes to endpoint-cache handling by @conradludgate in #9666
• feat(pageserver): add fast path for sparse keyspace read by @skyzh in #9631
• Rename compute_backpressure_throttling_ms to compute_backpressure_throttling_seconds by @tristan957 in #9668
• fix(test): use layer map dump in test_readonly_node_gc to validate layers protected by leases by @yliang412 in #9551
• refactor(pageserver): better pageservice command parsing by @skyzh in #9597
• Support copying layers in detach_ancestor from before shard splits by @arpad-m in #9669
• chore(compute): Bump pg_mooncake to the latest version by @dpxcc in #9672
• postgres_ffi: make WalGenerator generic over record generator by @erikgrinaker in #9614
• Fix direct reading from WAL buffers. by @arssher in #9639
• safekeeper: add basic WAL ingestion benchmarks by @erikgrinaker in #9531
• safekeeper: use set_len() to zero out segments by @erikgrinaker in #9665
• fix(pageserver): reduce aux compaction threshold by @skyzh in #9647
• Don't attach is_archived to debug output by @arpad-m in #9679
• [proxy/authorize]: improve JWKS reliability by @conradludgate in #9676
• add pgcopydb tool to build tools image by @Bodobolero in #9658
• Disable rust_2024_compatibility lint option by @hlinnaka in #9615
• ci: use set-docker-config-dir from dev-actions by @jcgruenhage in #9638
• pageserver: add no_sync for use in regression tests (1/2) by @jcsp in #9677
• Do not copy logical replicaiton slots to replica by @knizhnik in #9458
• Update TimescaleDB to 2.17.1 for PG 17 by @tristan957 in #9640
• fix(pageserver): drain upload queue before detaching ancestor by @skyzh in #9651
• tests: use tigther storcon scopes by @VladLazar in #9696
• CI: move check codestyle python to reusable workflow and run on a merge_group by @bayandin in #9683
• Switch compute-related locales to C.UTF-8 by default by @tristan957 in #6746
• feat(pageserver): more log lines around frozen layers by @skyzh in #9697
• safekeeper: remove unused WriteGuardSharedState::skip_update by @erikgrinaker in #9699
• fix(pageserver): drain upload queue before offloading timeline by @skyzh in #9682
• Improve type safety according to pyright by @tristan957 in #9633
• CI(pre-merge-checks): add required checks by @bayandin in #9700
• Storage & Compute release 2024-11-11 by @vipvap in #9710

New Contributors

• @dpxcc made their first contribution in #9672
• @jcgruenhage made their first contribution in #9638

Full Changelog: release-7044...release-7095

release-7044

What's Changed

• Add --replica parameter to basebackup by @knizhnik in #9553
• feat(pageserver): support partial gc-compaction for lowest retain lsn by @skyzh in #9134
• fix(pagectl): layer parsing and image layer dump by @skyzh in #9571
• Add tests for sql-exporter metrics by @tristan957 in #9184
• pageserver: generation-aware storage for TenantManifest by @jcsp in #9555
• Remove non-gzipped basebackup code path by @tristan957 in #9453
• chore(compute): Bump pg_mooncake to the latest version by @ololobus in #9576
• postgres_ffi: add WAL generator for tests/benchmarks by @erikgrinaker in #9503
• Collect compute backpressure throttling time by @tristan957 in #9176
• Disallow archived timelines to be detached or reparented by @arpad-m in #9578
• Don't keep around the timeline's remote_client by @arpad-m in #9583
• refactor(pageserver): use JSON field encoding for consumption metrics cache by @skyzh in #9470
• Disallow offloaded children during timeline deletion by @arpad-m in #9582
• pageserver: lift decoding and interpreting of wal into wal_decoder by @VladLazar in #9524
• pageserver: don't return option from DeletionQueue::new by @erikgrinaker in #9588
• Add support of extensions for v17 (part 4) by @lubennikovaav in #9568
• increase lifetime of AWS session token to 12 hours by @Bodobolero in #9590
• pageserver: fix spurious error logs in timeline lifecycle by @jcsp in #9589
• auth_broker: fix local_proxy conn count by @conradludgate in #9593
• update pg_session_jwt to use pgrx 0.12 for pg17 by @conradludgate in #9595
• storage_controller: allow deployment infra to use infra token by @VladLazar in #9596
• fix pre-commit hook with python stubs by @conradludgate in #9602
• test_runner: fix tenant_get_shards with one pageserver by @erikgrinaker in #9603
• revert higher token expiration by @Bodobolero in #9605
• safekeeper: log slow WalAcceptor sends by @erikgrinaker in #9564
• storcon: skip non-active projects in chaos injection by @jcsp in #9606
• Do not assign page LSN to new (uninitialized) page in ClearVisibilityMapFlags redo handler by @knizhnik in #9287
• test_runner/performance: add sharded ingest benchmark by @erikgrinaker in #9591
• pageserver: make local timeline deletion infallible by @jcsp in #9594
• Fix a time zone issue in PG17 test case by @MMeent in #9618
• remove aws-lc-rs dep and fix storage_broker tls by @conradludgate in #9613
• [proxy]: parse proxy protocol TLVs with aws/azure support by @conradludgate in #9610
• Storage & Compute release 2024-11-04 by @vipvap in #9617

Full Changelog: release-7011...release-7044

release-7011

What's Changed

• Compute hotfix release 2024-10-30 by @ololobus in #9577

Full Changelog: release-7009...release-7011

release-7009

What's Changed

• build(deps): bump werkzeug from 3.0.3 to 3.0.6 by @dependabot in #9527
• Don't consider archived timelines for synthetic size calculation by @arpad-m in #9497
• safekeeper: don't un-evict timelines during snapshot API handler by @jcsp in #9428
• storcon: add an API to cancel ongoing reconciler by @jcsp in #9520
• tests: add test_storage_controller_onboard_detached by @jcsp in #9431
• Route pgbouncer logs to virtio-serial by @petuhovskiy in #9488
• ci(proxy): Update GH action flag on proxy deployment by @rahulinux in #9535
• [proxy] demote warnings and remove dead-argument by @conradludgate in #9512
• Add pg_mooncake to compute-node.Dockerfile by @tristan957 in #9498
• We don't have legacy proxies anymore by @SergeyMelnikov in #9544
• safekeeper: refactor WalAcceptor to be event-driven by @erikgrinaker in #9462
• refactor(pageserver): remove aux v1 configs by @skyzh in #9494
• fix(pageserver): bump tokio-epoll-uring by @skyzh in #9546
• Build pgrag extensions x3 by @jawj in #8486
• Create the notion of unstable extensions by @knizhnik in #9540
• Get role name when not a C string by @tristan957 in #9526
• Assert the tenant to be active in unoffload_timeline by @arpad-m in #9539
• pageserver: move things around to prepare for decoding logic by @VladLazar in #9504
• Offloaded timeline deletion by @arpad-m in #9519
• temporarily increase timeout for clickbench benchmark until regression is resolved by @Bodobolero in #9554
• [proxy]: fix JWT handling for AWS cognito. by @conradludgate in #9536
• [auth_broker]: regress test by @conradludgate in #9541
• pageserver: refactor generation-aware loading code into generic by @jcsp in #9545
• pageserver: make concurrent offloaded timeline operations safe wrt manifest uploads by @jcsp in #9557
• fix allow_unstable_extensions GUC - make it USERSET, by @lubennikovaav in #9563
• proxy: more granular http status codes for sql-over-http errors by @mrl5 in #9549
• Use pg_mooncake from our fork. by @lubennikovaav in #9565
• Compute release 2024-10-29 by @lubennikovaav in #9573

Full Changelog: release-6981...release-7009

release-6981

What's Changed

• proxy: Make tests fail fast when test proxy exited early by @cloneable in #9432
• Rename dockerfiles from Dockerfile.<something> to <something>.Dockerfile by @bayandin in #9446
• CI: use build-tools on Debian 12 whenever we use Neon artifact by @bayandin in #9463
• Use '-f' for make clean in Makefile compute by @awarus in #9464
• proxy: Use common error interface for error handling with cplane by @awarus in #9454
• fix(pageserver): make image split layer writer finish atomic by @skyzh in #8841
• pageserver: use direct IO for delta and image layer reads by @yliang412 in #9326
• offloaded timeline list API by @arpad-m in #9461
• chore(compute): bumps pg_session_jwt to latest version by @davidgomes in #9474
• safekeeper: refactor timeline initialization by @arssher in #9362
• Use make's builtin RM variable by @tristan957 in #9468
• storage controller: add metrics for tenant shard, node count by @jcsp in #9475
• Fix local errors for the tests with the versions mix by @a-masterov in #9477
• Fix copy-paste error in checkpoints_timed metric by @tristan957 in #9483
• Timeline offloading persistence by @arpad-m in #9444
• fix(pageserver): make delta split layer writer finish atomic by @skyzh in #9048
• Rename IndexPart::{from_s3_bytes,to_s3_bytes} by @arpad-m in #9481
• proxy: clear lib.rs of code items by @cloneable in #9479
• Protect call to pg_current_wal_lsn() in retained_wal query by @tristan957 in #9425
• tests: avoid using background_process when invoking pg_ctl by @jcsp in #9469
• pageserver: add metric for number of zeroed pages on rel extend by @VladLazar in #9492
• Fix buggy sizeof by @tristan957 in #9495
• refactor(pageserver(load_remote_timeline)) remove dead code handling absence of IndexPart by @problame in #9408
• storcon: respect Reconciler::cancel during await_lsn by @jcsp in #9486
• CI: remove git config --add safe.directory hack by @bayandin in #9391
• refactor(pageserver): refactor split writers using batch layer writer by @skyzh in #9493
• pageserver: refactor ingest inplace to decouple decoding and handling by @VladLazar in #9472
• refactor(timeline creation): represent bootstrap vs branch using enum by @problame in #9366
• pageserver/walingest: log on gappy rel extend by @VladLazar in #9502
• Support offloaded timelines during shard split by @arpad-m in #9489
• fix(test): remove too strict layers_removed==0 check in test_readonly_node_gc by @yliang412 in #9506
• proxy: don't follow redirects for user provided JWKS urls + set custom user agent by @mrl5 in #9514
• pageserver: add testing API to scan layers for disposable keys by @problame in #9393
• remote_storage(local_fs): return correct file sizes by @problame in #9511
• Make test_pageserver_lsn_wait_error_safekeeper_stop less aggressive. by @arssher in #9517
• Move logical replication slot monitor by @tristan957 in #9496
• refactor(timeline creation): idempotency checking by @problame in #9501
• pageserver: fix N^2 I/O when processing relation drops in transaction abort by @jcsp in #9507
• proxy: propagate session IDs by @conradludgate in #9509
• safekeeper: retry eviction. by @arssher in #9485
• safekeeper: fix version in TimelinePersistentState::empty() by @erikgrinaker in #9521
• safekeeper: don't pass conf into storage constructors by @erikgrinaker in #9523
• Fix unoffload_timeline races with creation by @arpad-m in #9525
• pageserver: add tokio-epoll-uring slots waiters queue depth metrics by @yliang412 in #9482
• Storage & Compute release 2024-10-25 by @vipvap in #9528

Full Changelog: release-6935...release-6981

release-6935

What's Changed

• Compute release 2024-10-21 by @tristan957 in #9476

Full Changelog: release-6933...release-6935