pageserver: guard against WAL gaps in the interpreted protocol

[VladLazar]

Problem

The interpreted SK <-> PS protocol does not guard against gaps (neither does the Vanilla one, but that's beside the point).

Summary of changes

Extend the protocol to include the start LSN of the PG WAL section from which the records were interpreted.
Validation is enabled via a config flag on the pageserver and works as follows:

Case 1: raw_wal_start_lsn is smaller than the requested LSN
There can't be gaps here, but we check that the shard received records which it hasn't seen before.

Case 2: raw_wal_start_lsn is equal to the requested LSN
This is the happy case. No gap and nothing to check

Case 3: raw_wal_start_lsn is greater than the requested LSN
This is a gap.

To make Case 3 work I had to bend the protocol a bit.
We read record chunks of WAL which aren't record aligned and feed them to the decoder.
The picture below shows a shard which subscribes at a position somewhere within Record 2.
We already have a wal reader which is below that position so we wait to catch up.
We read some wal in Read 1 (all of Record 1 and some of Record 2). The new shard doesn't
need Record 1 (it has already processed it according to the starting position), but we read
past it's starting position. When we do Read 2, we decode Record 2 and ship it off to the shard,
but the starting position of Read 2 is greater than the starting position the shard requested.
This looks like a gap.

To make it work, we extend the protocol to send an empty InterpretedWalRecords to shards
if the WAL the records originated from ends the requested start position. On the pageserver,
that just updates the tracking LSNs in memory (no-op really). This gives us a workaround for
the fake gap.

As a drive by, make InterpretedWalRecords::next_record_lsn mandatory in the application level definition.
It's always included.

Related: https://github.com/neondatabase/cloud/issues/23935

[github-actions]

7557 tests run: 7184 passed, 0 failed, 373 skipped (full report)

---

Flaky tests (1)

Postgres 17

• test_migration_to_cold_secondary: release-arm64-without-lfc

Code coverage* (full report)

• functions: 32.9% (8625 of 26199 functions)
• lines: 48.9% (72771 of 148957 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
c642939 at 2025-02-20T15:52:13.121Z :recycle:}