History expiry planning

[fjl]

At Devcon 2024, it was decided that clients may drop pre-merge history. We definitely want to act on this in Geth, since it can save a lot of disk space. However, we also want to make it possible for people to access the history if required.

Basic preparations

We need to ensure Geth can operate with the history removed, i.e.

• RPC APIs should work without crashing, and should return an appropriate error
  • Notably, in most RPC APIs, returning null means the item is not found. We probably want to signal
    an error instead of returning null when trying to access a pre-merge block.
• All basic chain access callsites (in package core/ etc.) need to be checked for behavior with missing history.
• Chain indexing needs to be aware of a lower bound for the block number.
• Freezer truncation needs to be implemented.

era1 support

We already have code for reading era files, but it needs to be integrated into the database abstraction as a replacement
of the freezer for pre-merge blocks.

Sparse era1 coverage needs to be supported, i.e. the user should not be expected to always download all era1 files.
There will be no automatic downloading of era1. However, we need to document where/how to get these files.

Portal Network

The shisui2 Portal client will be added as a dependency. In order to integrate it as a chain access method, we need to refactor history access in core/rawdb to use some kind of Database object/interface instead of calling toplevel functions. The portal client can then be integrated at this basic database level.

shisui2 will be configured to store its data into geth's pebble database. We will set a size limit like 10GB. For now, we will not focus on integrating the geth post-merge history freezer with shisui, i.e. the shisui data is fully separate from Geth's DB. We can build the full integration later.

--syncmode=snap

For snap-sync on mainnet, we just have to modify the sync procedure to stop at a checkpoint block at the merge
height, instead of downloading all the way back to genesis. This is a change in eth/downloader.

--syncmode=full

For full sync, the full history is required. The sync process needs to be modified to import into core.BlockChain from
the existing DB for pre-merge blocks, then switch to downloading after the merge is reached.

[lightclient]

Once we have these things implemented, how do you want to expose on the network level that we do not have pre-merge history? EIP-7639 or something else? If we do eth/70, do we need to have that active along with Prague so that there is good service for eth/70 when clients start running only eth/70 and not a combo of 69/70?