Dealing with journal version upgrades

[sebastianst]

Whenever the journal version is upgraded, a new geth release will discard the outdated journal at startup with a message like

unexpected journal version want 3 got 2

This may then lead to the node being in a broken state, especially if it's a full node, e.g. complaining about missing trie nodes like

lvl=error msg="Failed to create sealing context" err="missing trie node 0a22215a54961846c48037c5b4e6ff243a96041f6262b57fe30f37e94b847442 (path ) state 0x0a22215a54961846c48037c5b4e6ff243a96041f6262b57fe30f37e94b847442 is not available"

The node then has to be manually recovered.

The list of journal version upgrades includes:

• To version 1: core/state, trie/triedb/pathdb: remove storage incomplete flag #28940
• To version 2: triedb/pathdb: track flat state changes in pathdb (snapshot integration pt 2) #30643
• To version 3: all: implement state history v2 #30107

When geth went from 0 to 1, we introduced a journal version upgrade path in op-geth (ethereum-optimism/op-geth#368). But it doesn't seem like a scalable approach to always add upgrade paths, now going from 1 to 2 and 3.

What is the recommended way how to deal with journal version upgrades? I couldn't find any recommendations in the geth release notes.

[rjl493456442]

This may then lead to the node being in a broken state, especially if it's a full node, e.g. complaining about missing trie nodes like

This behavior is not expected in Geth. The discarded journal corresponds to the layers in memory. Geth can recover from losing all memory states, which is equivalent to an unclean shutdown.

Maybe it's something specific with op-Geth?

[sebastianst]

This may then lead to the node being in a broken state, especially if it's a full node, e.g. complaining about missing trie nodes like

This behavior is not expected in Geth. The discarded journal corresponds to the layers in memory. Geth can recover from losing all memory states, which is equivalent to an unclean shutdown.

Maybe it's something specific with op-Geth?

Thanks for reply! Interesting, we'll investigate if it has something to do with our diff then. It doesn't always happen, just sometimes.

[fjl]

It could also be a bug of course!

[protolambda]

I investigated the logs of our node during the restart that removed the journal, and the pre-restart logs, and found: https://gist.github.com/protolambda/7e2002a0de7cf868fbc1617fffa656cd

I believe the journal became too large due to a large write-buffer during shutdown, and once the node removed the journal due to the version change, the gap was so large that the state was unavailable.

I opened a potential fix in op-geth to limit the number of difflayers in the write-buffer: ethereum-optimism/op-geth#497
If you think that's right, I can cherry-pick it and open a PR on upstream geth.

[sebastianst]

reopening, as we have only (hopefully) fixed it in our fork