Skip to content

WIP: Libevm upstream sync with pivot change #830

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 230 commits into
base: libevm-upstream-sync
Choose a base branch
from

Conversation

alarso16
Copy link
Contributor

@alarso16 alarso16 commented Feb 24, 2025

There's a few things that I should mention that are mostly related to structure.

  • The downloader.go logic is very similar to that of the downloader.go in geth. The SnapSync() method essentially processSnapSyncContent. I removed a lot of the (seemingly) excessive logic here, but it seems to successfully move the pivot with rather rare errors.
  • Prior to this PR, the state trie sync and atomic trie sync could be done parallelly or in either order. Now, it is changed to completely finish the atomic trie sync before beginning the snap sync, because we will perform atomic operations synchronously throughout the state sync. However, we could likely still do these parallelly. A channel was added to notify when atomic sync is completely, and we can thus start executing atomic operations as normal.
  • We cannot postpone verify/accept operations on atomic operations until the pivot, because this can cause a deadlock with the P-Chain in avalancego. Because of this, we perform atomic operations synchronously, and then after the sync is complete, we perform the non-atomic operations in order as they are buffered to catch up the state trie to the atomic trie.
  • Passing the lock from stateSyncerclient to Downloader is necessary because we need to block from queueing blocks once we finish the state sync, and then once the queue is cleared, we release the lock, allowing blocks to be verified as normal.
  • The weird channels in Downloader weren't my idea - copied from geth with minimal revision. I'm not married to it, if you have a more readable suggestion.

My next steps:

  1. There are likely some issues around exactly how atomic txs are handled during the final parts of the sync, and testing on mainnet (or testing multiple times on fuji)
  2. WIP resuming a sync after canceled by SIGINT or something else.
  3. The server must run in non-pruning mode for this to work, since the pivot can be too far back to respond to requests. A more permanent fix would be increasing the TipToBuffer and adjusting the pivot interval to find a "sweet spot".
  4. None of the FUpgrade is here, so I need to rebase (probably very carefully) and retest to be able to work on fuji again.

Darioush Jalali and others added 30 commits October 15, 2024 20:22
* Remove the bls dependency
---------
Co-authored-by: Darioush Jalali <[email protected]>
* Release notes and avalanchego update

* restore avago version

* bump avago to master

* ammend notes

* go mod tidy

* bump version

* update notes
* new atomic pkg

* bump avago

* bump versions

* change newimport clk to time

* move utils

* bump avago

* go mod tidy

* use address methods from avago

* bump avago

* bump e2e avago version
* new atomic pkg

* bump avago

* bump versions

* move attomic gossip

* lint

* change newimport clk to time

* move utils

* bump avago

* go mod tidy

* update releases md

* use address methods from avago

* bump avago

* bump e2e avago version

* Update plugin/evm/atomic/gossip_test.go

Co-authored-by: Quentin McGaw <[email protected]>
Signed-off-by: Ceyhun Onur <[email protected]>

* Update plugin/evm/atomic/mempool.go

Co-authored-by: Quentin McGaw <[email protected]>
Signed-off-by: Ceyhun Onur <[email protected]>

* Update plugin/evm/config/config.go

Co-authored-by: Quentin McGaw <[email protected]>
Signed-off-by: Ceyhun Onur <[email protected]>

* Update plugin/evm/atomic/tx_heap.go

Co-authored-by: Quentin McGaw <[email protected]>
Signed-off-by: Ceyhun Onur <[email protected]>

* Update plugin/evm/config/config.go

Co-authored-by: Quentin McGaw <[email protected]>
Signed-off-by: Ceyhun Onur <[email protected]>

* fix reviews

---------

Signed-off-by: Ceyhun Onur <[email protected]>
Signed-off-by: Ceyhun Onur <[email protected]>
Co-authored-by: Quentin McGaw <[email protected]>
* header.yml is missing so removing ineffective go-license

* Remove unneeded source constants

* Remove unneeded source versions

* Use go generate commands to generate mocks

* Remove the need to install mockgen in the script

* Add mock-up-to-date step to test CI workflow

* Remove all mocks before generating them in CI

* Add generate command for non-tracked mock
(Note using source mode was compulsory due to CGO)

* Use bash shell for mock check step

* Remove script mock.gen.sh

* docs(contributing): add mocks section

* mockgen version used is the one defined in go.mod
- requires a `tools.go` blank importing golang.org/x/tools/imports in order to have the `golang.org/x/tools` dependency satisfied for mockgen v0.4
- less future deltas when upgrading mockgen

* ci: use more precise go generate regex

* Add `git add --intent-to-add --all` to detect untracked files

* Add copyright notices
* small changes

* relevant warp changes

* use wrapped db package

* nit

* equivalent to subnet-evm 843e3f7

* add multiatch docker build to support arm (#1417)

* remove build_docker_image.sh

---------

Co-authored-by: Ceyhun Onur <[email protected]>
…blocks (#719)

- default behavior for pruning mode to reject blocks before the 32 blocks preceding the last accepted block
- default behavior for archive mode to reject blocks before ~24h worth of blocks preceding the last accepted block
- archive mode new option `historical-proof-query-window` to customize the blocks window, or set it to 0 to accept any block number
Co-authored-by: maskpp <[email protected]>
Co-authored-by: Aaron Chen <[email protected]>
Co-authored-by: Shiming Zhang <[email protected]>
Co-authored-by: Felix Lange <[email protected]>
Co-authored-by: Nathan <[email protected]>
Co-authored-by: Eric <[email protected]>
Co-authored-by: Darioush Jalali <[email protected]>
Co-authored-by: Quentin McGaw <[email protected]>
maru-ava and others added 30 commits March 26, 2025 17:55
- changes applied from comments on the review of #820
- suggested changes from the review on #828 
- add test to check PredicateBytesFromExtra is the inverse of SetPredicateBytesInExtra

Signed-off-by: Quentin McGaw <[email protected]>
Co-authored-by: Arran Schlosberg <[email protected]>
Signed-off-by: Darioush Jalali <[email protected]>
Co-authored-by: Quentin McGaw <[email protected]>
…erface used by x/sync. Both are managed by the dynamic syncer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.