Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement: Have harvester do multiple jobs at the same time #15

Open
mrseeker opened this issue Feb 27, 2023 · 1 comment
Open

Improvement: Have harvester do multiple jobs at the same time #15

mrseeker opened this issue Feb 27, 2023 · 1 comment

Comments

@mrseeker
Copy link

At the moment I am trying to synchronize a polkadot chain which has 3.5 million records, taking almost 3 days to get through the full chain. Is it possible to run other jobs at the same time it's receiving blocks or do they have to be in order?

@arjanz
Copy link
Member

arjanz commented Mar 1, 2023

There are dependencies for certain jobs, but technically speaking some could be running in parallel. In the previous version there actually was a degree of parallelisation, but then we ran into other issues like DB-locking. Also it is much easier to keep track of progress, but the take-away is indeed a longer time for the initial sync.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants