Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compiling a JSON schema upon receival of every message callback wastes memory and cycles #332

Open
TimDaub opened this issue Oct 23, 2022 · 1 comment

Comments

@TimDaub
Copy link
Collaborator

TimDaub commented Oct 23, 2022

  • if (message.schema) {
  • we're compiling a JSON schema for every message, of which there can be thousands a second of potentially
  • The problem of passing schema as a property in a worker message is that for each message in lifecycle we never know what to expect and so we have to continuously parse newly received schema JSON formats
  • we could cache them into a global file, but I'm not sure this is pretty either
  • Another option would be to define them somewhat on a strategy level and then load them and compile them when the strategy is loaded. Then we could match by the commissioner. There are pros and cons for this. The pros are that this would further cement our ideal of 1 strategy = 1 data source, the cons are that sometimes we necessarily need two data sources to e.g. get a block number first before crawling all block logs
@TimDaub TimDaub changed the title compiling a JSON schema upon receival of every message callback is very wasteful compiling a JSON schema upon receival of every message callback wastes memory and cycles Oct 23, 2022
@TimDaub
Copy link
Collaborator Author

TimDaub commented Oct 24, 2022

/cc @il3ven for opinions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant