Etherlink testnet does not index from 0 or forced index

[zamrokk]

Steps to reproduce:

Rerun the Dipdup Etherlink tutorial

What did you expect to happen:

Index from a specific index block number or even from 0

What actually happened:

Indexing from the last block

Environment

• DipDup version:8.1.2
• docker compose

INFO     dipdup.index         marketpulse_events: Index is behind the datasource level, syncing: 0 -> 16629146
INFO     dipdup.index         marketpulse_events: Synchronizing index to level 16629146

PS : I tried to use dipdup wipe, remove the docker volume, forcing from 0 or a specific index, removing the field first_level, etc ... It always start from the last block even it it seems to find the configuration on the logs

PPS: here is my dipdup project to play with : https://github.com/zamrokk/marketpulse-v2/tree/main/dipdup_indexer

[zamrokk]

I updated to dipdup 8.1.4, wiped the Db and started dipdup @droserasprout

I have a new issue

Any idea ?

zamrokk@wearelegion:~/marketpulse-v2/dipdup_indexer$    dipdup run
INFO     dipdup.package       Loading package `dipdup_indexer` from `/home/zamrokk/marketpulse-v2/dipdup_indexer`
INFO     dipdup.dipdup        Setting up database
INFO     dipdup.dipdup        Initializing database schema
INFO     dipdup.database      Reading SQL from `dipdup_approve.sql`
INFO     dipdup.database      Reading SQL from `dipdup_wipe.sql`
INFO     dipdup.database      Reading SQL from `dipdup_status.sql`
INFO     dipdup.hasura        Configuring Hasura
INFO     dipdup.hasura        Connecting to Hasura instance
INFO     dipdup.hasura        Connected to Hasura v2.17.1
INFO     dipdup.hasura        Fetching existing metadata
INFO     dipdup.hasura        Generating Hasura metadata based on project models
INFO     dipdup.hasura        Replacing metadata
INFO     dipdup.hasura        Applying table customizations (could take some time)
INFO     dipdup.hasura        Fetching existing metadata
INFO     dipdup.hasura        Adding 10 generated and user-defined queries
INFO     dipdup.hasura        Adding 10 REST endpoints
INFO     dipdup.hasura        Replacing metadata
INFO     dipdup.hasura        Fetching existing metadata
INFO     dipdup.hasura        Hasura instance has been configured
INFO     dipdup.scheduler     Starting job scheduler
INFO     dipdup.dipdup        Waiting for indexes to synchronize before spawning datasources
INFO     dipdup.dipdup        Starting index dispatcher
INFO     dipdup.dipdup        0 contracts fetched from database
INFO     dipdup.dipdup        1 indexes found in database
INFO     dipdup.index         marketpulse_events: Index is behind the datasource level, syncing: 16297151 -> 16841663
INFO     dipdup.index         marketpulse_events: Synchronizing index to level 16841663
INFO     dipdup.index         marketpulse_events: Status updated: IndexStatus.new -> IndexStatus.syncing
INFO     dipdup.index         marketpulse_events: Subsquid is 15673 levels behind; 528839 available
Traceback (most recent call last):
  File "/home/zamrokk/.local/bin/dipdup", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/cli.py", line 189, in wrapper
    raise e
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/cli.py", line 180, in wrapper
    uvloop.run(fn(ctx, *args, **kwargs))
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/cli.py", line 341, in run
    await dipdup.run()
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/dipdup.py", line 720, in run
    await gather(*tasks)
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/dipdup.py", line 155, in run
    indexes_processed = any(await gather(*tasks))
                            ^^^^^^^^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/index.py", line 267, in process
    await self._synchronize(sync_level)
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/indexes/_subsquid.py", line 81, in _synchronize
    await self._synchronize_subsquid(sync_level)
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/indexes/evm_events/index.py", line 33, in _synchronize_subsquid
    await self._process_level_data(logs, sync_level)
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/index.py", line 135, in _process_level_data
    matched_handlers = self._match_level_data(self._config.handlers, level_data)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/indexes/evm_events/index.py", line 97, in _match_level_data
    return match_events(
           ^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/indexes/evm_events/matcher.py", line 125, in match_events
    arg = prepare_event_handler_args(package, handler_config, event)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/indexes/evm_events/matcher.py", line 86, in prepare_event_handler_args
    typed_payload = parse_object(
                    ^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/utils.py", line 220, in parse_object
    model_dict[k] = parse_object(nested_type, v, plain=True)  # type: ignore[arg-type]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/utils.py", line 209, in parse_object
    model_keys = tuple(field.alias or key for key, field in type_.model_fields.items())
                                                            ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/typing.py", line 1185, in __getattr__
    return getattr(self.__origin__, attr)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: type object 'dict' has no attribute 'model_fields'
________________________________________________________________________________

An unexpected error has occurred! Most likely it's a framework bug.

Please, tell us about it: https://github.com/dipdup-io/dipdup/issues

Report saved; run `dipdup report show 8914442318` to view it

[droserasprout]

Hi @zamrokk !

I tried to test this fix on your project, but stuck with some non DipDup related problem; can't remember exactly, but not the one you're reporting. If you could create a separate issue with some context it would help greatly, as we currently have little time for investigation.

Consider running DipDup with DIPDUP_DEBUG=1 env variable and checking crash reports with dipdup report show latest command.

[zamrokk]

FYI, only look at the dipdup_indexer folder, this is where is the dipdup project, you don't need to look/depend on other folders

/me trying to rerun now

[zamrokk]

level: error
exception:
  values:
    - mechanism:
        type: generic
        handled: true
      type: AttributeError
      value: type object 'dict' has no attribute 'model_fields'
      stacktrace:
        frames:
          - filename: dipdup/cli.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/cli.py
            function: wrapper
            module: dipdup.cli
            lineno: 180
            vars:
              ctx: <click.core.Context object at 0x7fe9eb1bf3b0>
              args: []
              kwargs: {}
              e: AttributeError("type object 'dict' has no attribute 'model_fields'")
              save_report: <function save_report at 0x7fe9e85f1620>
              package: "'dipdup_indexer'"
              fn: <function run at 0x7fe9ea0d7560>
            code:
              - ''
              - 'def _cli_wrapper(fn: WrappedCommandT) -> WrappedCommandT:'
              - '    @wraps(fn)'
              - '    def wrapper(ctx: click.Context, *args: Any, **kwargs: Any) ->
                None:'
              - '        try:'
              - '            uvloop.run(fn(ctx, *args, **kwargs))'
              - '        except (KeyboardInterrupt, asyncio.CancelledError):'
              - '            pass'
              - '        except Exception as e:'
              - '            from dipdup.report import save_report'
              - ''
          - filename: __init__.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/uvloop/__init__.py
            function: run
            module: uvloop
            lineno: 109
            vars:
              main: <coroutine object run at 0x7fe9ea48af80>
              loop_factory: <function new_event_loop at 0x7fe9ea3d34c0>
              debug: None
              run_kwargs: {}
              wrapper: <function run.<locals>.wrapper at 0x7fe9eb3fa200>
              vi:
                - '3'
                - '12'
            code:
              - '            ) as runner:'
              - '                return runner.run(wrapper())'
              - ''
              - '        else:'
              - '            assert vi >= (3, 12)'
              - '            return __asyncio.run('
              - '                wrapper(),'
              - '                loop_factory=loop_factory,'
              - '                debug=debug,'
              - '                **run_kwargs'
              - '            )'
          - filename: asyncio/runners.py
            abs_path: /usr/lib/python3.12/asyncio/runners.py
            function: run
            module: asyncio.runners
            lineno: 194
            vars:
              main: <coroutine object run.<locals>.wrapper at 0x7fe9ea452f60>
              debug: None
              loop_factory: <function new_event_loop at 0x7fe9ea3d34c0>
              runner: <asyncio.runners.Runner object at 0x7fe9ea0de330>
            code:
              - '        # fail fast with short traceback'
              - '        raise RuntimeError('
              - '            "asyncio.run() cannot be called from a running event
                loop")'
              - ''
              - '    with Runner(debug=debug, loop_factory=loop_factory) as runner:'
              - '        return runner.run(main)'
              - ''
              - ''
              - 'def _cancel_all_tasks(loop):'
              - '    to_cancel = tasks.all_tasks(loop)'
              - '    if not to_cancel:'
          - filename: asyncio/runners.py
            abs_path: /usr/lib/python3.12/asyncio/runners.py
            function: run
            module: asyncio.runners
            lineno: 118
            vars:
              self: <asyncio.runners.Runner object at 0x7fe9ea0de330>
              coro: <coroutine object run.<locals>.wrapper at 0x7fe9ea452f60>
              context: <_contextvars.Context object at 0x7fe9ea0e2540>
              task: <Task finished name='Task-4' coro=<run.<locals>.wrapper() done,
                defined at 
                /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/uvloop/__init__.py:54>
                exception=AttributeError("type object 'dict' has no attribute 'model_fields'")>
              sigint_handler: functools.partial(<bound method Runner._on_sigint of
                <asyncio.runners.Runner object at 0x7fe9ea0de330>>, main_task=<Task
                finished name='Task-4' coro=<run.<locals>.wrapper() done, defined
                at 
                /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/uvloop/__init__.py:54>
                exception=AttributeError("type object 'dict' has no attribute 'model_fields'")>)
            code:
              - '        else:'
              - '            sigint_handler = None'
              - ''
              - '        self._interrupt_count = 0'
              - '        try:'
              - '            return self._loop.run_until_complete(task)'
              - '        except exceptions.CancelledError:'
              - '            if self._interrupt_count > 0:'
              - '                uncancel = getattr(task, "uncancel", None)'
              - '                if uncancel is not None and uncancel() == 0:'
              - '                    raise KeyboardInterrupt()'
          - filename: uvloop/loop.pyx
            abs_path: /home/zamrokk/marketpulse-v2/dipdup_indexer/uvloop/loop.pyx
            function: uvloop.loop.Loop.run_until_complete
            module: uvloop.loop
            lineno: 1518
            vars: {}
            code:
              - '        finally:'
              - '            future.remove_done_callback(done_cb)'
              - '        if not future.done():'
              - "            raise RuntimeError('Event loop stopped before Future
                completed.')"
              - ''
              - '        return future.result()'
              - ''
              - '    @cython.iterable_coroutine'
              - '    async def getaddrinfo(self, object host, object port, *,'
              - '                          int family=0, int type=0, int proto=0,
                int flags=0):'
              - ''
          - filename: __init__.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/uvloop/__init__.py
            function: wrapper
            module: uvloop
            lineno: 61
            vars:
              loop: <uvloop.Loop running=False closed=True debug=False>
              main: <coroutine object run at 0x7fe9ea48af80>
            code:
              - '            # either uvloop.Loop or a subtype of it, assuming the
                user'
              - '            # is using `uvloop.run()` intentionally.'
              - '            loop = __asyncio._get_running_loop()'
              - '            if not isinstance(loop, Loop):'
              - "                raise TypeError('uvloop.run() uses a non-uvloop event
                loop')"
              - '            return await main'
              - ''
              - '        vi = _sys.version_info[:2]'
              - ''
              - '        if vi <= (3, 10):'
              - '            # Copied from python/cpython'
          - filename: dipdup/cli.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/cli.py
            function: run
            module: dipdup.cli
            lineno: 341
            vars:
              ctx: <click.core.Context object at 0x7fe9eb1bf3b0>
              DipDup: <class 'dipdup.dipdup.DipDup'>
              config: "DipDupConfig(spec_version='3.0', package='dipdup_indexer',
                datasources={'subsquid': EvmSubsquidDatasourceConfig(kind='evm.subsquid',
                url='https://v2.archive.subsquid.io/network/etherlink-testnet', http=None),
                'etherscan': AbiEtherscanDatasourceConfig(kind='abi.etherscan', url='https://testnet.explorer.etherlink.com/api',
                api_key=None, http=None), 'evm_node': EvmNodeDatasourceConfig(kind='evm.node',
                url='https://node.ghostnet.etherlink.com', ws_url=None, http=None,
                rollback_depth=32)}, database=PostgresDatabaseConfig(kind='postgres',
                host='localhost', user='dipdup', database='dipdup', port=5432, schema_name='public',
                immune_tables=set(), connection_timeout=60), contracts={'marketpulse':
                EvmContractConfig(kind='evm', address='0x386dc5e8e0f8252880cfa9b9e607c749899bf13a',
                abi=None, typename=None)}, indexes={'marketpulse_events': EvmEventsIndexConfig(kind='evm.events',
                datasources=(EvmSubsquidDatasourceConfig(kind='evm.subsquid', url='https://v2.archive.subsquid.io/network/etherlink-testnet',
                http=None), AbiEth..."
              dipdup: DipDup(dipdup_indexer)
            code:
              - ''
              - '    config: DipDupConfig = ctx.obj.config'
              - '    config.initialize()'
              - ''
              - '    dipdup = DipDup(config)'
              - '    await dipdup.run()'
              - ''
              - ''
              - '@cli.command()'
              - "@click.option('--force', '-f', is_flag=True, help='Overwrite existing
                types and ABIs.')"
              - "@click.option('--base', '-b', is_flag=True, help='Include template
                base: pyproject.toml, Dockerfile, etc.')"
          - filename: dipdup/dipdup.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/dipdup.py
            function: run
            module: dipdup.dipdup
            lineno: 720
            vars:
              self: DipDup(dipdup_indexer)
              advanced: AdvancedConfig(reindex={}, scheduler=None, postpone_jobs=False,
                early_realtime=False, rollback_depth=32, decimal_precision=None, unsafe_sqlite=False,
                alt_operation_matcher=False)
              tasks:
                - <Task cancelled name='loop:status_loop' coro=<IndexDispatcher._status_loop()
                  done, defined at 
                  /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/dipdup.py:302>>
                - <Task cancelled name='loop:datasources' coro=<DipDup._spawn_datasources.<locals>._event_wrapper()
                  done, defined at 
                  /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/dipdup.py:904>>
                - <Task finished name='loop:scheduler' coro=<SchedulerManager.run()
                  done, defined at 
                  /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/scheduler.py:61>
                  result=None>
                - <Task finished name='loop:preload_cached_models' coro=<preload_cached_models()
                  done, defined at 
                  /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/database.py:465>
                  result=None>
                - <Task cancelled name='loop:metrics_loop' coro=<IndexDispatcher._metrics_loop()
                  done, defined at 
                  /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/dipdup.py:210>>
                - <Task cancelled name='loop:cleanup_loop' coro=<IndexDispatcher._cleanup_loop()
                  done, defined at 
                  /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/dipdup.py:215>>
                - <Task finished name='loop:run' coro=<IndexDispatcher.run() done,
                  defined at 
                  /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/dipdup.py:116>
                  exception=AttributeError("type object 'dict' has no attribute 'model_fields'")>
                - <Task cancelled name='loop:hooks_loop' coro=<DipDupContext._hooks_loop()
                  done, defined at 
                  /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/context.py:577>>
              stack: <contextlib.AsyncExitStack object at 0x7fe9e72d3890>
              hasura_gateway: <dipdup.hasura.HasuraGateway object at 0x7fe9e69ae300>
              name: "'marketpulse_events'"
              start_scheduler_event: <asyncio.locks.Event object at 0x7fe9e8ace0f0
                [set]>
              spawn_datasources_event: <asyncio.locks.Event object at 0x7fe9e47195b0
                [unset]>
            code:
              - '                start_scheduler_event=start_scheduler_event,'
              - '                early_realtime=advanced.early_realtime,'
              - '            )'
              - ''
              - '            if tasks:'
              - '                await gather(*tasks)'
              - ''
              - '    async def _create_datasources(self) -> None:'
              - '        for name, config in self._config.datasources.items():'
              - '            if name not in self._datasources:'
              - '                self._datasources[name] = create_datasource(config)'
          - filename: dipdup/dipdup.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/dipdup.py
            function: run
            module: dipdup.dipdup
            lineno: 155
            vars:
              self: IndexDispatcher(1)
              spawn_datasources_event: <asyncio.locks.Event object at 0x7fe9e47195b0
                [unset]>
              start_scheduler_event: <asyncio.locks.Event object at 0x7fe9e8ace0f0
                [set]>
              early_realtime: 'False'
              on_synchronized_fired: 'False'
              on_realtime_fired: 'False'
              index: EvmEventsIndex(name=marketpulse_events, level=16297151, status=IndexStatus.syncing,
                queue=0)>
              tasks:
                - <coroutine object Index.process at 0x7fe9e4843970>
              _: "'marketpulse_events'"
              indexes_processed: 'False'
            code:
              - '                if index.state.status == IndexStatus.disabled:'
              - '                    continue'
              - ''
              - '                tasks.append(index.process())'
              - ''
              - '            indexes_processed = any(await gather(*tasks))'
              - '            indexes_spawned = False'
              - ''
              - '            while not self._ctx._pending_indexes.empty():'
              - '                index = self._ctx._pending_indexes.get_nowait()'
              - '                self._indexes[index._config.name] = index'
          - filename: dipdup/index.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/index.py
            function: process
            module: dipdup.index
            lineno: 267
            vars:
              self: EvmEventsIndex(name=marketpulse_events, level=16297151, status=IndexStatus.syncing,
                queue=0)>
              last_level: '0'
              index_level: '16297151'
              sync_level: '16859639'
            code:
              - '        if index_level < sync_level:'
              - "            self._logger.info('Index is behind the datasource level,
                syncing: %s -> %s', index_level, sync_level)"
              - '            self.queue.clear()'
              - ''
              - '            with metrics._index_total_sync_duration.time():'
              - '                await self._synchronize(sync_level)'
              - '                return True'
              - ''
              - '        if self.queue:'
              - '            with metrics._index_total_realtime_duration.time():'
              - '                await self._process_queue()'
          - filename: dipdup/indexes/_subsquid.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/indexes/_subsquid.py
            function: _synchronize
            module: dipdup.indexes._subsquid
            lineno: 81
            vars:
              self: EvmEventsIndex(name=marketpulse_events, level=16297151, status=IndexStatus.syncing,
                queue=0)>
              sync_level: '16858230'
              index_level: '16297151'
              levels_left: '562488'
              subsquid_sync_level: '16858230'
              node_sync_level: None
            code:
              - '            sync_level = min(sync_level, node_sync_level)'
              - "            self._logger.debug('Using node datasource; sync level:
                %s', sync_level)"
              - '            await self._synchronize_node(sync_level)'
              - '        else:'
              - '            sync_level = min(sync_level, subsquid_sync_level)'
              - '            await self._synchronize_subsquid(sync_level)'
              - ''
              - '        if not self.node_datasources and not self._subsquid_started:'
              - '            self._subsquid_started = True'
              - "            self._logger.info('No `evm.node` datasources available;
                polling Subsquid')"
              - '            for datasource in self.subsquid_datasources:'
          - filename: dipdup/indexes/evm_events/index.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/indexes/evm_events/index.py
            function: _synchronize_subsquid
            module: dipdup.indexes.evm_events.index
            lineno: 33
            vars:
              self: EvmEventsIndex(name=marketpulse_events, level=16297151, status=IndexStatus.syncing,
                queue=0)>
              sync_level: '16858230'
              first_level: '16297152'
              fetcher: EvmSubsquidEventFetcher(name=marketpulse_events, head=0, buffer=0)
              _level: '16332523'
              logs:
                - EvmEventData(address='0x386dc5e8e0f8252880cfa9b9e607c749899bf13a',
                  block_hash='0x93d482c6deee63bce71cb09d9e9a1658a3b6cc170de43a67ba5e12a1c1d8655e',
                  data='0x0000000000000000000000000000000000000000000000000000000000000020347ea556a21638c65f924e538991f79006225ede71eb14e2fe7b962ea0f29e8a000000000000000000000000a73feb91fe6df51a6e36f2566030a0ab5c67646d0000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000e8d4a5100000000000000000000000000000000000000000000000000000000000000000057472756d70000000000000000000000000000000000000000000000000000000',
                  level=16332523, log_index=0, removed=False, timestamp=1734428215,
                  topics=('0x3da3958a2d60288c66fc825e0a5e95a83e03662afbe1595a85c7a47742264293',),
                  transaction_hash='0x2bc08aa1b08f4bba28934893f2163adec7ca62f6a8e69f9a1eeda7a585450bcb',
                  transaction_index=1)
            code:
              - '    async def _synchronize_subsquid(self, sync_level: int) -> None:'
              - '        first_level = self.state.level + 1'
              - '        fetcher = self._create_subsquid_fetcher(first_level, sync_level)'
              - ''
              - '        async for _level, logs in fetcher.fetch_by_level():'
              - '            await self._process_level_data(logs, sync_level)'
              - '            metrics._sqd_processor_last_block = _level'
              - ''
              - '    async def _synchronize_node(self, sync_level: int) -> None:'
              - '        first_level = self.state.level + 1'
              - '        fetcher = self._create_node_fetcher(first_level, sync_level)'
          - filename: dipdup/index.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/index.py
            function: _process_level_data
            module: dipdup.index
            lineno: 135
            vars:
              self: EvmEventsIndex(name=marketpulse_events, level=16297151, status=IndexStatus.syncing,
                queue=0)>
              level_data:
                - EvmEventData(address='0x386dc5e8e0f8252880cfa9b9e607c749899bf13a',
                  block_hash='0x93d482c6deee63bce71cb09d9e9a1658a3b6cc170de43a67ba5e12a1c1d8655e',
                  data='0x0000000000000000000000000000000000000000000000000000000000000020347ea556a21638c65f924e538991f79006225ede71eb14e2fe7b962ea0f29e8a000000000000000000000000a73feb91fe6df51a6e36f2566030a0ab5c67646d0000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000e8d4a5100000000000000000000000000000000000000000000000000000000000000000057472756d70000000000000000000000000000000000000000000000000000000',
                  level=16332523, log_index=0, removed=False, timestamp=1734428215,
                  topics=('0x3da3958a2d60288c66fc825e0a5e95a83e03662afbe1595a85c7a47742264293',),
                  transaction_hash='0x2bc08aa1b08f4bba28934893f2163adec7ca62f6a8e69f9a1eeda7a585450bcb',
                  transaction_index=1)
              sync_level: '16858230'
              index_level: '16297151'
              started_at: '1736934020.5532908'
              MatchedHandler: <class 'dipdup.index.MatchedHandler'>
              batch_level: '16332523'
            code:
              - "            raise FrameworkException(f'Batch level is lower than
                index level: {batch_level} <= {index_level}')"
              - ''
              - "        self._logger.debug('Processing data of level %s', batch_level)"
              - '        started_at = time.time()'
              - ''
              - '        matched_handlers = self._match_level_data(self._config.handlers,
                level_data)'
              - ''
              - '        total_matched = len(matched_handlers)'
              - '        # metrics.set_index_handlers_matched(total_matched)'
              - '        metrics.handlers_matched[self.name] += total_matched'
              - '        metrics.time_in_matcher[self.name] += time.time() - started_at'
          - filename: dipdup/indexes/evm_events/index.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/indexes/evm_events/index.py
            function: _match_level_data
            module: dipdup.indexes.evm_events.index
            lineno: 97
            vars:
              self: EvmEventsIndex(name=marketpulse_events, level=16297151, status=IndexStatus.syncing,
                queue=0)>
              handlers:
                - EvmEventsHandlerConfig(callback='on_newbet', contract=EvmContractConfig(kind='evm',
                  address='0x386dc5e8e0f8252880cfa9b9e607c749899bf13a', abi=None,
                  typename=None), name='NewBet')
              level_data:
                - EvmEventData(address='0x386dc5e8e0f8252880cfa9b9e607c749899bf13a',
                  block_hash='0x93d482c6deee63bce71cb09d9e9a1658a3b6cc170de43a67ba5e12a1c1d8655e',
                  data='0x0000000000000000000000000000000000000000000000000000000000000020347ea556a21638c65f924e538991f79006225ede71eb14e2fe7b962ea0f29e8a000000000000000000000000a73feb91fe6df51a6e36f2566030a0ab5c67646d0000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000e8d4a5100000000000000000000000000000000000000000000000000000000000000000057472756d70000000000000000000000000000000000000000000000000000000',
                  level=16332523, log_index=0, removed=False, timestamp=1734428215,
                  topics=('0x3da3958a2d60288c66fc825e0a5e95a83e03662afbe1595a85c7a47742264293',),
                  transaction_hash='0x2bc08aa1b08f4bba28934893f2163adec7ca62f6a8e69f9a1eeda7a585450bcb',
                  transaction_index=1)
            code:
              - '    def _match_level_data('
              - '        self,'
              - '        handlers: tuple[EvmEventsHandlerConfig, ...],'
              - '        level_data: Iterable[EvmEventData],'
              - '    ) -> deque[Any]:'
              - '        return match_events('
              - '            package=self._ctx.package,'
              - '            handlers=handlers,'
              - '            events=level_data,'
              - '        )'
          - filename: dipdup/indexes/evm_events/matcher.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/indexes/evm_events/matcher.py
            function: match_events
            module: dipdup.indexes.evm_events.matcher
            lineno: 125
            vars:
              package: DipDupPackage(/home/zamrokk/marketpulse-v2/dipdup_indexer)
              handlers:
                - EvmEventsHandlerConfig(callback='on_newbet', contract=EvmContractConfig(kind='evm',
                  address='0x386dc5e8e0f8252880cfa9b9e607c749899bf13a', abi=None,
                  typename=None), name='NewBet')
              events:
                - EvmEventData(address='0x386dc5e8e0f8252880cfa9b9e607c749899bf13a',
                  block_hash='0x93d482c6deee63bce71cb09d9e9a1658a3b6cc170de43a67ba5e12a1c1d8655e',
                  data='0x0000000000000000000000000000000000000000000000000000000000000020347ea556a21638c65f924e538991f79006225ede71eb14e2fe7b962ea0f29e8a000000000000000000000000a73feb91fe6df51a6e36f2566030a0ab5c67646d0000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000e8d4a5100000000000000000000000000000000000000000000000000000000000000000057472756d70000000000000000000000000000000000000000000000000000000',
                  level=16332523, log_index=0, removed=False, timestamp=1734428215,
                  topics=('0x3da3958a2d60288c66fc825e0a5e95a83e03662afbe1595a85c7a47742264293',),
                  transaction_hash='0x2bc08aa1b08f4bba28934893f2163adec7ca62f6a8e69f9a1eeda7a585450bcb',
                  transaction_index=1)
              matched_handlers: []
              event: EvmEventData(address='0x386dc5e8e0f8252880cfa9b9e607c749899bf13a',
                block_hash='0x93d482c6deee63bce71cb09d9e9a1658a3b6cc170de43a67ba5e12a1c1d8655e',
                data='0x0000000000000000000000000000000000000000000000000000000000000020347ea556a21638c65f924e538991f79006225ede71eb14e2fe7b962ea0f29e8a000000000000000000000000a73feb91fe6df51a6e36f2566030a0ab5c67646d0000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000e8d4a5100000000000000000000000000000000000000000000000000000000000000000057472756d70000000000000000000000000000000000000000000000000000000',
                level=16332523, log_index=0, removed=False, timestamp=1734428215,
                topics=('0x3da3958a2d60288c66fc825e0a5e95a83e03662afbe1595a85c7a47742264293',),
                transaction_hash='0x2bc08aa1b08f4bba28934893f2163adec7ca62f6a8e69f9a1eeda7a585450bcb',
                transaction_index=1)
              handler_config: EvmEventsHandlerConfig(callback='on_newbet', contract=EvmContractConfig(kind='evm',
                address='0x386dc5e8e0f8252880cfa9b9e607c749899bf13a', abi=None, typename=None),
                name='NewBet')
              typename: "'marketpulse'"
              abi:
                name: "'NewBet'"
                topic0: "'0x3da3958a2d60288c66fc825e0a5e95a83e03662afbe1595a85c7a47742264293'"
                inputs:
                  -   - "'(uint256,address,string,uint256)'"
                      - 'False'
                topic_count: '0'
              address: "'0x386dc5e8e0f8252880cfa9b9e607c749899bf13a'"
            code:
              - ''
              - '            address = handler_config.contract.address'
              - '            if address and address != event.address:'
              - '                continue'
              - ''
              - '            arg = prepare_event_handler_args(package, handler_config,
                event)'
              - '            matched_handlers.append((handler_config, arg))'
              - '            break'
              - ''
              - "    _logger.debug('%d handlers matched', len(matched_handlers))"
              - '    return matched_handlers'
          - filename: dipdup/indexes/evm_events/matcher.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/indexes/evm_events/matcher.py
            function: prepare_event_handler_args
            module: dipdup.indexes.evm_events.matcher
            lineno: 86
            vars:
              package: DipDupPackage(/home/zamrokk/marketpulse-v2/dipdup_indexer)
              handler_config: EvmEventsHandlerConfig(callback='on_newbet', contract=EvmContractConfig(kind='evm',
                address='0x386dc5e8e0f8252880cfa9b9e607c749899bf13a', abi=None, typename=None),
                name='NewBet')
              matched_event: EvmEventData(address='0x386dc5e8e0f8252880cfa9b9e607c749899bf13a',
                block_hash='0x93d482c6deee63bce71cb09d9e9a1658a3b6cc170de43a67ba5e12a1c1d8655e',
                data='0x0000000000000000000000000000000000000000000000000000000000000020347ea556a21638c65f924e538991f79006225ede71eb14e2fe7b962ea0f29e8a000000000000000000000000a73feb91fe6df51a6e36f2566030a0ab5c67646d0000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000e8d4a5100000000000000000000000000000000000000000000000000000000000000000057472756d70000000000000000000000000000000000000000000000000000000',
                level=16332523, log_index=0, removed=False, timestamp=1734428215,
                topics=('0x3da3958a2d60288c66fc825e0a5e95a83e03662afbe1595a85c7a47742264293',),
                transaction_hash='0x2bc08aa1b08f4bba28934893f2163adec7ca62f6a8e69f9a1eeda7a585450bcb',
                transaction_index=1)
              typename: "'marketpulse'"
              inputs:
                -   - "'(uint256,address,string,uint256)'"
                    - 'False'
              type_: <class 'dipdup_indexer.types.marketpulse.evm_events.new_bet.NewBetPayload'>
              data:
                -   - '23744031980264295198031860428657975945071486402437129818234307741465995026058'
                    - "'0xa73feb91fe6df51a6e36f2566030a0ab5c67646d'"
                    - "'trump'"
                    - '1000000000000'
            code:
              - '        data=matched_event.data,'
              - '        topics=tuple(matched_event.topics),'
              - '        inputs=inputs,'
              - '    )'
              - ''
              - '    typed_payload = parse_object('
              - '        type_=type_,'
              - '        data=data,'
              - '        plain=True,'
              - '        nested=True,'
              - '    )'
          - filename: dipdup/utils.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/utils.py
            function: parse_object
            module: dipdup.utils
            lineno: 220
            vars:
              type_: <class 'dipdup_indexer.types.marketpulse.evm_events.new_bet.NewBetPayload'>
              data:
                -   - '23744031980264295198031860428657975945071486402437129818234307741465995026058'
                    - "'0xa73feb91fe6df51a6e36f2566030a0ab5c67646d'"
                    - "'trump'"
                    - '1000000000000'
              plain: 'True'
              nested: 'True'
              model_keys:
                - "'bet'"
              model_dict:
                bet:
                  - '23744031980264295198031860428657975945071486402437129818234307741465995026058'
                  - "'0xa73feb91fe6df51a6e36f2566030a0ab5c67646d'"
                  - "'trump'"
                  - '1000000000000'
              k: "'bet'"
              v:
                - '23744031980264295198031860428657975945071486402437129818234307741465995026058'
                - "'0xa73feb91fe6df51a6e36f2566030a0ab5c67646d'"
                - "'trump'"
                - '1000000000000'
              field_k: "'bet'"
              nested_type: typing.Dict[str, dipdup.fields.Any]
            code:
              - '                    continue'
              - ''
              - '                # NOTE: Might be `from_` or other reserved keyword'
              - "                field_k = '{k}_ ' if k not in type_.model_fields
                else k"
              - '                nested_type = type_.model_fields[field_k].annotation'
              - '                model_dict[k] = parse_object(nested_type, v, plain=True)  #
                type: ignore[arg-type]'
              - ''
              - '        return type_(**model_dict)'
              - '    except ValidationError as e:'
              - "        raise InvalidDataError(f'Failed to parse: {e.errors()}',
                type_, data) from e"
              - '    except ValueError as e:'
          - filename: dipdup/utils.py
            abs_path: 
              /home/zamrokk/.local/share/pipx/venvs/dipdup/lib/python3.12/site-packages/dipdup/utils.py
            function: parse_object
            module: dipdup.utils
            lineno: 209
            vars:
              type_: typing.Dict[str, dipdup.fields.Any]
              data:
                - '23744031980264295198031860428657975945071486402437129818234307741465995026058'
                - "'0xa73feb91fe6df51a6e36f2566030a0ab5c67646d'"
                - "'trump'"
                - '1000000000000'
              plain: 'True'
              nested: 'False'
            code:
              - ') -> ObjectT:'
              - '    try:'
              - '        if plain is False or data is None:'
              - '            return type_.model_validate(data)'
              - ''
              - '        model_keys = tuple(field.alias or key for key, field in type_.model_fields.items())'
              - '        model_dict = dict(zip(model_keys, data, strict=True))'
              - ''
              - '        if nested:'
              - '            for k, v in model_dict.items():'
              - '                if not isinstance(v, list | tuple):'
          - filename: typing.py
            abs_path: /usr/lib/python3.12/typing.py
            function: __getattr__
            module: typing
            lineno: 1185
            vars:
              self: typing.Dict[str, dipdup.fields.Any]
              attr: "'model_fields'"
            code:
              - '            return self._name or self.__origin__.__name__'
              - ''
              - '        # We are careful for copy and pickle.'
              - "        # Also for simplicity we don't relay any dunder names"
              - "        if '__origin__' in self.__dict__ and not _is_dunder(attr):"
              - '            return getattr(self.__origin__, attr)'
              - '        raise AttributeError(attr)'
              - ''
              - '    def __setattr__(self, attr, val):'
              - "        if _is_dunder(attr) or attr in {'_name', '_inst', '_nparams'}:"
              - '            super().__setattr__(attr, val)'
metrics:
  caches: {}
  queues:
    marketpulse_events:realtime:
      size: 0
    marketpulse_events:readahead:
      size: 1
      limit: 10000
      full: 0.0001
  metrics:
    handlers_matched: {}
    levels_indexed: 0.0
    levels_nonempty: 0.0
    levels_nonempty_speed: 0.0
    levels_speed: 0.0
    levels_speed_average: 0.0
    levels_total: 0.0
    metrics_updated_at: 0.0
    objects_indexed: 0.0
    objects_speed: 0.0
    progress: 0.0
    realtime_at: 0.0
    requests_total:
      subsquid: 6.0
      evm_node: 2.0
      hasura: 27.0
    started_at: 1736934019.72
    synchronized_at: 0.0
    time_in_callbacks: {}
    time_in_matcher: {}
    time_in_requests:
      subsquid: 0.88
      evm_node: 0.15
      hasura: 0.61
    time_left: 0.0
    time_passed: 0.0
id: bcedb15d01
package: dipdup_indexer
reason: AttributeError("type object 'dict' has no attribute 'model_fields'")
date: '2025-01-15T10:40:20.585571'
content: error,stats