Merge branch 'next' into dev/substrate-events

.github/workflows/docs.yml

@@ -8,6 +8,7 @@ on:
   push:
     tags:
       - '8.*.*'
+      - 'docs'
       - 'docs-next'
     branches:
       - '**'

CHANGELOG.md

@@ -6,7 +6,7 @@ The format is based on [Keep a Changelog], and this project adheres to [Semantic
 
 Releases prior to 7.0 has been removed from this file to declutter search results; see the [archived copy](https://github.com/dipdup-io/dipdup/blob/8.0.0b5/CHANGELOG.md) for the full list.
 
-## [Unreleased]
+## [8.1.3] - 2024-12-20
 
 ### Added
 
@@ -23,7 +23,11 @@ Releases prior to 7.0 has been removed from this file to declutter search result
 - cli: Fixed `--template` option being ignored when `--quiet` flag is set.
 - config: Fixed setting default loglevels when `logging` is a dict.
 - config: Fixed parsing config files after updating to pydantic 2.10.3.
+- config: Fixed starknet index validation error.
 - metrics: Fixed indexed objects counter.
+- starknet: Added support for struct and array types, as well as u256 and ByteArray handlers.
+- starknet: Fixed event payload parsing (account for keys field).
+- starknet: Fixed missing class property in node datasource.
 
 ## [8.1.2] - 2024-12-10
 
@@ -565,9 +569,10 @@ Releases prior to 7.0 has been removed from this file to declutter search result
 [semantic versioning]: https://semver.org/spec/v2.0.0.html
 
 <!-- Versions -->
-[Unreleased]: https://github.com/dipdup-io/dipdup/compare/8.1.2...HEAD
-[8.1.2]: https://github.com/dipdup-io/dipdup/compare/8.0.1...8.1.2
-[8.1.1]: https://github.com/dipdup-io/dipdup/compare/8.0.0...8.1.1
+[Unreleased]: https://github.com/dipdup-io/dipdup/compare/8.1.3...HEAD
+[8.1.3]: https://github.com/dipdup-io/dipdup/compare/8.1.2...8.1.3
+[8.1.2]: https://github.com/dipdup-io/dipdup/compare/8.1.1...8.1.2
+[8.1.1]: https://github.com/dipdup-io/dipdup/compare/8.1.0...8.1.1
 [8.1.0]: https://github.com/dipdup-io/dipdup/compare/8.0.0...8.1.0
 [8.0.0]: https://github.com/dipdup-io/dipdup/compare/8.0.0b5...8.0.0
 [8.0.0b5]: https://github.com/dipdup-io/dipdup/compare/8.0.0b4...8.0.0b5

docs/7.references/4.models.md

@@ -399,6 +399,49 @@ description: "Models reference"
 
 ## Substrate
 
+<dl class="py class">
+
+## dipdup.models.substrate.SubstrateEvent
+
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.models.substrate.</span></span><span class="sig-name descname"><span class="pre">SubstrateEvent</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">data</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><a class="reference internal" href="#dipdupmodelssubstratesubstrateeventdata" title="dipdup.models.substrate.SubstrateEventData" target="_self"><span class="pre">dipdup.models.substrate.SubstrateEventData</span></a></span></em>, <em class="sig-param"><span class="n"><span class="pre">runtime</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.runtimes.SubstrateRuntime</span></span></em><span class="sig-paren">)</span></dt>
+<dd><dl class="field-list simple">
+<dt class="field-odd" style="color: var(--txt-primary);">Parameters<span class="colon">:</span></dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>data</strong> (<a class="reference internal" href="#dipdupmodelssubstratesubstrateeventdata" title="dipdup.models.substrate.SubstrateEventData" target="_self"><em>SubstrateEventData</em></a>)</p></li>
+<li><p><strong>runtime</strong> (<em>SubstrateRuntime</em>)</p></li>
+</ul>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="py class">
+
+## dipdup.models.substrate.SubstrateEventData
+
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.models.substrate.</span></span><span class="sig-name descname"><span class="pre">SubstrateEventData</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="o"><span class="pre">*</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">name</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">str</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">index</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">extrinsic_index</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">int</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">call_address</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">args</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">list</span><span class="p"><span class="pre">[</span></span><span class="pre">dipdup.fields.Any</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">decoded_args</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dict</span><span class="p"><span class="pre">[</span></span><span class="pre">str</span><span class="p"><span class="pre">,</span></span><span class="w"> </span><span class="pre">dipdup.fields.Any</span><span class="p"><span class="pre">]</span></span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span><span class="w"> </span><span class="o"><span class="pre">=</span></span><span class="w"> </span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">header</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.models.substrate._BlockHeader</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">header_extra</span></span><span class="p"><span class="pre">:</span></span><span class="w"> </span><span class="n"><span class="pre">dipdup.models.substrate._BlockHeaderExtra</span><span class="w"> </span><span class="p"><span class="pre">|</span></span><span class="w"> </span><span class="pre">None</span></span></em><span class="sig-paren">)</span></dt>
+<dd><dl class="field-list simple">
+<dt class="field-odd" style="color: var(--txt-primary);">Parameters<span class="colon">:</span></dt>
+<dd class="field-odd"><ul class="simple">
+<li><p><strong>name</strong> (<em>str</em>)</p></li>
+<li><p><strong>index</strong> (<em>int</em>)</p></li>
+<li><p><strong>extrinsic_index</strong> (<em>int</em>)</p></li>
+<li><p><strong>call_address</strong> (<em>list</em><em>[</em><em>str</em><em>] </em><em>| </em><em>None</em>)</p></li>
+<li><p><strong>args</strong> (<em>list</em><em>[</em><em>Any</em><em>] </em><em>| </em><em>None</em>)</p></li>
+<li><p><strong>decoded_args</strong> (<em>dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>] </em><em>| </em><em>None</em>)</p></li>
+<li><p><strong>header</strong> (<em>_BlockHeader</em>)</p></li>
+<li><p><strong>header_extra</strong> (<em>_BlockHeaderExtra</em><em> | </em><em>None</em>)</p></li>
+</ul>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="py class">
+
+## dipdup.models.substrate.SubstrateHeadBlockData
+
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.models.substrate.</span></span><span class="sig-name descname"><span class="pre">SubstrateHeadBlockData</span></span></dt>
+<dd></dd></dl>
+
 </section>
 
 

docs/8.examples/_demos_table.md

@@ -1,22 +1,22 @@
 <!-- markdownlint-disable first-line-h1 -->
 | name | network | description | source |
 |-|-|-|-|
-| demo_blank |  | Empty config for a fresh start | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_blank) |
-| demo_substrate_events |  | Substrate balance transfers | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_substrate_events) |
-| demo_evm_events | EVM | ERC-20 token transfers (from event logs) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_evm_events) |
-| demo_evm_transactions | EVM | ERC-20 token transfers (from transactions) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_evm_transactions) |
-| demo_evm_uniswap | EVM | Uniswap V3 pools, positions, etc. (advanced, uses TimescaleDB) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_evm_uniswap) |
-| demo_starknet_events | Starknet | ERC-20 token transfers (from events) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_starknet_events) |
-| demo_tezos_auction | Tezos | NFT marketplace (TzColors) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_auction) |
-| demo_tezos_dao | Tezos | DAO registry (Homebase DAO) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_dao) |
-| demo_tezos_dex | Tezos | DEX balances and liquidity (Quipuswap) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_dex) |
-| demo_tezos_domains | Tezos | Domain name service (Tezos Domains) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_domains) |
-| demo_tezos_etherlink | Tezos | Etherlink smart rollup transactions | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_etherlink) |
-| demo_tezos_events | Tezos | Processing contract events | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_events) |
-| demo_tezos_factories | Tezos | Example of spawning indexes in runtime | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_factories) |
-| demo_tezos_head | Tezos | Processing head block metadata (realtime only) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_head) |
-| demo_tezos_nft_marketplace | Tezos | NFT marketplace (hic at nunc) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_nft_marketplace) |
-| demo_tezos_raw | Tezos | Process raw operations without filtering and typed payloads | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_raw) |
-| demo_tezos_token | Tezos | FA1.2 token contract operations | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_token) |
-| demo_tezos_token_balances | Tezos | FA1.2 token balances | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_token_balances) |
-| demo_tezos_token_transfers | Tezos | FA1.2 token transfers | [link](https://github.com/dipdup-io/dipdup/tree/8.1.2/src/demo_tezos_token_transfers) |
+| demo_blank |  | Empty config for a fresh start | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_blank) |
+| demo_substrate_events |  | Substrate balance transfers | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_substrate_events) |
+| demo_evm_events | EVM | ERC-20 token transfers (from event logs) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_evm_events) |
+| demo_evm_transactions | EVM | ERC-20 token transfers (from transactions) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_evm_transactions) |
+| demo_evm_uniswap | EVM | Uniswap V3 pools, positions, etc. (advanced, uses TimescaleDB) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_evm_uniswap) |
+| demo_starknet_events | Starknet | ERC-20 token transfers (from events) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_starknet_events) |
+| demo_tezos_auction | Tezos | NFT marketplace (TzColors) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_auction) |
+| demo_tezos_dao | Tezos | DAO registry (Homebase DAO) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_dao) |
+| demo_tezos_dex | Tezos | DEX balances and liquidity (Quipuswap) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_dex) |
+| demo_tezos_domains | Tezos | Domain name service (Tezos Domains) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_domains) |
+| demo_tezos_etherlink | Tezos | Etherlink smart rollup transactions | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_etherlink) |
+| demo_tezos_events | Tezos | Processing contract events | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_events) |
+| demo_tezos_factories | Tezos | Example of spawning indexes in runtime | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_factories) |
+| demo_tezos_head | Tezos | Processing head block metadata (realtime only) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_head) |
+| demo_tezos_nft_marketplace | Tezos | NFT marketplace (hic at nunc) | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_nft_marketplace) |
+| demo_tezos_raw | Tezos | Process raw operations without filtering and typed payloads | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_raw) |
+| demo_tezos_token | Tezos | FA1.2 token contract operations | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_token) |
+| demo_tezos_token_balances | Tezos | FA1.2 token balances | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_token_balances) |
+| demo_tezos_token_transfers | Tezos | FA1.2 token transfers | [link](https://github.com/dipdup-io/dipdup/tree/8.1.3/src/demo_tezos_token_transfers) |

docs/9.release-notes/_8.0_changelog.md

@@ -18,10 +18,6 @@
 - starknet.events: Added `starknet.events` index kind to process Starknet events.
 - starknet.node: Added Starknet node datasource for last mile indexing.
 - starknet.subsquid: Added `starknet.subsquid` datasource to fetch historical data from Subsquid Archives.
-- substrate.events: Added `subtrate.events` index kind to process Substrate events.
-- substrate.node: Added `subtrate.node` datasource to receive data from Substrate node.
-- substrate.subscan: Added `substrate.subscan` datasource to fetch ABIs from Subscan.
-- substrate.subsquid: Added `substrate.subsquid` datasource to fetch historical data from Squid Network.
 - tezos.operations: Added `sr_cement` operation type to process Smart Rollup Cemented Commitments.
 
 ### Fixed
@@ -41,6 +37,7 @@
 - config: Fixed parsing config files after updating to pydantic 2.10.3.
 - config: Fixed setting default loglevels when `logging` is a dict.
 - config: Fixed setting logging levels according to the config.
+- config: Fixed starknet index validation error.
 - database: Fixed concurrency issue when using `get_or_create` method.
 - evm.events: Fixed matching logs when filtering by topic0.
 - evm.events: Improve fetching event batches from node.
@@ -54,6 +51,9 @@
 - performance: Add index name to fetcher and realtime queues.
 - performance: Fixed estimation indexing speed in levels per second.
 - starknet.events: Fixed filtering events by key.
+- starknet: Added support for struct and array types, as well as u256 and ByteArray handlers.
+- starknet: Fixed event payload parsing (account for keys field).
+- starknet: Fixed missing class property in node datasource.
 - subsquid: Fixed missing entry in `dipdup_head` internal table.
 - tezos.big_maps: Fixed logging status message in `skip_history` mode.
 - tezos.big_maps: Respect order of handlers in `skip_history` mode.

docs/9.release-notes/_8.1_changelog.md

@@ -8,6 +8,10 @@
 - cli: Added interactive mode for `new` command.
 - database: Support database migrations using [`aerich`](https://github.com/tortoise/aerich).
 - hasura: Added `hide` and `hide_internal` config options to make specified tables/views private.
+- substrate.events: Added `subtrate.events` index kind to process Substrate events.
+- substrate.node: Added `subtrate.node` datasource to receive data from Substrate node.
+- substrate.subscan: Added `substrate.subscan` datasource to fetch ABIs from Subscan.
+- substrate.subsquid: Added `substrate.subsquid` datasource to fetch historical data from Squid Network.
 
 ### Fixed