Merge branch 'next' into feat/pydantic-v2

dipdup-io · Oct 17, 2023 · b96cc33 · b96cc33
2 parents bc2d388 + f1ad9cb
commit b96cc33
Show file tree

Hide file tree

Showing 27 changed files with 296 additions and 198 deletions.
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -17,7 +17,7 @@ on:
 
 env:
   FRONTEND_BRANCH: master
-  GH_TOKEN: ${{ secrets.API_TOKEN_EXT }}
+  GITHUB_TOKEN: ${{ secrets.DOCS_GITHUB_TOKEN }}
 
 jobs:
   docs:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,23 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog], and this project adheres to [Semantic Versioning].
 
+## [7.0.2] - 2023-10-10
+
+### Added
+
+- database: Added `dipdup_wipe` and `dipdup_approve` SQL functions to the schema.
+
+### Fixed
+
+- cli: Fixed `schema wipe` command for SQLite databases.
+- tezos.tzkt: Fixed regression in `get_transactions` method pagination.
+
+## [6.5.13] - 2023-10-10
+
+### Fixed
+
+- tzkt: Fixed regression in `get_transactions` method pagination.
+
 ## [7.0.1] - 2023-09-30
 
 ### Added

diff --git a/docs/5.advanced/1.reindexing.md b/docs/5.advanced/1.reindexing.md
@@ -7,13 +7,13 @@ description: "In some cases, DipDup can't proceed with indexing without a full w
 
 In some cases, DipDup can't proceed with indexing without a full wipe. Several reasons trigger reindexing:
 
-| reason            | description                                                                                                               |
-| ----------------- | ------------------------------------------------------------------------------------------------------------------------- |
-| `manual`          | Reindexing triggered manually from callback with `ctx.reindex`.                                                           |
-| `migration`       | Applied migration requires reindexing. Check release notes before switching between major DipDup versions to be prepared. |
-| `rollback`        | Reorg message received from TzKT can not be processed.                                                                    |
-| `config_modified` | One of the index configs has been modified.                                                                               |
-| `schema_modified` | Database schema has been modified. Try to avoid manual schema modifications in favor of [sql](../5.advanced/6.sql.md).   |
+| reason            | description                                                                                                                    |
+| ----------------- | ------------------------------------------------------------------------------------------------------------------------------ |
+| `manual`          | Reindexing triggered manually from callback with `ctx.reindex`.                                                                |
+| `migration`       | Applied migration requires reindexing. Check release notes before switching between major DipDup versions to be prepared.      |
+| `rollback`        | Reorg message received from datasource and can not be processed.                                                               |
+| `config_modified` | One of the index configs has been modified.                                                                                    |
+| `schema_modified` | Database schema has been modified. Try to avoid manual schema modifications in favor of [SQL scripts](../5.advanced/3.sql.md). |
 
 It is possible to configure desirable action on reindexing triggered by a specific reason.
 

diff --git a/docs/5.advanced/3.internal-tables.md → docs/5.advanced/3.sql.md b/docs/5.advanced/3.internal-tables.md → docs/5.advanced/3.sql.md
@@ -1,11 +1,13 @@
 ---
-title: "Internal tables"
-description: "This page describes the internal tables used by DipDup. They are created automatically and are not intended to be modified by the user. However, they can be useful for external monitoring and debugging."
+title: "Advanced SQL"
+description: "Put your *.sql scripts to dipdup_indexer/sql. You can run these scripts from any callback with ctx.execute_sql('name'). If name is a directory, each script it contains will be executed."
 ---
 
-# Internal tables
+# Advanced SQL
 
-This page describes the internal tables used by DipDup. They are created automatically and are not intended to be modified by the user. However, they can be useful for external monitoring and debugging.
+## Internal tables
+
+Several tables haing `dipdup_` prefix are created by DipDup automatically and are not intended to be modified by the user. However, they can be useful for external monitoring and debugging.
 
 | table                      | description                                                                                                                               |
 |:-------------------------- |:----------------------------------------------------------------------------------------------------------------------------------------- |
@@ -15,8 +17,8 @@ This page describes the internal tables used by DipDup. They are created automat
 | `dipdup_contract`          | Info about contracts used by all indexes, including ones added in runtime.                                                                |
 | `dipdup_model_update`      | Service table to store model diffs for database rollback. Configured by `advanced.rollback_depth`                                         |
 | `dipdup_meta`              | Arbitrary key-value storage for DipDup internal use. Survives reindexing. You can use it too, but don't touch keys with `dipdup_` prefix. |
-| `dipdup_contract_metadata` | See Metadata interface page                                                                                                               |
-| `dipdup_token_metadata`    | See Metadata interface page                                                                                                               |
+| `dipdup_contract_metadata` | See [Metadata interface](/docs/advanced/metadata-interface)                                                                               |
+| `dipdup_token_metadata`    | See [Metadata interface](/docs/advanced/metadata-interface)                                                                               |
 
 See [`dipdup.models` module](https://github.com/dipdup-io/dipdup/blob/next/src/dipdup/models/__init__.py) for exact table definitions.
 
@@ -32,3 +34,28 @@ SELECT name, status FROM dipdup_index;
 -- Get last reindex time
 SELECT created_at FROM dipdup_schema WHERE name = 'public';
 ```
+
+## Scripts
+
+Put your `*.sql` scripts to `{{ project.package }}/sql`. You can run these scripts from any callback with `ctx.execute_sql('name')`. If `name` is a directory, each script it contains will be executed.
+
+Scripts are executed without being wrapped with SQL transactions. It's generally a good idea to avoid touching table data in scripts.
+
+By default, an empty `sql/<hook_name>` directory is generated for every hook in config during init. Remove `ctx.execute_sql` call from hook callback to avoid executing them.
+
+```python
+# Execute all scripts in sql/my_hook directory
+await ctx.execute_sql('my_hook')
+
+# Execute a single script
+await ctx.execute_sql('my_hook/my_script.sql')
+```
+
+## Managing schema
+
+When using PostgreSQL as database engine you can use `dipdup_approve` and `dipdup_wipe` functions to manage schema state from SQL console if needed:
+
+```sql
+SELECT dipdup_approve('public');
+SELECT dipdup_wipe('public');
+```
diff --git a/docs/5.advanced/6.sql.md b/docs/5.advanced/6.sql.md
-Original file line number
+Diff line change
@@ Expand Up / @@ -17,7 +17,7 @@ on: @@
     env:
       FRONTEND_BRANCH: master
-      GH_TOKEN: ${{ secrets.API_TOKEN_EXT }}
+      GITHUB_TOKEN: ${{ secrets.DOCS_GITHUB_TOKEN }}
     jobs:
       docs:
@@ Expand Down @@