7.1.0 release (#879)

Co-authored-by: Igor Sereda <[email protected]>

CHANGELOG.md

@@ -4,11 +4,12 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog], and this project adheres to [Semantic Versioning].
 
-## [Unreleased]
+## [7.1.0] - 2023-10-27
 
 ### Added
 
-- cli: Added `--unsafe` and `--compose` flags to `config env` command.
+- cli: Added `--unsafe`, `--compose`, `--internal` flags to `config env` command.
+- cli: Added missing short equivalents for options in some commands.
 - cli: Relative paths to be initialized now can be passed to the `init` command as arguments.
 - tezos.tzkt.token_balances: Added new index.
 
@@ -1223,7 +1224,9 @@ This release contains no changes except for the version number.
 [semantic versioning]: https://semver.org/spec/v2.0.0.html
 
 <!-- Versions -->
-[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.0.1...HEAD
+[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.1.0...HEAD
+[7.1.0]: https://github.com/dipdup-io/dipdup/compare/7.0.2...7.1.0
+[7.0.2]: https://github.com/dipdup-io/dipdup/compare/7.0.1...7.0.2
 [7.0.1]: https://github.com/dipdup-io/dipdup/compare/7.0.0...7.0.1
 [7.0.0]: https://github.com/dipdup-io/dipdup/compare/7.0.0rc5...7.0.0
 [7.0.0rc5]: https://github.com/dipdup-io/dipdup/compare/7.0.0rc4...7.0.0rc5

docs/13.troubleshooting.md

@@ -78,7 +78,7 @@ Exceptions that occurred during callback execution are reraised as `CallbackErro
 
 ```shell [Terminal]
 # PDM
-pdm fmt
+pdm format
 pdm lint
 
 # Manually

docs/5.advanced/2.feature-flags.md

@@ -35,6 +35,7 @@ Please note that they are not currently a part of the public API and can be chan
 | env variable         | module path              | description                                              |
 | -------------------- | ------------------------ | -------------------------------------------------------- |
 | `DIPDUP_CI`          | `dipdup.env.CI`          | Running in GitHub Actions                                |
+| `DIPDUP_DEBUG`       | `dipdup.env.DEBUG`       | Enable debug logging and additional checks               |
 | `DIPDUP_DOCKER`      | `dipdup.env.DOCKER`      | Running in Docker                                        |
 | `DIPDUP_NEXT`        | `dipdup.env.NEXT`        | Enable experimental features that require schema changes |
 | `DIPDUP_REPLAY_PATH` | `dipdup.env.REPLAY_PATH` | Path to datasource replay files; used in tests           |

docs/5.advanced/3.sql.md

@@ -53,9 +53,13 @@ await ctx.execute_sql('my_hook/my_script.sql')
 
 ## Managing schema
 
-When using PostgreSQL as database engine you can use `dipdup_approve` and `dipdup_wipe` functions to manage schema state from SQL console if needed:
+When using PostgreSQL or Timescale as database engine you can use `dipdup_approve` and `dipdup_wipe` functions to manage schema state from SQL console if needed:
 
 ```sql
 SELECT dipdup_approve('public');
+
+-- WARNING: This action is irreversible! All indexed data will be lost!
 SELECT dipdup_wipe('public');
 ```
+
+Please note that `dipdup_wipe` function doesn't support preserving immune tables.

docs/9.release-notes/1.v7.1.md

@@ -0,0 +1,157 @@
+---
+title: 7.1.0
+description: DipDup 7.1 release notes
+---
+
+# Release Notes: 7.1
+
+Welcome, developers!
+
+After maybe-too-long awaited 7.0 stable release we decided to establish shorter and more predictable release cycle. So, expect new DipDup versions to arrive more frequently, but maybe be less groundbreaking. Today we're releasing DipDup 7.1 with new index kind, CLI options, and lots of small improvements. Let's dive in!
+
+## Indexing Tezos token balances
+
+Starting with 7.1 a new index kind `tezos.tzkt.token_balances` is available. It allows indexing token balances of Tezos contracts compatible with [FA1.2](https://gitlab.com/tzip/tzip/-/blob/master/proposals/tzip-7/README.md) or [FA2](https://gitlab.com/tzip/tzip/-/blob/master/proposals/tzip-12/tzip-12.md) standards. This index is similar to existing `tezos.tzkt.token_transfers`, but provides only the current balances skipping historical data. In cases when you don't need the history of changes, this index is much more efficient.
+
+```yaml [dipdup.yaml]
+spec_version: 2.0
+package: demo_token_balances
+
+contracts:
+  tzbtc_mainnet:
+    kind: tezos
+    address: KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn
+    typename: tzbtc
+
+datasources:
+  tzkt:
+    kind: tezos.tzkt
+    url: https://api.tzkt.io
+
+indexes:
+  tzbtc_holders_mainnet:
+    kind: tezos.tzkt.token_balances
+    datasource: tzkt
+    handlers:
+      - callback: on_balance_update
+        contract: tzbtc_mainnet
+```
+
+## SQL helpers to manage database schema
+
+Two new SQL functions were added to the schema: `dipdup_approve` and `dipdup_wipe`. These helpers are equivalent to `schema approve` and `schema wipe` CLI commands, but can be used directly in SQL queries.
+
+```sql
+SELECT dipdup_approve('public');
+
+-- WARNING: This action is irreversible! All indexed data will be lost!
+SELECT dipdup_wipe('public');
+```
+
+Please note that `dipdup_wipe` function doesn't support preserving immune tables.
+
+DipDup has lots of tools to work with SQL like parameterized scripts, queries, and more. Visit the [Advanced SQL](../5.advanced/3.sql.md) page to learn more.
+
+## More `config env` command options
+
+Managing environment variables could be a pain, still it's an important part of a modern 12-factor application. Try to keep your project config completely environment-agnostic to make deploy easier and ensure secrets are safe. This release brings a few new options to the `config env` command to make it easier to work with environment variables.
+
+Compose stack definitions in `deploy` package directory include environment variables from `.env` files (generated with `config env` command without arguments). New `--compose, -c` flag also allows exporting config variables in Compose format. Copy necessary strings from the output to the `environment` section of the service definition and you're good.
+
+```shell [Terminal]
+$ dipdup config env --compose
+services:
+  dipdup:
+    environment:
+      - API_KEY=${API_KEY}
+      - API_URL=${API_URL:-https://api.example.com}
+```
+
+`--internal, -i` flag allows including internal variables (with `DIPDUP_` prefix) in the command output. All of them are optional and rarely needed, but handy for debugging. See the [Feature Flags](../5.advanced/2.feature-flags.md) page for the full list of internal variables.
+
+```shell [Terminal]
+$ dipdup config env --internal
+DIPDUP_CI=
+DIPDUP_DEBUG=
+DIPDUP_DOCKER=
+DIPDUP_NEXT=
+DIPDUP_REPLAY_PATH=
+DIPDUP_TEST=
+```
+
+You could notice a new variable here, `DIPDUP_DEBUG`. Setting it to true will increase logging verbosity (equal to `logging: debug` in config) and enable some additional checks. Make sure that this flag is set if you're gathering logs for a bug report.
+
+Finally, `--unsafe` flag allows exporting environment variables from the current shell session. Missing short option is intentional to prevent accidental usage. Missing example output here is also intentional, because it's, you know, unsafe to share your secrets. Don't do that.
+
+## Improved base template
+
+One of DipDup killer features is that we scaffold everything: package metadata, types and callback stubs, Compose definitions etc. We constantly improve the base template to make it more useful. You can run `dipdup new` once, answer a few questions get a fully working project. But you also use `init` command later to refresh files from the base template in your project.
+
+```shell [Terminal]
+# First make sure that repo is clean
+git status --porcelane
+
+# Refresh only `pyproject.toml` file
+dipdup init --force pyproject.toml
+
+# Refresh everything
+dipdup init --force --base
+
+# Check what has changed
+git diff
+```
+
+This feature is a huge time saver when you maintain multiple projects and want them to be consistent. Or to modify the `replay.yaml` file and apply changes to the whole project instead of generating a new one or finding and replacing all occurrences manually.
+
+In this release we have updated included scripts to run CI checks and format code. Run `pdm run -l` to see the full list of available scripts.
+
+```shell
+$ pdm run -l
+╭────────┬───────────┬──────────────────────────────────╮
+│ Name   │ Type      │ Description                      │
+├────────┼───────────┼──────────────────────────────────┤
+│ all    │ composite │ Run all checks                   │
+│ format │ composite │ Format code with isort and black │
+│ image  │ cmd       │ Build Docker image               │
+│ lint   │ composite │ Check code with ruff and mypy    │
+╰────────┴───────────┴──────────────────────────────────╯
+```
+
+Inspect `pyproject.toml` to see how these scripts are defined. You can also add your own scripts to automate common tasks.
+
+Here's a basic example of a GitHub Actions workflow to run all checks on every push:
+
+```yaml [.github/workflows/test.yml]
+name: Test DipDup project
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+on:
+  push:
+    branches:
+      - '**'
+
+jobs:
+  test:
+    name: Test
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install PDM
+        run: pipx install pdm
+
+      - name: Set up Python
+        uses: actions/setup-python@main
+        with:
+          python-version: '3.11'
+          cache: 'pip'
+
+      - name: Install project dependencies
+        run: pdm install
+
+      - name: Run all checks
+        run: pdm all
+```
+
+That's all for today. Thanks for reading!
+
+{{ #include 9.release-notes/_footer.md }}

docs/9.release-notes/1.v7.0.md → docs/9.release-notes/2.v7.0.md

@@ -147,3 +147,5 @@ If you have any questions or issues with migration, please open a ticket on GitH
 |                   | `event`                | `tezos.tzkt.events`                |
 |                   | `head`                 | `tezos.tzkt.head`                  |
 |                   | `token_transfer`       | `tezos.tzkt.token_transfers`       |
+
+{{ #include 9.release-notes/_footer.md }}