Add dbt models and tests to the Execution Topology view (the same as currently shown in the Gantt view)

[anna-geller]

Extend the dbtRun, dbtBuild and dbtCll tasks with 2 new properties parse and outputRunResults:

• allow rendering dbt dag at runtime in the Execution topology view if parse (new property) is set to true
• add another boolean flag outputRunResults indicating whether to output dbt models with custom metadata like duration, which will allow later to expose those as events — ideally, an ION file with one row per dbt model or test (data can be extracted from run_results.json generated by dbt after dbt Build/Run completion)

Name	Type	Status	Duration	Completed At
model.dwh.stg_customers	Model	success	0.022	2024-03-25T08:46:21.888471Z
model.dwh.stg_orders	Model	success	0.022	2024-03-25T08:46:21.888663Z
model.dwh.stg_locations	Model	success	0.023	2024-03-25T08:46:21.889137Z
model.dwh.stg_order_items	Model	success	0.023	2024-03-25T08:46:21.889313Z
model.dwh.stg_products	Model	success	0.023	2024-03-25T08:46:21.890272Z
model.dwh.stg_supplies	Model	success	0.023	2024-03-25T08:46:21.890450Z
test.dwh.not_null_stg_customers_customer_id	Test	success	0.042	2024-03-25T08:46:21.923868Z
test.dwh.unique_stg_customers_customer_id	Test	success	0.044	2024-03-25T08:46:21.924386Z
test.dwh.not_null_stg_orders_order_id	Test	success	0.044	2024-03-25T08:46:21.935854Z
test.dwh.unique_stg_orders_order_id	Test	success	0.046	2024-03-25T08:46:21.938104Z

Context

dbt models and tests are difficult to read and inspect in the Gantt view:

We would like to see those in the Execution Topology view:

Simple reproducer to inspect/iterate on the design (you can run it directly with no setup):

id: dbtGitDockerDuckDB
namespace: dev

tasks:
  - id: dbt
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
      - id: cloneRepository
        type: io.kestra.plugin.git.Clone
        url: https://github.com/kestra-io/dbt-demo
        branch: main

      - id: dbt-build
        type: io.kestra.plugin.dbt.cli.Build
        debug: true
        dockerOptions:
          image: ghcr.io/kestra-io/dbt-duckdb:latest
        runner: DOCKER
        dbtPath: /usr/local/bin/dbt
        inputFiles:
          .profile/profiles.yml: |
            jaffle_shop:
              outputs:
                dev:
                  type: duckdb
                  path: ':memory:'
                  extensions:
                    - parquet
              target: dev

[aballiet]

Yes could be nice, but I think for project with quite a lot of models (800+ in our case) would start to be overwhelming.

Would be very useful to focus on warnings and errors for models & tests.

Displaying logs for the successful models built is enough IMO.

Dbt cloud displays metrics like this and is super useful for analysts etc :

Adding quick video showing our current needs in terms of UI : https://www.loom.com/share/0556984bd9564b328ea4cf413954a514

[anna-geller]

Based on this feedback, this issue is deprioritized in favor of this more generic log-level representation kestra-io/kestra#2045