DBT json log file

[shanebusenitz]

I'm running several Assets with:

DbtCliResource(
            project_dir= DBT_PROJECT_PATH,
            profiles_dir= DBT_PROFILES,
            global_config_flags=["--log-format-file", "json"]
    )

and then:

@dbt_assets(
    manifest=MANIFEST,
    dagster_dbt_translator=CustomDagsterDbtTranslator()
)
def my_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource):
    dbt_build_invocation = dbt.cli(["build"], context=context)

    yield from dbt_build_invocation.stream().fetch_row_counts().fetch_column_metadata()

I'm using the S3ComputeLogManager to log to S3 however that appears to just log the stderr and stdout. How would I get the JSON logs/dbt.log file to S3 as well?

[garethbrickman]

I think you may be able to use DbtCliInvocation. get_artifact to fetch the dbt.log. You could then either log its contents into the run logs (like in this example), or use a separate process to upload it entirely to S3.