How to track down intermittent DagsterSubprocessError from dbt manifest reads? 🤯

[somiandras]

I'm looking for ideas/suggestions how to track down and solve a strange intermittent error I keep getting in Dagster. I suspect it's infra related, but couldn't find the source yet.

• They happen in the middle of runs between two asset materializations but at different steps -- sometimes early, sometimes late in the run.
• It's always orjson.JSONDecodeError at orjson.loads(manifest_path.read_bytes())) in dagster_dbt/dbt_manifest.py but at different locations in the json (I have two dagster projects, this can happen with both manifests)
• The dbt projects are not particularly large (about 40 models total in two projects)
• Sometimes happens with auto-materialized runs of an asset selection without any dbt assets (the error is still comes from reading the manifest)
• Complete or "from failure" re-executions always finish successfully (never got repeated errors)
• Can't really tie the error to any user code as identical runs finish without problems
• This can happen with concurrent runs in progress but also with a single active run
• I used to get one of these once in hundreds of similar runs (with only one dbt project), now it's 1 out of 5 to 10
  • They became much more frequent after upgrading to 1.7.10, but some other things were added to the repo as well
• It's a simple docker compose setup on a single box with separate user code / daemon / webserver containers -- is it the user code container becoming momentarily unavailable?
• I have DAGSTER_DBT_PARSE_PROJECT_ON_LOAD set to True, manifest.json is created when Dagster loads the assets after the user code container is started

Here's an example error (it's always this, only the column number changes).

dagster._core.errors.DagsterSubprocessError: During multiprocess execution errors occurred in child processes:
In process 1816: orjson.JSONDecodeError: unexpected end of data: line 1 column 459801 (char 459800)

Stack Trace:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/child_process_executor.py", line 78, in _execute_command_in_child_process
    for step_event in command.execute():
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/executor/multiprocess.py", line 88, in execute
    execution_plan = create_execution_plan(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/execution/api.py", line 710, in create_execution_plan
    job_def = job.get_definition()
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 265, in get_definition
    return check.not_none(self.get_repository_definition()).get_maybe_subset_job_def(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 259, in get_repository_definition
    return self.repository.get_definition()
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 119, in get_definition
    return repository_def_from_pointer(self.pointer, self.repository_load_data)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 743, in repository_def_from_pointer
    target = def_from_pointer(pointer)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/reconstruct.py", line 639, in def_from_pointer
    target = pointer.load_target()
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 174, in load_target
    module = load_python_file(self.python_file, self.working_directory)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 83, in load_python_file
    return import_module_from_path(module_name, python_file)
  File "/usr/local/lib/python3.10/site-packages/dagster/_seven/__init__.py", line 47, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/dagster/app/collective_dagster/main.py", line 121, in <module>
    assets=load_assets_from_package_module(assets),
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/load_assets_from_modules.py", line 270, in load_assets_from_package_module
    ) = assets_from_package_module(package_module)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/load_assets_from_modules.py", line 218, in assets_from_package_module
    return assets_from_modules(
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/load_assets_from_modules.py", line 56, in assets_from_modules
    for module in modules:
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/load_assets_from_modules.py", line 337, in find_modules_in_package
    yield from find_modules_in_package(submodule)
  File "/usr/local/lib/python3.10/site-packages/dagster/_core/definitions/load_assets_from_modules.py", line 335, in find_modules_in_package
    submodule = import_module(modname)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/dagster/app/collective_dagster/assets/analytics/dbt_assets.py", line 36, in <module>
    @dbt_assets(
  File "/usr/local/lib/python3.10/site-packages/dagster_dbt/asset_decorator.py", line 335, in dbt_assets
    manifest = validate_manifest(manifest)
  File "/usr/local/lib/python3.10/site-packages/dagster_dbt/dbt_manifest.py", line 37, in validate_manifest
    manifest = read_manifest_path(manifest.resolve())
  File "/usr/local/lib/python3.10/site-packages/dagster_dbt/dbt_manifest.py", line 26, in read_manifest_path
    return cast(Mapping[str, Any], orjson.loads(manifest_path.read_bytes()))

[rexledesma]

From https://docs.dagster.io/integrations/dbt/reference#loading-dbt-models-from-a-dbt-project:

When deploying your Dagster project to production, we recommend generating the manifest at build time to avoid the overhead of recompiling your dbt project every time your Dagster code is executed. A manifest.json should be precompiled and included in the Python package for your Dagster code.

Creating the manifest at runtime in production is known to cause issues and is not recommended. This is often caused by the following code and should be avoided.

You seem to be creating your manifest at runtime in production, since you have DAGSTER_DBT_PARSE_PROJECT_ON_LOAD set to True. Our recommendation would be to pre-compile your manifest so that these issues go away.

[somiandras]

Thanks @rexledesma! I finally found that this was our issue indeed, we accidentally had that env var set to true in production. Running dbt parse during build (and making sure we don't regenerate the manifest during runtime) solved the issue. It's worth noting that we never saw this issue until we added the second dbt project to Dagster.

[XBeg9]

We are also getting this but on local env. Do you have any ideas if it's possible to fix the issue? Very easily reproducible if you create graph_asset with dynamicout (50-100)... every other is going to fail due to this issue. @rexledesma

[rexledesma]

#22661 (comment)

Don't create your dbt manifest at run time. Build your manifest before running dagster dev.