Skip to content

Commit

Permalink
fix backfill and DA backcompat with single implicit asset jobs (#23808)
Browse files Browse the repository at this point in the history
## Summary & Motivation

Here's the situation that's broken:
- The user code is on a Dagster version where there's an implicit asset
job per `PartitionsDefinition`
- The host code is on a Dagster version where there's a single implicit
asset job
- The user code has assets with two different `PartitionsDefinition`s,
meaning it has `__ASSET_JOB_0` and `__ASSET_JOB_1`
- DA wants to submit a run to materialize one of those
`PartitionsDefinition`s
- It needs to figure out what job to submit the run for

Prior to this PR, it expects there to be a job just named `__ASSET_JOB`,
but because there isn't, this gets raised:

```python
Traceback (most recent call last):
  File "/Users/sryza/dagster/python_modules/dagster/dagster/_core/execution/asset_backfill.py", line 754, in _submit_runs_and_update_backfill_in_chunks
    submit_asset_run(
  File "/Users/sryza/dagster/python_modules/dagster/dagster/_core/execution/submit_asset_runs.py", line 260, in submit_asset_run
    run_to_submit = _create_asset_run(
  File "/Users/sryza/dagster/python_modules/dagster/dagster/_core/execution/submit_asset_runs.py", line 135, in _create_asset_run
    execution_data = _get_job_execution_data_from_run_request(
  File "/Users/sryza/dagster/python_modules/dagster/dagster/_core/execution/submit_asset_runs.py", line 67, in _get_job_execution_data_from_run_request
    check.failed(
  File "/Users/sryza/dagster/python_modules/dagster/dagster/_check/functions.py", line 1642, in failed
    raise CheckError(f"Failure condition: {desc}")
dagster._check.functions.CheckError: Failure condition: Could not find an implicit asset job for the given assets: [AssetKey(['upstream_daily_partitioned_asset'])]
```

Now, it will look for a job whose name _starts with_ `__ASSET_JOB` and
targets all the partitions.

## How I Tested These Changes

Locally:
- Ran a code server on 1.8.0 (no single implicit asset job)
- Ran a daemon on master (without this fix applied)
- Kicked off a backfill
- Observed the above error
- Killed the daemon
- Applied this fix
- Restarted the daemon
- Kicked off a backfill
- Observed the backfill complete successfully

## Changelog

NOCHANGELOG

(cherry picked from commit e8f3f1d)
  • Loading branch information
sryza authored and jmsanders committed Aug 22, 2024
1 parent de61649 commit 53bb814
Showing 1 changed file with 3 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,9 @@ def _get_implicit_job_name_for_assets(
for asset_key in asset_keys[1:]:
job_names &= set(asset_graph.get_materialization_job_names(asset_key))

return next((job_name for job_name in job_names if job_name == IMPLICIT_ASSET_JOB_NAME), None)
return next(
(job_name for job_name in job_names if job_name.startswith(IMPLICIT_ASSET_JOB_NAME)), None
)


def _get_execution_plan_asset_keys(
Expand Down

0 comments on commit 53bb814

Please sign in to comment.