Skip to content

Commit

Permalink
Cache JobDefinition.get_job_index (#23722)
Browse files Browse the repository at this point in the history
Summary:
This can take seconds on large jobs, and we call it at twice when
loading a new code location when using defer_snapshots=True - once when
generating the IDs for the JobRefs,and then again when loading the
actual jobs.

Test Plan: BK, verify when running an agent locally that this is called
only once per job when loading a new code locatoin

## Summary & Motivation

## How I Tested These Changes
  • Loading branch information
gibsondan authored Aug 27, 2024
1 parent a928964 commit 90fb55e
Showing 1 changed file with 2 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@
from dagster._core.types.dagster_type import DagsterType
from dagster._core.utils import str_format_set
from dagster._utils import IHasInternalInit
from dagster._utils.cached_method import cached_method
from dagster._utils.merger import merge_dicts

from .asset_layer import AssetLayer
Expand Down Expand Up @@ -979,6 +980,7 @@ def get_config_schema_snapshot(self) -> "ConfigSchemaSnapshot":
def get_job_snapshot(self) -> "JobSnapshot":
return self.get_job_index().job_snapshot

@cached_method
def get_job_index(self) -> "JobIndex":
from dagster._core.remote_representation import JobIndex
from dagster._core.snap import JobSnapshot
Expand Down

0 comments on commit 90fb55e

Please sign in to comment.