Code / repository directory monitoring for triggering job subsets

[SgtGilly]

Hello. I have a task for my project and was wondering if there are capbilities within Dagster that I can leverage to accomplish this task.

I have my dagster project as a subdirectory within my overall projects repository. I want to know if there is an effective way for Dagster to monitor my projects repo for changes to any files and kick off a specific job or job subset if that changed file pertains to that job. For example, say I update the source code of a python utility file and I know that that file and its functionality is used by the second Op (Op_2) of my dagster job (Job_2). Could I use maybe a sensor to monitor the repo, see this update, and run the subset of Job_2 starting at Op_2? Can I specify within a RunRequest to run a subset of a job?

Is there also the possibility that I can treat my projects repository as an asset and then leverage an observable source asset so that I can track when the repo changes and then utilize a multi_asset_sensor to kick of the jobs pertaining to the changed file/s?

Sorry, very long winded question with alot of different parts.

[sryza]

Hi @SgtGilly the most relevant functionality in Dagster is that you can set code_versions on ops or software-defined assets. However, Dagster doesn't have the built-in functionality for automatically determining code versions based on the content of Python files, so it would be up to you to implement that.

E.g. you could do something like:

def get_file_hash(path: str) -> str:
    """Some code that loads the file contents as a string and computes its hash"""


@asset(code_version=get_file_hash(__file__))
def asset1():
    ...

Once you've assigned a code version to an asset, the UI will show you which assets have changed code versions since their latest materailization. And you can select the "Materialize changed and missing" option from the "Materialize" dropdown, to only materializes assets that have changed.

And here's an issue where we're tracking the capability to automatically materialize an asset when its code version changes: #15686.