Is it possible to use multiple Github repositories with Definitions?

[j-adamczyk]

I am looking into possibility of moving from Airflow to Dagster. One of main concerns is being able to use multiple Github repositories to push code to an orchestrator. So I want to have one Dagster instance, where multiple independent projects can submit DAGs (or graphs, I think this is a Dagster term), with their own code and dependencies.

I already know that dependency separation is much better in Dagster compared to Airflow. But I can't understand from the docs how Definitions work for multiple Github repositories. So assume that I have 5 repositories, each with its own DAG code (or multiple DAGs) and separate Poetry environment (so a separate pyproject.toml, or requirements.txt, or whatever format is required).

1. Does Dagster support this?
2. Is there any example of this? E.g. what paths to code should look like in workflows.yaml?

[alangenfeld]

The places to load Definitions are called "code locations"
https://docs.dagster.io/concepts/code-locations#code-locations

and here is an example for loading several locations from different python environments via workspace.yaml
https://docs.dagster.io/concepts/code-locations/workspace-files#loading-multiple-python-environments

[j-adamczyk]

I have seen that docs, but they only point to local repository. For example, from the second link:

load_from:
  - python_file:
      relative_path: path/to/dataengineering_spark_team.py
      location_name: dataengineering_spark_team_py_38_virtual_env
      executable_path: venvs/path/to/dataengineering_spark_team/bin/python
  - python_file:
      relative_path: path/to/team_code_location.py
      location_name: ml_team_py_36_virtual_env
      executable_path: venvs/path/to/ml_tensorflow/bin/python

So is it actually possible to set e.g. relative_path as a Git repository path? I can also use an NFS, like in Airflow, but then I do not know my DAGs in advance, so I can't specify this in one file. In Airflow there is a "DAG-puller DAG" pattern for this, which pulls the DAG code periodically. Could I do something similar and update workspace.yaml dynamically after deployment to register new code?

[alangenfeld]

So is it actually possible to set e.g. relative_path as a Git repository path?

No, dagster does not currently support targeting a git repo and managing cloning it / refreshing it for you.

Could I do something similar and update workspace.yaml dynamically after deployment to register new code?

You would have to restart any processes referring to the to the workspace.yaml after changing it but theoretically i believe something like that could work.

Many users who have requirements like this end up using containerization (docker, k8s, ecs), and having each git repository build a new container image and then update the dagster deployment to the latest image tag via CI/CD pipelines.

[konnndr]

good info)

[jdbodyfelt]

One possible solution would be to submodule in your other repos within your main dagster repo...