How you organize your Dagster code

[yuhan]

Hi everyone, I’m looking to learn more about how you organize your complex Dagster projects and the troubles you are facing when organizing the code. Besides, when structuring your projects, are there any goals you’re trying to optimize?

We’re working on a more comprehensive recommendation for Dagster project structure. In the past, we’ve come up with the experimental Create a New Project which comes with a CLI command that scaffolds a project skeleton, but we’ve heard great feedback about it not being well suited for larger complex and prod-scale projects. So I’d love to learn from you about any good practices when you design and structure your data projects.

[stkbailey]

At Whatnot, we have tried three approaches (in order):

1. Multiple code locations, with one Github repo per code location
2. Multiple code locations, within one Github repo
3. One code location, within one Github repo

The single code location, with a single repo, is by far the most intuitive for new (and old, honestly) users. The conclusion I came to was that you should use multiple code locations only when Python requirements for jobs begin to diverge. In most other cases, repo level control is practically sufficient.

Within our one repo, we have our Python module organized like so:

my_package/
  common/
    __init__.py
    resources/
      snowflake.py
      dbt_cloud.py
      ...
    utils.py
  repo_a/
    __init__.py
    code/
      __init__.py
      something.py
    jobs.py
    ops.py
    sensors.py
  repo_b/
    __init__.py
    jobs.py
    ops.py
  repo_c/
    __init__.py
    jobs.py
    ops.py

What is nice about this structure is that all of the repositories exist as submodules within the project, so they can easily be added or removed without impacting any other repositories. System-level utilities such as resources, hooks, sensors, and default job configs -- we have a @whatnot_job decorator that adds the same resources, K8s tags, etc. to every job -- can easily be imported into every repo.

A typical repo_a/__init__.py file will look like:

from my_package.common import whatnot_job
from dagster import repository
from .ops import op_1, op_2


@whatnot_job
def my_awesome_job_with_default_resources_and_stuff():
    op_1()
    op_2()

@repository
def repo_a():
    return [my_awesome_job_with_default_resources_and_stuff]

After iterating a bunch, I'm very happy with this approach, and I can see it scaling to a large number of repos nicely. The one drawback with this approach is that you have "all your eggs in one basket", so to speak, so if someone pushes a change to prod that stops the code location from compiling, you have a big problem. But you can mitigate that risk pretty well with good testing and dev processes.

[creativedutchmen]

We started off with different (dagster) repos for different responsibilities (ingestion, transformation, ... ) but with the new asset-based way of doing things that's not great, because it means you can't kick off an airbyte sync and have dbt transform only the updated tables. Moving dbt and airbyte together is something I might do soon.

Some of our ingestion pipelines are spark jobs that run in databricks, and having everything in one repo also means I have to install all dependencies in databricks - including the secrets these dependencies need. This quickly gets painful, so I think this should remain as its own repo?

Then we come to ML models, here the dependencies are really all over the place, and I can't imagine not splitting it up in separate git repos.

To me, if we have to split some things up anyway, why not make this one of the "gold" options? I'd really prefer to have a separate repo (dagster + git) for every project than having several monorepos with no clear responsibility other than "their dependencies are compatible. for now"

[stkbailey]

One consideration is that the new branch_deployments are tied to a single repo in Dagster Cloud, so I think they are going to have to flesh out this pattern as well. I was able to get the multiple code locations, one github repo working pretty well, actually, and it looked like this:

.github
code/
    location_1/
        Dockerfile
        my_pytorch_workflows/
             __init.py, etc.
    location_2/
         Dockerfile
         my_databricks_workflows/
    location_3/
         Dockerfile
         my_saas_workflows/
locations.yaml
docker-compose.yml
etc.

Then in your top-level GH Action, you can diff against the subrepos, and only rebuild when changes are made. It works pretty well -- the only issue for me was when non-engineers got in the repo, it wasn't 100% intuitive, and for us, 95% of our cases involve vanilla dependencies. Also, we read all secrets at runtime using a fetch_aws_secrets_manager function, which works great in this setup.

[geoHeil]

How is the: whatnot_job defined?

What about using conda envs for various repositories inside a single git monorepo?

[yuhan]

Totally agree with the ideas here. IMO, in most cases, one single Dagster repository should get you pretty far, and I wouldn't recommend over abstract things too early - if it's for organizational purpose, multiple AssetGroup within a single repository should be sufficient to start with.

But multiple Dagster repositories can be pretty helpful when you want to keep conflicting Python dependencies separate, or execute your data pipelines in different Python environments (e.g. one in py3.6 and the other in py3.10). To do so, each Dagster repository can keep their own package requirements (e.g., setup.py) and deployment specs (e.g., Dockerfile) -- Dagster provides different ways to load multiple repos in one workspace (docs); Dagster Cloud allows multi code locations in one deployment (docs), then you can add code location per repository to your deployment as your project grows.

https://github.com/yuhan/demo-my-dagster-project-with-multi-repos
I wrote a starter git repo to show how I'd structure a multi-repo Dagster project with the newly released dagster project CLI. You can find the instructions in README, and you can also find each step in the commit history. It's pretty naive at the moment and I plan to improve the scaffolding hoping to eliminate the manual steps.

[slamer59]

Hello, you can still have everything as subfolders, each are packages with their own repositories/dependencies AND construct one image for each packages.

I have one Dockerfile at the root folder (taken from dagster exemple) that copy all the files, enter in the subfolder (specified by ARGS statement)
install its dependencies.
This package see other packages if needed.

Then I give a list in the CI to loop over the subfolder to construct all those images. This is where ARGS is important to specify which installation you want.

If dependencies are broken, it fails at the construction of the image before production.

[slamer59]

Hello,
I am using this folder structure as well. I am just entering my_package and call dagster new-project MY_NEW_PROJ && cd MY_NEW_PROJ && pip install -e .

In fact I am using Copier instead of dagster new-project but the idea is the same.

[SaulBurgos]

I am trying to create a dagster structure could handle something like this. Think that each node could potentially growth and change