[RFC] Workspace/repository elimination from UI/tools and new opt-in entrypoint Definitions

[schrockn]

Summary

Dagster has some duplicative and unnecessary concepts, which can introduce friction to the onboarding process and add to the cognitive load of existing users. Workspaces and repositores stick out as problem areas. Workspaces don't do that much––they are a list of code locations––but occupy a lot of headspace in our UI and tools. Repositories can cause immense confusion as (1) they collide with the notion of GitHub repositories but have a different scope leading to confusing states where you can have multiple Dagster repositories within a single GitHub repository and (2) for most users they also did not provide that much value beyond the code location organizational unit, leaving them with the question of why the even exist in the first place.

For these reasons, we are eliminating the concepts of Workspace and Repository from the UI and adding a new entry point API Definitions to replace @repository. There will be a single layer of hierarchy of code locations in our tools.

Note: this is not a breaking change and will not require any code changes from existing users.

Ontology Before	Ontology After

API

Instead of the decorator @repository to group Dagster definitions together, there will instead be a Definitions class. Dagster tools will autodiscover a Definitions object in a module when it is set to the defs variable.

Example:

defs = Definitions(
    assets=[items, comments, stories],
    resources=resources[deployment_name],
)

Rather than previous API

@repository
def repo():
    return [
        *with_resources([items, comments, stories], resource_defs=resource_defs[deployment_name])
    ]

A few notable changes

1. Rather than returning a list of heterogenous definitions, Definitions has typed, named arguments. This is better for discovery, documentation, and robustness.
2. Definitions takes a top-level resources argument, which is automatically applied to assets, rather than requiring the use of with_resources.
3. There can only be one Definitions object per code location, whereas before there could be multiple repositories in the code location.
4. The resources dictionary in Definitions can accept raw objects, not just resource definitions.

Instead of

@resource
def some_object():
   return SomeObject()
   
@repository
def a_repo():
   return [
        *with_resources(asset_requires_some, {"some_object" : some_object})
   ]

it can be

defs = Definitions(
    assets=[asset_requires_some],
    resources={"some_object": SomeObject()}
)

Workspace

A workspace is defined as a set of code locations. In our UIs it served very little purpose and did little else other than confuse users. The workspace.yaml file had some real utility in local development, as it allowed a user to use the CLI or dagit without any additional arguments.

Our project templates now include a pyproject.toml file. This file is part of the Python standard (see PEP). Its primary purpose is to replace setup.py but it is increasingly a standard to use this file to configure any tool that interacts with your python code and it has an extensibility point designed for that purpose. We propose using that file instead of workspace.yaml to enable automatic loading.

Whereas before one would have a workspace.yaml file like so:

load_from:
  - python_package: quickstart_etl

Now you will have a section in pyproject.toml like so:

[tool.dagster]
python_package = "quickstart_etl"

For OSS deployments the workspace was used to point Dagit at grpc servers. This will continue to be the supported way to do that.

What about existing code

You will not need to change any code to adjust to this change. The UI will treat existing repositories as a code location with the name repo_name@location_name.

If you have multiple repositories within a single code location, there will be some potentially counterintuitive behavior in our operational tools. For example, when you click "reload" on a code location from the Code Locations tab, in will in fact reload two entries in this table as they share a process.

We do recommend migrating code as we will be moving our documentation and content to refer to this new construct and avoid using the term Workspace and Repository in documentation and our tools.

If you want to hide the repository name in the UI without migrating to Definitions you can simply set your repository name to "__repository__" if the repository is the only one in the code location.

There are no immediate plans to eliminate @repository.

Alternatives considered

Definitions set to variable rather than register_definitions

Right now the API is to set the constructed Definitions object on a variable at module scope called defs. Our tools search for this name and only this name.

Alternative approaches considered included a function the registered definitions.

register_definitions(assets=[...], resources={...})

While this had some advantages, in the end we found the registration of global state to be, for lack of a better term, distastefully stateful.

However that approach does avoid the error case where one forgets to assign the Definitions object to a variable and it silently fails.

CodeLocation rather than Definitions

We consider creating an object that represented a code location directly. However this introduced the concept of code location very early in tutorials. Additionally some of the information about a code location is determined by the tools that load them. The mental model here is the Definitions object is the set of definitions that will be loading within a code location. In the UI a code location is a live-running process, not just a set of definitions.

Conclusion

We're excited to move forward with this change as we believe it removes substantial cognitive load at low cost. We're looking for feedback about this change. We plan on shipping it (with feedback incorporated) Dec 8th.

[benpankow]

Will there be some sort of functionality to replace multiple repositories in a workspace.yaml for ease-of-loading into Dagit, e.g.

workspace.yaml

load_from:
  - python_package: my_first_package
  - python_package: my_second_package

In other words, how is this fan-out supported in a local OSS instance:

Since a pyproject.toml is scoped to a single python package / code location, what is the way to point Dagit at multiple pyproject.toml files/code locations? Or do we envision local dev to be isolated to a single code location and restrict the multiple code location case to

For OSS deployments the workspace was used to point Dagit at grpc servers. This will continue to be the supported way to do that.

[schrockn]

Great question Ben. We're going to move dagit to support multiple instances of -m and -f so one can just do dagit -m my_first_package -m my_second_package

[benpankow]

That seems pretty intuitive to me 👍

[adam-bloom]

There can only be one Definitions object per code location, whereas before there could be multiple repositories in the code location.

If we have a code location with hundreds of jobs, it’s unclear to me what the recommended mechanism for grouping them in dagit with this change. Repositories may have an odd name, but they work very well for that purpose and having the “folders” in dagit is great. Should we split into multiple python modules/code locations to keep this behavior? Could those still be served by one user code deployment? (Apologies if this was answered and I just missed it)

[adam-bloom]

Hmm. We have over 20 repositories currently (single user code deployment), and can add new ones easily. That’s a lot of grpc servers to spin up, but it is certainly doable. My concern is that we’d experience a fair bit more friction in the process of adding another (can’t just change your user code image, you’d also need to add a deployment).

Is there going to be another mechanism to organize jobs in dagit besides code locations? I need to look at this all more carefully - just trying to understand options for our job/assets layout/organization.

[schrockn]

Given your situation I recommend continuing to use @repository for now, which we will continue to support. Adding additional organization constructs does make sense, but we've been thinking along the lines of tags. Some users want to group across code locations as an example, and repositories don't handle that well.

[ShahBinoy]

I am in the same boat. We deploy on Kubernetes and user-code-deployments. We currently use @repository and our __init__.py file imports one or more modules with it's own @repository. We'll surely need a better construct for easier build management of such scenario

[simonvanderveldt]

Wanted to +1 this, we run a single GRPC server that contains all our repositories as well, these repositories get auto-discovered from the main .py file that's passed to the GRPC server. Creating separate GRPC deployments for each of these isn't really an option.

[schrockn]

@simonvanderveldt @ShahBinoy thanks for feedback.

For now you can use create_repository_using_definitions_args in dagster._core.definitions.definitions_class` to create named repositories with the new API. We will continue to support this for the foreseeable future for existing users who rely on this capability to organize their Dagster deployments.

Can you detail your use case a little more? Exactly how are you organizing your code? This is useful context for us.

[PadenZach]

This looks very promising! I like the simplification a lot! One of my first questions is how this will impact projects where there's multiple different python environments. I know there's some work that's been done on enabling custom docker images and environments in dagster.

EG: Assuming we use poetry managed dependencies, a tree like:

dagster_monorepo/
    domain_a/
        pyproject.toml
        docs/
        src/
            __init__.py
            assets/
            jobs/
    domain_b/
        pyproject.toml
        docs/
        src/
            __init__.py
            assets/
            jobs/

How would I instruct dagster to look at domain_a pyproject.toml and domain_b pyproject.toml? From my understanding of the proposal we only cover pyproject.toml -> domain_a, domain_b and not "domain_a->pyproject.toml, domain_b->pyproject.toml"

[schrockn]

Hey Paden that depends on context.

If domain_a and domain_b were both locally installed, editable modules, then you could load them in one fell swoop using dagit -m domain_one -m domain_two. For more complex use cases I'd need to understand more of how what you wanted to do on which infrastructure.

[johannkm]

At a glance it makes sense to me that @repository has special behavior, while defs = looks more inconsequential and easy to forget. Maybe drilling it home in the docs solves that.

How should we handle users including two defs by mistake?

[shalabhc]

I think dunder names imply something is special to the Python runtime and we should not use that convention. Folks might search Python docs for our dunder name, for instance.

I'd like to add another suggestion to the mix:

Use Python's standard advertize-and-discover mechanism for entry points

Entry points more generally allow a packager to advertise behavior for discovery by other libraries and applications. This feature enables “plug-in”-like functionality, where one library solicits entry points and any number of other libraries provide those entry points.

(more)

The pyproject.toml would look like something this:

# pyproject.toml

[project]
name = "my_project"
version = "0.0.1"

[project.entry-points."dagster.definition"]
my_project = "my_project:defs"

The my_project:defs is a fully qualified reference to a Python object using the entry point syntax.

my_project/__init__.py would define this top level object.

defs = "Definition(...)" 

In fact with just the simple code above, I can install this package and discover the entry point:

>>> from importlib.metadata import entry_points
>>> entry_points().keys()
dict_keys(['console_scripts', 'dagster.definition', 'distutils.commands', 'distutils.setup_keywords', 'egg_info.writers', 'setuptools.finalize_distribution_options'])
>>> entry_points()["dagster.definition"]
(EntryPoint(name='my_project', value='my_project:defs', group='dagster.definition'),)
>>> entry_points()["dagster.definition"][0].load()
'Definition(...)'
>>> 

Advantages:

• standard python mechanism
• can discover the set of definition objects without having to import or load the package

Problems:

• allows multiple entry points
• heavyweight?
• configured outside the code

---

If we don't want to use the full discovery mechanism, we can also just use the entry point syntax in pyproject.toml to point to the Definitions object. Advantages:

• We dont need to come up with a special name but we can use defs by convention. The pyproject.toml would contain the full path to the Definitions() rather than just the package name.
• Discovery is easy - any reader can discover the name and does not have to remember or look up docs.
• Commands like dagit can also accept the entry point name eg dagit -e my_project:my_defs, avoiding knowledge of special name and making it explicit which name is being referred to.

Problems:

• Looking at the code directly doesn't tell you what object is special - you have to look up pyproject.toml.

---

Whatever option we choose, we should probably add a top level dagster project info command or similar that prints out the filename+line used for the top level definition.

[schrockn]

We still need to support loading bare python files, so we can't require the use pyproject.toml to use our tooling.

[dpeng817]

Coming in a bit late to the discussion here, but would like to second that the name defs feels inconsequential to its weight. Additionally, when considering the perspective of a user that is reading a github repository containing dagster definitions, but isn't necessarily super aware of dagster concepts, I think it could be especially confusing. (IE, not clear that defs is a dagster-only thing, although perhaps the fact that the import is from dagster makes it a little more clear). Why not name it dagster_defs? I find it unlikely that a user is going to accidentally stumble on a name like that, and it also makes it a bit clearer for newcomers to a codebase.

I like the option of including in pyproject.toml for more sophisticated users, but seems like a strict downgrade for most users than just setting a variable and allowing us to perform the magic, especially during spinup.

Agreed that we should avoid dunder, especially if we're using generic names such as defs.

[NicolasPA]

I agree with the comment that a magic variable deps is not noticeable enough, and I'd add that it is not uniform with the rest of the Dagster objects using decorators.

So, why not using a decorator?
Dagster would check that the return object is of class Definitions and that only one @definitions is defined.

@definitions
def my_definitions():
  return Definitions(
      assets=[asset_requires_some],
      resources={"some_object": SomeObject()}
  )

Yes, there's some redundancy and a useless function name, but this may be a situation where redundancy is acceptable in exchange for more clarity.

[schrockn]

@NicolasPA we considered that but two things override it:

1. The extra boilerplate you mention
2. For repositories we immediately evaluate the body of the function but that would be true here as well, which is counterintuitive. We've gotten to that behavior based on feedback about catching errors too late.

[shalabhc]

I really like this. The reduction in concepts is great and think the Definitions object is a good approach. Using pyproject.toml also looks like the right thing to do.

IIUC a top level defs will be defined in (or imported into) the top level __init__.py for packages. Will this execute code on import? For example:

# my_project/__init__.py

defs = Definitions(
    assets=[*load_assets_from_dbt_project(), *load_assets_from_some_warehouse()],
    resources={"my_resource": expensive_computation()},
)

Will this make import my_project expensive and also possibly fail if a specific environment is not available? Even if I import a submodule, the top level __init__.py will still execute.

In general I think it is good to have imports not do too much and have a separate phase for one time initialization. It allows tools or repls to import code and introspect it. One simple alternative is to use a function instead:

# my_project/__init__.py

def defs():
   return Definitions(...)

Another option may be to separate the construction of the Definitions object from the construction of inner objects, but not sure what API would work well.

[schrockn]

Timing of this happened such that I responded to Johann's question with the proposed alternatives the involved functions: #10772 (reply in thread)

FWIW I also prefer approaches that have functions that are lazily evaluated by tools (one of the reasons for the original @repository API) but this has caused perpetual confusion (as preventable errors were caught later than they would have otherwise, as an example) and we ended up going with more eager initialization approaches.

The reality is that in the examples you provided those computations would be happening at import time even with the use of the @repository decorator, so the direct construction of Definitions is actually more honest in this regard.

[schrockn]

Treating a defs function at module scope as "special" for this case could make sense and we could add that as an advanced feature.

[shalabhc]

The reality is that in the examples you provided those computations would be happening at import time even with the use of the @repository decorator, so the direct construction of Definitions is actually more honest in this regard.

Ah ok - to your point I didn't even realize this about @repository, but eager evaluation is obvious with the proposed defs = and it is doing the same import time evaluation so seems reasonable.

[markfickett]

This sounds like a great change. The new proposal covers our (pretty simple) use cases, and the UI simplifications do seem easier to navigate.

[dmosesson]

How would loading assets with different resource configurations for the same keys work in this model? Examples that we have of doing this now are things like:

• Using different IO manager configurations for different assets through setting the io_manager key (changing the location of where they are stored for example)
• More generically, using different resource configurations for different assets for the same resources (different credential sets for requests sessions for example)
• Will there be a way to share assets across code locations? This is something we are doing now across repositories, and we don't have a clean way to split out all of our assets into discrete sets

[schrockn]

Hey @dmosesson I was traveling the past few days so I appreciate your patience here.

To take the questions in detail:

Using different IO manager configurations for different assets through setting the io_manager key (changing the location of where they are stored for example)

If you provide the I/O manager without providing the .configured call you would still be able to provide configuration on a per-run basis. However if you are setting different configurations with with_resources we will have to provide an alternative here. How exactly are you setting this up? Code example would be lovely.

More generically, using different resource configurations for different assets for the same resources (different credential sets for requests sessions for example

Similar to above I'd love some more details on exactly how you are doing this.

Will there be a way to share assets across code locations? This is something we are doing now across repositories, and we don't have a clean way to split out all of our assets into discrete sets

The current proposed constraint here is one Definitions per module, so it is entirely possible to set this up so that there is a single python environment with multiple modules with Definitions objects which then import shared code.

If you prefer to send code samples privately feel free to reach out to me directly on Slack!

[dmosesson]

I'll try and work up some code samples later and either post them here or send them directly, but as context, the source of all of our data is basically REST calls, and even the things that you would think are more specialized than that (like s3 compatible storage) all have quirks when it comes to authentication since they all use mTLS, potentially in combination with things like JWT, API tokens etc.

We get data from multiple organizations that put things in their own systems, and to keep ourselves somewhat sane, we have a basic resource that looks something like this:

@resource(config_schema={'ca_cert': StringSource, 'cert': StringSource})
@contextmanager
def my_requests(context):
      #set the ssl context
      #set raise on error
      #set other adapter settings like retries and the like
      # yield a requests session

That resource and others go in a python package called my_project_commons that all of the dagster projects import.

Potentially for each organization whose data we pick up, and for testing purposes on those transformations, we have a file or two that defines all of those assets (~60%) graph backed assets (~20%) and bare ops/jobs (remaining 10%) that uses that my_requests resource. Many of these things are similar enough that for ease of deployment, we just put them all in the same dagster project.

We then have a repo.py file that looks something like:

import my.long.module.here
import my.long.module.there
from my_project_commons import my_requests

@repository
def org1():
    confed_my_requests = my_requests.configured(...)
    assets = load_assets_from_modules([my.long.module.here], group_name='org1')
    confed_assets = with_resources(assets, resource_defs={'requests_session': confed_my_requests})
    return assets
    
@repository
def org2():
    confed_my_requests = my_requests.configured(...)
    assets = load_assets_from_modules([my.long.module.there], group_name='org2')
    confed_assets = with_resources(assets, resource_defs={'requests_session': confed_my_requests})
    return assets

For testing, we replace the resource with a simpler resource that has way more debugging turned on, and no credentials needed (since the testing resource hits a dev-machine local s3 compatible store).

Does this make sense? It probably is the case that there is an easier way to do this, and there are other complexities in terms of how we are passing multiple resources into some of these with different configs (sometimes the same org has two systems with multiple ways of logging in) that I can share examples of if you thing that would help.

On the last point about multiple modules per env, I understand how that could work, but I have to work through how I would integrate that into CI/CD. At first blush, it seems that managing pytoml dependency changes/updates across modules where a module is a unit of deployment is tricky.

[NicolasPA]

Thanks for working on the mental load! I think it will be important for Dagster adoption, as its inclination to introduce innovative and powerful concepts has the negative side effect of creating a steeper learning curve. It reminds me when you "hid" @graph and it went well!

In our on-premise environment, we have been using workspace for a very clear data side reason: separating the definitions used for two strictly separated VLANs that handles different assets and resources. We have one instance of Dagster running in each VLAN, each instance loading only what's relevant to its VLAN. Each VLAN's code is also grouped inside two distinct Dagster repositories. So in our case a Dagster workspace and a Dagster repository are equivalent, therefor one of the two levels is redundant, but we still need one to separate both sides. While the assets are separated by directories, we keep both code bases in a single git repository and a single Python package because many other aspects are shared, including a script to start all the orchestration services.

In this script, depending on the value of an environment variable, we will start Dagster by pointing to one of the two workspaces:

if [[ $VLAN == "special" ]]; then
  workspace_path="special_workspace.yaml"
else
  workspace_path="workspace.yaml"
fi

dagster-daemon run -w $workspace_path &
dagit -w $workspace_path

How could we handle our use case with your proposal ?

[schrockn]

Hey @NicolasPA thanks for the kind words. I think this depends on the complexity of your workspace.yaml files. Those will be sticking around for the forseeable future, but targeted for advanced use cases for loading multiple code locations. For example, in an OSS k8s deployment where you want dagit to point to multiple grpc servers.

If it workspace.yaml file is just loading a couple python files or modules, you will be able to do dagit -m package_one -m package_two and dagit will spin up each package as a separate code location.

[NicolasPA]

Thank you for your answer! It currently targets a specific module. So as far as I understand, we just have to replace the modules containing the repository with ones containing the defs = Definitions() and then loading one module or the other depending on the VLAN.

if [[ $VLAN == "special" ]]; then
  definitions="special_definitions"
else
  definitions="definitions"
fi

dagster-daemon run -w $definitions&
dagit -w $definitions

[dpeng817]

Something particularly nice about this proposal is the subsuming of with_resources, which IMO is a major usability win. We've seen a lot of footguns with with_resources since its introduction, namely with calling with_resources on different sets of assets resulting in resource key collisions. Including a single top-level resources argument makes this impossible to run into (although to be fair, the same could be achieved by adding a resources kwarg to the repo decorator I guess)

[schrockn]

Quick update. We've decided to keep the Definitions class but to not require defs as the required module-scope variable name. Dagster will discover an instance of the Definitions class regardless of variable name. If there is more than one Definitions in scope, it will throw an error.

[askvinni]

I like the sound of it, even though I'd agree with a few other comments talking about the added friction in having only one repository/definitions object per gRPC server.

I especially like having all the information in the pyproject.toml file. Considering we're already using poetry for everything, this feels very natural. As far as formatting is concerned, am I right in assuming that this would be the format to load from multiple gRPC servers?

[[tool.dagster]]
[tool.dagster.grpc_server]
host = "my-grpc.dagster"
port = 4266
location_name = "my_grpc"

[[tool.dagster]]
[tool.dagster.grpc_server]
host = "my-other-grpc.dagster"
port = 4266
location_name = "my_other_grpc"

[schrockn]

For advanced use cases like loading multiple grpc servers, you will still need to use the workspace.yaml file.

[askvinni]

Makes sense. I was under the impression that the workspace.yaml would not be necessary anymore after this change.

[schrockn]

Workspace is going away as a concept in our UIs. The workspace.yaml file remains for advanced use cases of loading sets of code locations.

[ShahBinoy]

@schrockn, what are your thoughts on how will this affect the Kubernetes deployment model.

• Kubernetes deployment via Helm charts of dagster-user-deployments subchart
  • Each workflow module will have to be added to deployments ?
  • If using mono-repo with multiple modules of workflows, what will the recommended deployment model be ?
    • OPTION: If using separate user-deployments for each module ? i.e each module is run via grpc and deployed as separate container, so many GRPC servers can come up
    • OPTION: Single user deployment, what will be the recommended approach on running multiple modules within single user-deployment ?

[schrockn]

So for most people I believe option 1 will be the best one. However if you have an very large number of repositories right I would suggest not migrating in the short-term. We are planning on offering a way to organize lots of definitions in a single code location at some point.

[ShahBinoy]

So for most people I believe option 1 will be the best one. However if you have an very large number of repositories right I would suggest not migrating in the short-term. We are planning on offering a way to organize lots of definitions in a single code location at some point.

Will the grpc server behave and accept multiple modules from the same way dagit command accepts it ?

[schrockn]

It current does not. We thought that enabling grpc to take multiple -m and -f would be confusing in that case since it would end up having substantial different behavior than dagit with the identical parameters. grpc servers are by definition single process, so it would end up scanning multiple modules for Definitions and then loading them as repositories. You can still, however, place multiple repositories in a single module and load them.

I also just added b16f30c (PR #11141) which will allow you to use the same arguments as Definitions to create named repositories if you want to do that as an intermediate step.

[schrockn]

This feature is now live and #11167 and our examples and most of our content now is written in terms of Definitions rather than @repository. The UI no longer refers to Workspaces or Repositories. Thanks everyone for the great discussion and feedback.

Quick notes on changes as a result of this RFC.

1. We no longer require the definitions variable to be called defs. It can be any named. However by convention in all our examples it is defs.
2. There are some users who do use @repository to organize large code locations (with hundreds of Dagster definitions). They should continue to use @repository for now. I've also merged a PR (https://github.com/dagster-io/dagster/pull/11141/files which will go out next week) that will allow you to use the API for constructing Definitions to create named repositories for the time being.