[RFC] Ergonomic improvements coming to the config and resources systems

[benpankow]

Summary

Recently, the Dagster team has been focused on tackling programming ergonomics issues which are a hurdle for users. The first of these changes was a move from @repository decorators to the Definitions API.

Clear areas for improvement are the configuration and resources systems. The Dagster config system relies heavily on string indirection, requires users to learn a custom type system, and doesn't take advantage of Python's native typing system for parameterizing or accessing values.

Alongside similar challenges with typing and indirection, a lack of observability into the resource system has become apparent. With users deploying Dagster across environments ranging from a local machine to branch deployments to production, configuration of external services via resources is increasingly important. These resources until now have remained hidden away, an implementation detail rather than a first-class object in the Dagster UI.

We are introducing a new ergonomic layer onto the Dagster config and resources system, referred to here as “Pythonic config” and “Pythonic resources,” with the goal of increasing ergonomic accessibility, simplicity, and observability.

This discussion will provide a brief overview of the API changes which will launch as experimental with the 1.2 release scheduled for March 9. The goal of this document is to gather early feedback and answer questions about these APIs before they launch.

After the 1.2 release in early March this system will be generally available to the community, at which point we we hope to get users actively using the new APIs. We have strong conviction that these APIs are a marked improvement, and will transition towards using it as the new default for examples and content alongside the 1.3 release, scheduled for mid-April.

For those interested in delving into implementation details or more complex use-cases, see the further reading section below.

Note: This is not a breaking change, existing code will continue to work. For the foreseeable future, this ergonomic layer is an opt-in API.

Notable changes

Python typing frontend for the configuration system

Config schemas for assets and ops can now be defined using a Pydantic frontend. Strongly typed config objects can be accessed through the config parameter to asset and op functions rather than pulled from the context . These config objects can be directly constructed when launching runs from code.

Before:

@asset(config_schema={"a_string": str, "an_int": int})
def an_asset(context):
    assert context.op_config["a_string"] == "foo"
    assert context.op_config["an_int"] == 2

materialize(
    [after],
    run_config={"ops": 
        {"after": {"config": {"a_string": "foo", "an_int": 2}}}
    },
)

an_asset(build_op_context(config={"a_string": "foo", "an_int": 2}))

After:

class AnAssetConfig(Config):
    a_string: str
    an_int: int

@asset
def an_asset(config: AnAssetConfig):
    assert config.a_string == "foo"
    assert config.an_int == 2

# This can still be invoked using old syntax and in the config editor
materialize(
    [after],
    run_config=RunConfig(
        ops={"after": AnAssetConfig(a_string="foo", an_int=2)}
    )
)

an_asset(AnAssetConfig(a_string="foo", an_int=2))

Here we can see a number of advantages:

• Asset and op definitions utilize Python types, eliminating the need to understand config_schema and the custom Dagster config schema system.
• No need to access context to obtain config.
• Strongly typed, no more accessing untyped op_config dictionary or building untyped config input when launching runs in code.
• Direct invocation is straightforward.

Class-based resources

Resources and their config schemas can be defined by subclassing a new ConfiguredResource class, also a Pydantic model. Assets and ops can encode their resource dependencies in annotated parameters to the asset or op functions.

Before:

class MyDBResource:
    def __init__(self, connection_uri: str):
        self._connection_uri = connection_uri

    def query(self, query: str):
        return create_engine(self._connection_uri).query(query)

@resource(config_schema={"connection_uri": str})
def my_db(context: InitResourceContext):
    return MyDBResource(context.resource_config["connection_uri"])

@asset(required_resource_keys={"db"})
def get_users(context):
    return context.resources.db.query("SELECT * FROM USERS")

defs = Definitions(
    assets=[an_asset], resources={
        "db": my_db.configured({"connection_uri": "postgresql://user@localhost"})
    }
)

After:

class MyDBResource(Resource):
    connection_uri: str

    def query(self, query: str):
        return create_engine(self.connection_uri).query(query)

@asset
def get_users(db: MyDBResource):
    return db.query("SELECT * FROM USERS")

defs = Definitions(assets=[an_asset], resources={
    "db": MyDBResource(connection_uri="postgresql://user@localhost")}
)

This change also leads to a number of simplifications:

• No need for two-part resource definition with resource class and @resource decorator.
• Elimination of required_resource_keys in favor of strongly typed resource parameters.
• No need to access context to use resources in op or asset body.
• Strongly typed access of resources in ops and assets, explicit Python-object construction of resources in Definitions.

Resources in the UI

To enhance observability of resources and their configuration, a resource page will be added to the Dagster UI.

The resource page surfaces resource metadata, including descriptions of the resource, the assets and ops which rely on it, and the set of config keys and values in the current deployment environment. Values sourced from environment variables are specifically highlighted. The resource page makes it easier to tell at-a-glance what external services your Dagster instance is configured to interact with.

Environment variables and secrets

Often, resources will be configured with secret values (such as a password to a database) or values which vary between deployed environments (such as a target database schema). These values are most often provided through environment variables. The current config syntax allows sourcing config values via environment variables with [StringSource](https://docs.dagster.io/_apidocs/config#dagster.StringSource), but a new API makes this even more natural:

class MyDBResource(Resource):
    connection_uri: str

    def query(self, query: str):
        return create_engine(self.connection_uri).query(query)

@asset
def get_users(db: MyDBResource):
    return db.query("SELECT * FROM USERS")

defs = Definitions(assets=[an_asset], resources={
    "db": MyDBResource(connection_uri=EnvVar("DB_CONN_URL")}
)

# In place of
defs = Definitions(assets=[an_asset], resources={
    "db": MyDBResource(connection_uri="postgres://..."}
)

An instance of the EnvVar class can be substituted for any string config value, meaning users never have to think about StringSource. Lookup of the environment variable is deferred until runtime, when the value of the env var is substituted in the resource’s config. EnvVar also provides improved observability, with config sourced from env vars specifically highlighted in the resource UI.

Potential extension points - resource validation

With resources reframed as Python classes and as first-class APIs, it’s easier for us to layer optional functionality into resources. One use-case we’ve considered is an optional Validated interface which resources could implement. This would allow resources to provide a sort of status check that config is valid and that the resource works:

class SomeResource(ConfigurableResource, Validated):
    username: str
    password: str
    host: str
    
    def validate(self):
        return can_connect(host, username, password)

For example, a resource connecting to an outside SaaS service could return whether it can connect successfully or has a credentials error. This can make deploying code to a production environment easier, and could quickly highlight issues such as missing or misconfigured env vars.

Existing codebases

No changes are needed to existing code - the current config and resource APIs will continue to work alongside this new ergonomic layer.

Users may choose to try out or gradually adopt this system by adding Pythonic config to ops and assets incrementally.

Existing, function-style resources can be adapted into strictly typed Pythonic resources using adapter classes. For more details, see our Pythonic resources docs.

Further reading

For more details on advanced use-cases and implementation details, we have published a set of experimental docs outlining the new config and resource APIs:

• Pythonic asset & op configuration
• Pythonic resources

[PadenZach]

Love it! Been using dataclasses and such a lot in our dagster repo internally and this definitely kicks that sort of pattern up a level! Can't wait to use it

[askvinni]

I really like this change, feels very natural as the typing system also pushed me towards using pydantic for my inputs and outputs, so this extension really helps. Especially like the added visibility over resource configs and types. Always felt a little odd to specify resource keys but not have to specify the class/superclass for them.

My only question is related to a use case I've had for a while that led me down the path of some interesting workarounds: will this new system include first-class support for supplying default configs to assets without having to go the define_asset_job route? I try to keep code as static as possible, meaning some of the assets end up requiring a large amount of configuration to keep the code behind them abstracted.

Editing to add:
After a quick test, I successfully managed to pre configure assets by using pydantic.create_model() and inheriting from the typed config. Seems like a good enough solution for now.

As a followup, are io_managers also going to be defined within the asset parameters, something along the lines of:

@asset
def my_asset(io_manager: MyIOManager):
    ...

[schrockn]

@askvinni what is your use case for getting access to io_manager in the body of an asset? Just want to make sure we understand the goal.

[askvinni]

@schrockn I don't necessarily need to access it from the body of the asset, just caught myself wondering if they will also be a parameter to the assets in the future.

On the one hand, passing them with the io_manager_key parameter and as a resource to Definitions() feels a little inconsistent. On the other, assets shouldn't have to access them directly. More of a stylistic choice.

I think I'm having a hard time wrapping my head around the new concepts and trying to draw parallels between the old configuration system. I often configure resources/io_managers differently for different assets, and was wondering how this would work now, or if we would still have to go the define_asset_job route, which is currently holding me back a little from full declarative scheduling.

[Bonnevie]

This looks amazing! We are heavy users of pydantic at our company, so this is one of the things we've been wishing for :)
Would it be wrong to think of the new resources as akin to assets, just without persistence/IO and always computed in-memory? Are you considering lineage-like features where you can e.g. see which assets use a database connection resource?

[benpankow]

Yes, we're hoping to couple resources with the assets and ops that utilize them in the UI, so as you mentioned it's easier to tell which assets are stored where or interact with which outside systems.

[spenczar]

There's a lot to like here!

What happens for assets which will still use context, like for logging? Do those become def my_asset(config, context, some_upstream, ...), or def my_asset(context, config, some_upstream, ...), or is either fine? I'd like it if the docs were very, very explicit about how this works - is it based on the parameter name, or order?

Relatedly - if I have a signature like def my_asset(config, context), will context.op_config == config?

[benpankow]

Context can be included alongside config as a parameter; either ordering is fine, the distinction is made based on parameter name.

context.op_config will continue to be a dict representation of config, whereas the config parameter is passed as a structured object, for example:

class AnAssetConfig(Config):
    a_string: str
    an_int: int

@asset
def an_asset(context, config: AnAssetConfig):
    assert context.op_config["a_string"] == "foo"
    assert context.op_config["an_int"] == 2

    assert config.a_string == "foo"
    assert config.an_int == 2

[spenczar]

This RFC describes config changes for user-side code, particularly for OpConfigs and Resources. What about the configuration objects passed in other places? For example, JobDefinitions accept a config. Will that accept the new API? How will I specify an Asset Job's configuration schema in this system, which are constructed with define_asset_job?

Another type of config passed in through dictionaries is config for libraries. For example, to set up the container configuration with dagster-k8s used in an asset_job, the configuration gets passed in via tags as a string-typed dictionary:

precovery_index_job = define_asset_job(
    "k8s_asset_name",
    selection="k8s_asset",
    tags={
        "dagster-k8s/config": {
            "container_config": {
                "resources": {
                    "requests": {
                        "cpu": "2000m",
                        "memory": "10Gi",
                        "ephemeral-storage": "1T",
                    },
                },
                "volume_mounts": [{"mount_path": "/tmp/run/", "name": "run-volume"}],
            },
            "pod_spec_config": {
                "volumes": [{"name": "run-volume", "empty_dir": {"size_limit": "1T"}}],
            },
        }
    },
)

Will the schemas for libraries like dagster-k8s be definable with this new system? That would be really really really helpful for clarifying interfaces.

[schrockn]

@spenczar This is currently out-of-scope for this line of work, but thanks for flagging. I agree that typing would greatly improved the DX for config like this.

[bendnorman]

These are super exciting changes! In PUDL we're currently converting our existing pydantic to use dagster config types. This works but would be much more straightforward if dagster objects could be configured using pydantic classes.

@benpankow you mentioned potentially adding validation methods to the Config classes. If the Config classes are now pydantic classes, can we validate the config using the standard pydantic validators?

[benpankow]

This raises a good point on the naming front - the config system does support pydantic validators.

The "resource validation" described above is a runtime check to determine if e.g. passed credentials can access an outside service. Such a check is more heavyweight and probably not something you'd want to implicitly check like with a Pydantic validator. We'll probably want to find another name that's less overloaded.

[bendnorman]

Ohh I see. Thanks for clarifying!

[stkbailey]

This looks great! This makes it much plainer how asset and op/job files should be written (import a resource --> define a config class --> specify asset parameters --> write op job. I can also see this leading to much more use of configuration in general, which I have found challenging to work with previously.

The more I lean on the configuration API, the more I expect it will be useful to view what is being passed implicitly at runtime. This might be outside the scope for this proposal, but it would be useful to also have:

1. the default config values displayed in the Launchpad by default
2. the default run configs inspectable using the "View Tags and Config" for the job

Great work!

[schrockn]

🫡 @stkbailey those changes are on our list. Still figuring out exact timing. One outstanding question here is how important it is to support default values that cannot be changed. If we make the values passed to the pydantic config classes the default values, they become editable in the launchpad, which is a behavior change from configured. Interested in your feedback on the importance of that use case.

[stkbailey]

I don't have a strong understanding of what happens under the hood between configured and editable configurations, but here are my expectations as a user:

• if a value is editable in the launchpad (i.e. configur-able), it should always show with default values in the launchpad, and the evaluated values are attached to "view run configs"
• if a value is not editable in the launchpad (i.e. configured), it should not appear in the launchpad

The latter seems okay to me, because there is an in-code step to run the configured step, which means there is also an in-code opportunity to add the default values to op descriptions, etc. if the data needs to be surfaced. It also allows for configuring secret values, although that seems to not be as important anymore.

[simonvanderveldt]

FYI I believe showing the config at runtime is the same thing that's tracked here #10594
It might be slightly different though. We're looking for a way to show the config for ops that have been configured in code.

[wannesghielens]

I was hoping for more advanced config validation functionality (we currently use an in house written dataclass to dagster config conversion function already to achieve the same kind of functionality described here). Things like min max value constraints, or even fully custom validation functions on configuration.

We currently do this as a first step in our workflows but it would be so much nicer if it would already be signalled to the user even before the workflow can be launched.

[schrockn]

@wannesghielens you should be able to use all of pydantic's functionality to do custom validation. cc: @benpankow

[wannesghielens]

:o ok that's amazing!

[wannesghielens]

I just checked the linked documentation at the bottom in "further reading" which gives an example of what I described, I was a bit to quick during my first read so I didn't notice the extra detailed documentation pages linked at the bottom. Anyways, super stoked for this :D

[f-tremblay]

Awesome news! I'm excited to try the new "Pythonic config" and "Pythonic resources". It's great to see improvements to the configuration and resources system that would simplify and increase ergonomics and observability. Thanks for your hard work!

[f-tremblay]

I have a suggestion for improvement regarding error messages for invalid configs.
Currently, the error tracebacks can be challenging to work with due to a few frustrating issues.

1. It can be difficult to locate the exact location of the mistake. The messages provided are often not specific enough, or hard to understand. Ideally, the error messages should always include a link to the root cause that users could click on in their IDE.
2. The suggested configs are presented in a compact format that is hard to read. If they were presented in a nicely formatted way with indentation, it would be much easier to understand and fix the mistake, especially with deeply nested graphs and subgraphs.

[benpankow]

Hi Felix, this is a good call-out. Config error messages should definitely be given some attention, they are not particularly actionable right now.

[f-tremblay]

I also noticed that there was no mention of how inputs would be configured with the new config API, and I couldn't find any support for inputs in the RunConfig class.

Since we use inputs for graphs and configure them when triggering jobs (e.g. run_config={"inputs": {"a_string": "foo", "an_int": 2}}), I'm wondering if this aspect is also being considered with the new API, and if so, how it will be managed.

[f-tremblay]

When working with ops and graphs, I've found that it's more effective and ergonomic to separate "inputs" and "ops configs". If there's a sensible default value, it should be defined as an "op config" (which can be configured in the run_config if needed). However, if the value is required and cannot be set by default, it should be defined as an "input" (which must be provided in the run_config, under "inputs").

Using "inputs" is also preferable over "ops config" in cases where multiple ops require the same configurable value, or when a subgraph needs a configurable value. These situations can be awkward to handle when using normal "ops config"

[erinov1]

1. How does this interact with PartitionedConfig? We have been unable to adopt partitions for our batch pipelines (because of difficulties with global business calendars, shout out to non-cron schedules based on arbitrary iterators of execution times #9501 and Support using rrule for defining schedule cadences #8713). These jobs have configurable start/end dates. On the other hand, in production we have some workarounds to use partitions, but still rely on PartitionedConfigs because we want batch and production assets to have identical signatures and configs (otherwise the prod assets would not have a start/end date config and we would need custom code to inject info about the asset key, whereas the batch assets need to have a start/end date config, but not access any asset key).
2. Could this be extended to the point where we are simply decorating a typed function? For example:

class AnAssetConfig(Config):
    a_string: str
    an_int: int

@asset(config=AnAssetConfig) # <=== specify the config class here in some way?
def an_asset(a_string: str, an_int: int):
    ...

Especially if resources are now just viewed as classes instances that can be passed as arguments, one can further decouple business logic from dagster itself and simplify testing. (Admittedly that seems easy for users to implement themselves with a new decorator.)

[Daniel-Vetter-Coverwhale]

Is there a way to partially configure the new pythonic resources? Or to override just a piece of their configuration at launch? Previously I was doing .configured and passing most of what I needed in, and I tried to do .configure_at_launch() with most of my values passed through, which I assumed would work like configured() did on the old style of resources, but now when I go to the launchpad I get nothing and if I try to override just the remaining configuration value dagit asks me to reconfigure everything.

[benpankow]

Hi Daniel, currently there isn't a way to do this, but this is something we're considering implementing. Set up #12898 to track this issue.

[Daniel-Vetter-Coverwhale]

When would the validate method be run for Validated resources? I'm thinking probably at resource initialization time to account for secret rotation between deployment and runs, etc, but I could also see it happening at code location loading time. Either way I think it's important to have specified.

If it's at resource initialization time, thenI think that also makes it a good candidate to use for establishing connections, creating db engines, or otherwise warming up resources via @functools.computed_property or other methods, but then validate seems maybe too specific? Am I thinking along the right track or would we want another extension point for those sorts of things?

[benpankow]

This is still something we're working out, but our initial thinking is that it would be run on code location load (with the ability to opt-out for expensive validations) and manually via the UI.

Since validation is a human-triggerable interaction, we aren't thinking of it in the critical path (e.g. warming resources), this is something that might better fit in a more purpose-built "initialization" lifecycle hook. In general, the new class-based structure for resources lends itself well to these sort of hooks.

[Daniel-Vetter-Coverwhale]

The direction and extensibility we get with this class-based approach is pretty awesome, and I look forward to more hooks! I agree with the second point that warming is better in it's own hook.

However, I really think validation should run again during resource initialization, even if it already ran during code location load. Imagine the scenario where you have a secret that exists with correct information during code location load, but perhaps disappears or changes between then and the next run of a pipeline or asset materialization. Or a database that you can connect to on code location load, but not from the spun out k8s pods or jobs themselves do to a temporary network partition or a configuration error... Don't judge me too harshly but this happened to us with our kubernetes deployment and service accounts being bound to the deployment of the code location server, but not the job pods when they launched. It was definitely an error on my part, but I imagine something like a resource that claims to be validated but doesn't actually work to lead to quite a bit of confusion.

If instead the resource is validated at initialization time (or probably better, again at initialization time) you can scope your troubleshooting to the resource itself, and if there's a validate method that users are overriding themselves they should be able to provide detailed messaging around what has broken. I also think validating at initializaiton time will prevent confusion around multiple ops/assets that use the same resource all dying. It removes the step of figuring out whether some code change in those pieces themselves has caused a problem.

[Daniel-Vetter-Coverwhale]

Or maybe we are meant to override create_resource? As I think more on this maybe another hook for resource initialization gives us the same benefits as running validate again in that if the resource isn't valid then it won't correctly initialize. I just know init isn't the right place for actually setting up the resource and I haven't yet seen the recommended place to set up actual initialization that requires code execution, so I've been dealing with it in computed properties, but the code in those won't happen until they get called, so generally post resource initialization I believe