Dagster Python API Festivus: The Airing of Grievances

[schrockn]

Dagster is a programming model and has a lot of concepts. While we believe that in the end state of a data platform that all these features (or some similar features) make sense and are necessary, we can do a better job of making them more approachable and introducing fewer concepts "upfront" when they seem unnecessary. Additionally, these concepts can often introduce quite a bit of boilerplate and noise. The goal is to make these abstractions "appear" ergonomically when you need them and have them be silent when you don't. We fully acknowledge that we do not hit this bar in all cases at this juncture. Hence, Festivus.

l would love to have users and members of the community weigh in on the relative importance of various aspects of the Python API in particular. Infrastructure and deployment issues are a separate concern.

Here are some example topics that come to mind. This is not meant to be exhaustive:

• Modes and Resources
• Input Output Definition Syntax
  • Use of the type annotation system for dagster type domain
  • Verbosity
• Dagster Types
  • Type Loaders and Materializations and relationship with config generally
  • Relationship with python type system
  • Defining custom types
• Composite Solids, Pipelines, Solids
  • See: RFC: @pipeline and @composite_solid become @graph #2902
• Configuration Schema
• The Event System
• Execution APIs and Testing

[blarghmatey]

I think that there's some opportunity to rely on things like type annotations and docstrings to automatically fill in things like the InputDefinitions and description fields of the @solid decorators for instance. This way it reduces some of the repetition for folks who want to have a thoroughly documented set of solids while also populating the standard structures for a Python program. The values in the decorator can take precedence when defined, but allow for a pass through to the function contents when undefined.

I'm sure I'll have other thoughts as I ruminate on it, but that's my first pass.

[coderanger]

To echo a few of mine from Slack (in no particular order):

• Lack of iteration in pipelines (fan-in and fan-out)
  • More generally it would also be cool to be able to launch a whole different pipeline as a sub-graph
• Conditionals exist using optional outputs but they are confusing to work with and don't look like normal Python if/else
• Arguments in the decorators (@solid and @pipeline especially) feels very verbose in ways that don't always add value
• Configurable/configured pattern is more consistent than ModeDefs, would be nice to shift towards that more fully
• Field(StringSource) vs Field(str), if there is a difference it should be clearer, if there isn't then why have both?
• Making a parameterized pipeline is very doable, but feels harder than it probably should. Maybe just have an implicit resource named params or config that is on all solids by default? Would need to work out where the type schema for it would live though.

[amarrella]

The fact that decorators are untyped raises a lot of mypy warnings (which i can suppress, i know) but more importantly reduces the ability for mypy to check my code in the decorated functions.

In general some other parts of the dagster python code (e.g SerializationStrategy) don't have type annotations and make it hard for a user to guess what the function expects.