RFC: composites invocation

[dpeng817]

In 0.11.10, we added a feature to allow invocation of solids like so:

@solid
def basic_solid():
    return "foo"

assert basic_solid() == 5

The ethos behind this decision was that testing the business logic of a solid should be as close to invoking the underlying business logic as possible.

Composites (composite solids, pipelines, graphs) are a bit of a different story, however. While a solid wraps a more-or-less ordinary python function, composites are a DSL that defines dependencies between solids.

What is the most ergonomic API for testing the business logic within a composite?

Here, we provide several options.

1. Binding context information to composite and invoking
   Composites may require resources and config for the underlying solids to be run, and a main question with a testing API is how to provide this information.

One option is to allow "binding" of resources and config to the composite; like so:

my_composite_solid.bind(resources={"foo": "bar"}, config={"baz_str": "baz"})

This would support providing direct resource instances or definitions. After resources and config are bound however, then we potentially have all the information we need to execute (assuming we use the default, in-memory executor, which is sufficient for most testing purposes). Maybe, then, we can just invoke the bound thing.

bound_composite = my_composite_solid.bind(resources={"foo": "bar"}, config={"baz_str": "baz"})
result = bound_composite()

This is a pretty ergonomic approach, few lines of code, and will likely be intuitive for users approaching dagster for the first time.

However, it may lead to confusion once people realize the magic that underlies an implementation like this. There's already a lot of "magic" surrounding the composites dsl, and introducing more by making them invokable might steepen the learning curve later.

2. Providing a context to invocation
   Another option is to provide a context object at invocation instead of binding things onto the composite.

my_composite_solid(context=build_execution_context(resources={"foo": "bar"}, config={"baz_str": "baz"})

This has the same issues issues of magic implementation as the binding approach has, but is at least in line with the other direct invocation APIs (you similarly construct a context to provide resources to a solid when invoking).

3. Distinct execute method on the instance

Another approach is an execution method on the composite instance that ingests the required information (resources, config, etc). This can either be in the form of a context, or directly.

# option 1
my_composite_solid.execute(resources={"foo": "bar"}, config={"baz_str": "baz"})

# option 2
context = build_execution_context(resources={"foo": "bar"}, config={"baz_str": "baz"})
my_composite_solid.execute(context)

Both of these options are similarly ergonomic to invocation, without being quite as magical.

4. Distinct execution method
   The fourth option is a distinct execute function similar to the execute_pipeline function that already exists.

execute_composite(my_composite_solid, resources={"foo": "bar"}, config={"baz_str": "baz"})

[schrockn]

High-level question: Given CRAG why are we talking about composite-specific invocation. Shouldn't we just have one generic graph execution API?

No idea why this is forcing me to call this an "Answer" :-/

[dpeng817]

that's correct - however, I was trying to not to orient this conversation around CRAG, since the APIs are (somewhat)? still subject to change, and also remove the need for users to have to ramp up on CRAG before reading this. Do you think it would be more useful to orient around a post-CRAG world?

[sryza]

Aside from the name, composite_solids and graphs are almost exactly the same, so I think the idea is that whatever we figure out here would work for graph as well.