Looking for SDK Feedback!

[mrkaye97]

Hi everyone!

We're working on making our SDKs easier to use, and are looking for feedback from all of you on what you like, dislike, would like to see, and so on about them from your experiences working with them day to day.

In particular, we're thinking about how workflows and steps are defined with the SDK, and how to improve type support when working with them in your codebases. For example, you might define a workflow with one step like this:

@hatchet.workflow()
class ExampleWorkflow:
    @hatchet.step()
    def step1(self, context: Context) -> int:
        input = context.workflow_input()
        return input.get("the_meaning_of_life", 42)

In our view, there are a couple of cons here:

1. If your workflow does not include multiple steps (i.e. if it's not a DAG, and instead if just one task to execute), then there's a lot of boilerplate here (the workflow decorator and the class).
2. To trigger this workflow, you'd want to run something like hatchet.admin.aio.run_workflow(name="ExampleWorkflow", input={"the_meaning_of_life": 314}). There are a couple of clear cons here:
   • You need to refer to ExampleWorkflow by its name (the string) when triggering it, which is difficult for maintainability in large code bases.
   • input is untyped, so you need to refer back to the workflow definition to understand the shape of the input.
3. After you've defined this workflow, you still need to register it on a worker with worker.register_workflow(ExampleWorkflow().

Proposal

I opened up this PR with a proposal for how we might leverage the V2 SDK for solving all three of these problems. The description of that change has some details on how usage would look and what it would allow.

At a high level, it attempts to solve all three of the problems above.

Feedback

If you've made it this far, thank you! We'd love some feedback on these approaches or any others that you think might make sense! In particular:

1. Does the example laid out in the PoC PR make sense to you, or seem like something that would be useful?
2. Are these pain points relevant to you?
3. Are there other things you'd like us to consider?

[nsbradford]

I'd love to see super-lightweight functional annotations, similar to what i can do in celery:

@hatchet.fn("my-custom-workflow")
def foo(x: int, y: str) -> dict
  ...

[knksmith57]

There are so many things that I really like about the proposed changes:

1. Deferred declaration/binding of workflow behavior to the workflow itself
2. Types. I love and greatly appreciate the focus on rich type support
3. Referencing workflows as objects (rather than as ~magic strings)

These changes make meaningful improvements to some gaps that I feel when working with Hatchet in Python:

1. Composability + Reuse
2. Runtime control over DAG routing

That second point, though, still feels lacking. It's cumbersome to define + reason about + exert runtime control over DAG edges, and the best (I think) I can do is really lean into direct child workflow execution of very small, often single-step, workflows.

Here are a few ideas for focus areas to improve ^:

1. First-class support for "skipping" steps
2. First-class support for conditional edges (dependencies, dependent execution, post/pre-execution context transforms)
3. First-class support for routing between steps and workflows based on post conditions of upstream step + workflow dependencies (e.g.: only on <failure|success|retry[failure|success]|skip>, only on triggered by event)

Happy to elaborate on any of these!

Again, love the proposed changes and excited for where Hatchet is headed 🚀

[mrkaye97]

Great feedback! And totally agreed on the points on runtime control and routing - I'll continue to think about better support for all of these going forward 😄. A lot of what you're describing feels similar to features of (e.g.) Airflow to me (I think it supports all of these in different ways), so there's some prior art here for sure