Access Op inputs in failure hook

[thgeorgiou]

Hello all!

I am new to Dagster, hope this isn't too obvious. I have a job where the first Op produces an ID, associated with an entity in another service. The rest of the Ops take this ID as an input and perform some actions on the entity. I would like this entity to be deleted in case of failure, so I tried to use a failure_hook. However, I can't seem to find how to get my id inside the failure hook. Consider the following example:

@op
def get_id():
    id = ...
    return id

@op
def work_2(id):
    # ...

@op
def work_3(id):
    # ...

@failure_hook
def clean_up(context):
    id = # <--- How to get id from context???
    # clean up ...

@job
def my_job():
    id = get_id()
    work_1.with_hooks({clean_up})(id)
    work_2.with_hooks({clean_up})(id)
    work_3.with_hooks({clean_up})(id)

Any ideas how the Id (or any other input of the failing Op) can be accessed in the hook, through HookContext?

Thanks in advance!

[nehiljain]

I had the same question. I ended up using op_exception and op_output_values to solve this kind of a problem. Docs are at https://docs.dagster.io/_apidocs/hooks#dagster.failure_hook

Is there a better way to do it given that hook context doesn’t have access to the inputs of the op.

[OwenKephart]

This is definitely a tricky problem, and it's correct that it's not currently possible to directly access the op inputs. I wrote up an issue to track this here: #9256

Right now, the best solution for passing runtime information to the HookContext is indeed a hacky thing involving op_exception (at least for failure hooks, where there will be no outputs).

That could look something like:

class MyCustomException(Exception):
    def __init__(self, info):
        self.info = info
        super().__init__("Custom Exception")
        
@op        
def my_op(id):
    try:
         risky_operation()
    except SomeException as e:
         raise MyCustomException(info={"id": id}) from e             

This way, the op_exception will be an instance of MyCustomException, which will have that extra info