AttributeError: '' object has no attribute '__name__' when giving a Callable model to Evaluation.evaluate()

[itay-verkh-lightricks]

Bug Description:

Getting a AttributeError: '' object has no attribute '__name__' when I pass a callable subclass of weave.Model to Evaluation.evaluate()

Weave Version:

name : weave
version : 0.51.19
description : A toolkit for building composable interactive data driven applications.

Error Trace:

Traceback (most recent call last):
  File ".venv/lib/python3.11/site-packages/weave/flow/eval.py", line 454, in eval_example
    eval_row = await self.predict_and_score(model, example)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/weave/trace/op.py", line 615, in wrapper
    res, _ = await _do_call_async(
             ^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/weave/trace/op.py", line 494, in _do_call_async
    res, call = await execute_result
                ^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/weave/trace/op.py", line 341, in _call_async
    return handle_exception(e)
           ^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/weave/trace/op.py", line 339, in _call_async
    res = await func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/weave/flow/eval.py", line 172, in predict_and_score
    else model_predict.__name__
         ^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/pydantic/main.py", line 850, in __getattr__
    raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}') from exc

Reproduction

import weave

class MyModel(weave.Model):
    def forward(self, input):
        return {
            "output": input
        }
    
    def __call__(self, *args, **kwargs):
        return self.forward(*args, **kwargs)

dataset = weave.Dataset(
    name="Dataset",
    rows=[{"input": "Hello, World!"}]
)

@weave.op
def score(input, output):
    return {
        "score": input == output
    }

evaluation = weave.Evaluation(
    dataset=dataset,
    scorers=[score]
)
await evaluation.evaluate(MyModel())

notes:

This happens if I decorate MyModel.forward or MyModel.__call__ too, it only starts working when I completely remove the __call__ for MyModel. I need weave.Model to be callable so it can be easily inheritable and replacable

[gtarpenning]

Hey @itay-verkh-lightricks, this is interesting case. I like the PR, going to discuss internally with the team if this should be a supported mechanism going forward. In the meantime, any specific reason why you are using __call__ instead of MyModel.predict? That is the recommended way of getting model predictions in an Eval.

[itay-verkh-lightricks]

Thank you @gtarpenning
my use cases is quite unique I think. I read about weave's integration with dspy and I wanted a dspy.Module (which has a forward and a call method) to subclass weave.Model. The reason I want this is I already have a module's forward that acts as a predict method, and I also want it to be versioned, traced and evaluated with weave. Having the __call__ method on the weave.Model allows me to use my dspy modules without implementing a layer on top of them.
I did run into another issue when writing a base class that inherits both weave.Model and dspy.Module (the predict_and_score function fails when giving self as a keyword argument) and the same when inheriting from weave.Scorer