add is_cross_encoder

[Samoed]

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

Small step to #1841. There are a lot of T5 and LLM that used for reranking tasks, but I'm not sure how to annotate them.

[sam-hey]

Suggestion: If I understand the intent of this PR correctly, the primary focus is to enable filtering on the leaderboard. Wouldn't it make more sense to add a list of model types as an Enum? This way, it would also allow filtering for specific categories like Late Interaction models or other types that might be added in the future.

[x-tabdeveloping]

Thanks for looking into this. I think setting False as a default would make more sense, since most models are not cross-encoders, but I'd love to here your arguments about this.

[x-tabdeveloping]

Shouldn't the default be False? Doesn't None signal that we don't know?

[Samoed]

Yes, like don't know. I think we should specify True or False only if someone added this, because in other cases this might be misleading

[x-tabdeveloping]

Why is it None? Do we not know whether it is a cross-encoder or not?

[Samoed]

This is LLM Mistral and because it is not truly cross-encoder I marked it as None (like other LLM and T5 models)

[orionw]

I think I added this as a cross-encoder. I used it like the MonoT5 reranker but zero-shot (no retrieval training).

It’s actually not bad zero-shot, although ofc worse than if trained.

I’d add it as a cross-encoder in this setting since it’s doing full attention between query and doc. But agree it’s a weird case

[x-tabdeveloping]

@sam-hey Yeah, I think, that would probably make sense (an enum or Literal[...]).
What categories would you then choose though?

[sam-hey]

Thanks for raising @sam-hey!

I can definitely see the benefit! On the other hand, having it standardized makes it so each model class has the same function and is more reliable that way.

I can see both sides, but personally I think I would prefer to keep the core search functions in MTEB, so users can see them there and assume each model searches the same within their own “class” (eg that all dense retrievers use the same base functionality). I think it’d be great if we made BM25 a first class MTEB model so we didn’t have to rely on that (and could also add other sparse non-neural versions like Pyserini).

OTOH, there are probably 3 ish other model “classes” or types that would involve a different search functionality: multi-vector (like ColBERT as you say), and then perhaps neural sparse retrieval (like Splade) and generative retrieval.

So we should definitely make it so that each of these could be added, which as @KennethEnevoldsen says likely involves a change to the interface. But since there are less than 10 model “classes”, it seems like we could do that with an if statement. But perhaps it’s too early in the morning and I’m missing something!

Originally posted by @orionw in #1826

I believe the categories would be more or less the same @x-tabdeveloping.

[Samoed]

I think we can add these model types:

• Encoder
• SparseEncoder (ColBERT, BM25)
• BiEncoder
• CrossEncoder
• T5 (rename)
• LLM

[orionw]

Catching up a bit, but are these supposed to be model "classes" or "types" or just general descriptions? This is for the leaderboard, so I could see either way.

If model "classes", it should probably be each type that has a unique way of searching (and correspond to searching code in the repo). e.g. for encoders it's a dense encode followed by some similarity func. For cross-encoders it's prediction over all (query, doc) pairs. For sparse models, it's an index-building step followed by a search step. And so on. If this is the case I think we can simplify that list a bit and exclude LLMs since they're cross-encoders.

If we're adding general descriptors then I think it makes sense to add fine-grained labels like LLM (zero-shot), Cross-Encoder (which implies fine-tuning), Bi-Encoder, Late-Interaction, etc.

[Samoed]

These namings are only for description and the leaderboard.

[Samoed]

@x-tabdeveloping What do you think we should do?