You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
entity_identification:
prompt: |-
You are comparing a submitted answer to an expert answer on a given question. Here is the data:
[BEGIN DATA]
************
[Question]: {input}
************
[Expert]: {ideal}
************
[Submission]: {completion}
************
[END DATA]
Compare the factual content of the submitted answer with the expert answer. Ignore any differences in style, grammar, or punctuation.
The submitted answer may either be a subset or superset of the expert answer, or it may conflict with it. Determine which case applies. Answer the question by selecting one of the following options:
(A) The submitted answer contains all the same details as the expert answer.
(B) The answers differ, but these differences don't matter from the perspective of factuality.
choice_strings: AB
input_outputs:
input: completion
When running the eval using a reasoning model, I encounter the following error:
openai.NotFoundError: Error code: 404 - {'error': {'message': 'This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?', 'type': 'invalid_request_error', 'param': 'model', 'code': None}}
I have written an eval using the Model-Graded Classification template. Below is my eval YAML and modelgraded_spec
oaieval --registry_path=evals/registry/ gpt-4o-mini custom_eval.dev.match-v1
To Reproduce
oaieval --registry_path=evals/registry/ gpt-4o-mini "any eval template"
Code snippets
OS
ubuntu 22.04
Python version
3.12
Library version
oaieval==1.0.6
The text was updated successfully, but these errors were encountered: