Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid closing the choice when an exception happens during its generation #203

Open
adubovik opened this issue Dec 16, 2024 · 0 comments
Open

Comments

@adubovik
Copy link
Collaborator

Currently, a choice is closed even when an exception was thrown during its generation:

def __exit__(
self,
exc_type: Optional[Type[BaseException]],
exc: Optional[BaseException],
traceback: Optional[TracebackType],
) -> Optional[bool]:
self.close()
return False

This leads to confusing streams of chunks like this one:

data: {"choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant"}}],"usage":null,"id":"7b8ef0df-26a6-4382-866a-98cf0a36bd1c","created":1732815811,"object":"chat.completion.chunk"}
data: {"choices":[{"index":0,"finish_reason":"stop","delta":{}}],"usage":null,"id":"7b8ef0df-26a6-4382-866a-98cf0a36bd1c","created":1732815811,"object":"chat.completion.chunk"}
data: {"error":{"message":"Something went wrong!","type":"internal_server_error"}}
data: [DONE]

This creates a false impression that the generation of a choice was successful, but something went wrong later, after the choice was generated.

Note that the stages are closed with status=failed in cases of an exception:

def __exit__(
self,
exc_type: Optional[Type[BaseException]],
exc: Optional[BaseException],
traceback: Optional[TracebackType],
) -> Optional[bool]:
if not exc:
if not self._closed:
self.close(Status.COMPLETED)
else:
self.close(Status.FAILED)
return False

for which this no analog in the choice API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

1 participant