Skip to content

fix(ollama): handle incomplete JSON chunks in stream #995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions lib/langchain/llm/ollama.rb
Original file line number Diff line number Diff line change
Expand Up @@ -287,10 +287,28 @@ def auth_headers
end

def json_responses_chunk_handler(&block)
incomplete_chunk_line = nil
proc do |chunk, _size|
chunk.split("\n").each do |chunk_line|
parsed_chunk = JSON.parse(chunk_line)
block.call(parsed_chunk)
if incomplete_chunk_line
chunk_line = incomplete_chunk_line + chunk_line
incomplete_chunk_line = nil
end

parsed_chunk = begin
JSON.parse(chunk_line)

# In some instance the chunk exceeds the buffer size and the JSON parser fails
rescue JSON::ParserError
unless chunk_line.end_with?("}")
incomplete_chunk_line = chunk_line
nil
else
raise
end
end

block.call(parsed_chunk) unless parsed_chunk.nil?
end
end
end
Expand Down