fix: return single-chunk streamed content in Ollama create_stream#7947
Open
HumphreySun98 wants to merge 1 commit into
Open
fix: return single-chunk streamed content in Ollama create_stream#7947HumphreySun98 wants to merge 1 commit into
HumphreySun98 wants to merge 1 commit into
Conversation
When a streamed Ollama completion produced exactly one content chunk and no tool calls (a short response returned in a single chunk), the final-result assembly matched neither the tool-call branch nor the text branch (which required `len(content_chunks) > 1`) and fell through to the `else`, setting `content` to the empty tool-call list and `completion_tokens` to 0. The text had already been streamed to the consumer, but the returned `CreateResult` reported empty content and zero completion tokens. Return the joined content whenever any content chunk was received, and read `completion_tokens` from the final chunk's `eval_count` for all branches (which also fixes the token undercount for tool-call-only streamed responses). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
When a streamed Ollama completion produces exactly one content chunk and no
tool calls (a short response returned in a single chunk), the final-result
assembly in
create_streammatches neither the tool-call branch(
len(content_chunks) > 0 and full_tool_calls) nor the text branch — whichrequired
len(content_chunks) > 1— and falls through to theelse, settingcontentto the empty tool-call list andcompletion_tokensto 0.The text has already been streamed to the consumer chunk-by-chunk, but the
returned
CreateResultreportscontent == []and zero completion tokens, soany consumer reading
result.contentfrom a short streamed response gets anempty list instead of the text, and usage accounting under-reports.
This PR:
len(content_chunks) > 0, so single-chunk textresponses return their content;
completion_tokensfrom the final chunk'seval_countfor allbranches, which also fixes the token undercount for tool-call-only streamed
responses (the old
elsehardcoded 0).The existing multi-chunk streaming tests pass unchanged; a new regression test
covers the single-chunk case (which was previously untested — the existing test
always slices content into ≥2 chunks).
Related issue number
N/A — found during review of the streaming result assembly.
Checks
ruff,mypy, and the relevantpytestlocally).