Skip to content

fix: return single-chunk streamed content in Ollama create_stream#7947

Open
HumphreySun98 wants to merge 1 commit into
microsoft:mainfrom
HumphreySun98:fix/ollama-stream-single-chunk-content
Open

fix: return single-chunk streamed content in Ollama create_stream#7947
HumphreySun98 wants to merge 1 commit into
microsoft:mainfrom
HumphreySun98:fix/ollama-stream-single-chunk-content

Conversation

@HumphreySun98

Copy link
Copy Markdown

Why are these changes needed?

When a streamed Ollama completion produces exactly one content chunk and no
tool calls (a short response returned in a single chunk), the final-result
assembly in create_stream matches neither the tool-call branch
(len(content_chunks) > 0 and full_tool_calls) nor the text branch — which
required len(content_chunks) > 1 — and falls through to the else, setting
content to the empty tool-call list and completion_tokens to 0.

The text has already been streamed to the consumer chunk-by-chunk, but the
returned CreateResult reports content == [] and zero completion tokens, so
any consumer reading result.content from a short streamed response gets an
empty list instead of the text, and usage accounting under-reports.

This PR:

  • changes the text branch to len(content_chunks) > 0, so single-chunk text
    responses return their content;
  • reads completion_tokens from the final chunk's eval_count for all
    branches, which also fixes the token undercount for tool-call-only streamed
    responses (the old else hardcoded 0).

The existing multi-chunk streaming tests pass unchanged; a new regression test
covers the single-chunk case (which was previously untested — the existing test
always slices content into ≥2 chunks).

Related issue number

N/A — found during review of the streaming result assembly.

Checks

  • I've included any doc changes needed for https://microsoft.github.io/autogen/ (none required for this change).
  • I've added tests (if relevant) corresponding to the changes introduced in this PR.
  • I've made sure all auto checks have passed (ran ruff, mypy, and the relevant pytest locally).

When a streamed Ollama completion produced exactly one content chunk and no
tool calls (a short response returned in a single chunk), the final-result
assembly matched neither the tool-call branch nor the text branch (which
required `len(content_chunks) > 1`) and fell through to the `else`, setting
`content` to the empty tool-call list and `completion_tokens` to 0. The text
had already been streamed to the consumer, but the returned `CreateResult`
reported empty content and zero completion tokens.

Return the joined content whenever any content chunk was received, and read
`completion_tokens` from the final chunk's `eval_count` for all branches (which
also fixes the token undercount for tool-call-only streamed responses).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant