fix(providers/openai): keep last usage when stream ends with usage-less chunk - #52
Merged
Merged
Conversation
…ss chunk
Some OpenAI-compatible backends (e.g. Poolside laguna-xs when tools are
declared) report cumulative usage on every delta chunk and end the
stream with a finish_reason chunk whose usage is null, without a
trailing usage-only chunk. The stream loops reassigned usage from every
chunk, so the trailing usage-less chunk wiped the real usage (and
provider metadata) to zero, and consumers saw Usage{0,0,0}.
Only adopt the stream usage hook's result when it actually reports
usage, in both the chat-completions stream loop and the JSON-mode
object stream loop.
ibetitsmike
added a commit
to coder/coder
that referenced
this pull request
Aug 12, 2026
…28068) Bumps the coder/fantasy fork pin to pick up coder/fantasy#52. ## Problem Chats on `poolside/laguna-xs-2.1` showed no context usage: every assistant message persisted NULL token columns, and automatic compaction never triggered, so chats ran to context overflow. AIBridge recorded correct usage for the same requests, so the loss was client-side in fantasy. When tools are declared, laguna-xs reports cumulative usage on every delta chunk and ends the stream with a `finish_reason` chunk whose `usage` is null, with no trailing usage-only chunk. Fantasy's chat-completions stream loops reassigned usage from every chunk, and the default stream usage hook returns zero usage for usage-less chunks, so the trailing finish chunk wiped the real usage one chunk earlier. The Finish part then reported `Usage{0,0,0}`, which chatd persists as NULL (`nullInt64IfNonZero`). ## Fix coder/fantasy#52 adopts the stream usage hook's result only when the chunk actually carries usage, in both the chat-completions stream loop and the JSON-mode object stream loop. This mirrors the aibridge fix in #27967, which is why the gateway recorded usage correctly while fantasy lost it. Spec-compliant backends that emit usage once on the final chunk are unaffected. This PR pins the fork at the merged commit and documents the fork-only patch in the go.mod comment block. ## Validation - coder/fantasy#52: new regression tests for both stream loops, proven red against the unguarded code; full module tests, vet, gofmt, and golangci-lint green; fork CI green before merge. - Here: `go build ./...`, `go vet ./coderd/x/chatd/...`, and `go test ./coderd/x/chatd/...` (including the chatdebug field-coverage guard) all pass with the bumped pin. > Mux acted on Mike's behalf to create this PR.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Some OpenAI-compatible backends (observed with Poolside laguna-xs 2.1 when tools are declared) report cumulative usage on every delta chunk and end the stream with a
finish_reasonchunk whoseusageisnull, with no trailing usage-only chunk.Both chat-completions stream loops in
providers/openai/language_model.goreassignusage, providerMetadatafromstreamUsageFuncon every chunk, andDefaultStreamUsageFuncreturns a zerofantasy.Usage{}(and nil metadata) for usage-less chunks. The trailing usage-less finish chunk therefore wipes the real usage seen one chunk earlier, and the Finish stream part reportsUsage{0,0,0}.Downstream in coder/coder chatd this surfaced as chats with no context usage (NULL token columns on every assistant message) and automatic compaction never triggering for such models.
Fix
Adopt the stream usage hook's result only when it actually reports usage, in both the text/tool stream loop and the JSON-mode object stream loop. Spec-compliant backends that emit usage once on the final chunk are unaffected.
Validation
go test ./... -count=1,go vet ./...,gofmt, andgolangci-lint run providers/openai/...(v2.6.2) all pass.