feat: telemetry usage metering - #7
Open
Ali (alifayed02) wants to merge 7 commits into
Open
Conversation
…rizedFetch Step 1 probe: @opentelemetry/otlp-transformer@0.220.0 exports ProtobufLogsSerializer/ProtobufMetricsSerializer/ProtobufTraceSerializer exactly as expected, no fallback needed.
…112 KiB limit BatchLogRecordProcessor's default maxExportBatchSize (512) can serialize to a protobuf payload larger than workers/telemetry's MAX_BODY_BYTES cap in the elide cloud repo, which 413s the whole batch — dropping every record in it, including any aura.usage.tokens billing records riding along with observability logs. Pin a conservative 64-record cap on the LoggerProvider's batch processor.
emitOtelLog (init.ts) filtered every record — including the aura.usage.tokens billing record — on OTEL_LOG_LEVEL, so an operator setting OTEL_LOG_LEVEL=warn (or lower) to quiet observability noise would silently stop usage metering too. The billing record is metering, not observability, so exempt it from the level gate by eventName. Covered by a new out-of-process probe (mirroring the existing otel-*-probe.ts pattern so the global LoggerProvider singleton never leaks into the test runner): with OTEL_LOG_LEVEL=error, an ordinary info-level bridged log is still suppressed (the gate still works), while a billable chat.usage event's aura.usage.tokens record still reaches the collector (the exemption holds).
… path sendAuthorized did one authorizedFetch and nothing else: a transient 429/5xx permanently dropped the batch (BatchLogRecordProcessor does not re-queue a FAILED export), and no signal meant an export could hang forever. Mirror the stock otlp-exporter-base behavior in spirit: retry on 429/502/503/504 and network rejection with jittered exponential backoff (honoring Retry-After when present) up to 3 attempts total, and bound every attempt with AbortSignal.timeout (10s, the stock default). Non-retryable statuses still fail immediately, with no retry. sleep is injectable via AuthorizedExporterOptions so the new retry tests run instantly instead of on real timers. New tests: retry-then-success on 503 and on a network rejection, exhaustion after persistent 503s -> FAILED, a non-retryable 400 fails immediately without retrying, Retry-After overrides the computed backoff, each attempt carries an AbortSignal, and the previously-missing direct happy-path tests for AuthorizedTraceExporter and AuthorizedMetricExporter through the shared send.
Ali (alifayed02)
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Emit a billing record per chat call and authenticate Aura-tier telemetry exports.
sink-otlp.tsemits oneaura.usage.tokenslog record perchat.usageevent, with one attribute per positive token count (input,output,cache_read,cache_write,reasoning) plus provider/model.Authorized{Log,Trace,Metric}Exporterserialize with the sameotlp-transformerthe stock exporters use, then send throughTokenManager.authorizedFetch— fresh token per export. Only the Aura tier constructs them; env endpoints and an explicittelemetry.endpointboth outrank it and never authenticate.Plus three fixes, each of which silently dropped billing records:
OTEL_LOG_LEVELgated the billing record, so quieting log noise also stopped metering. Now exempt byeventName.sendAuthorizeddid one fetch with no retry, so a transient 503 dropped the batch permanently (BatchLogRecordProcessordoesn't re-queue a FAILED export). Now 3 attempts with jittered backoff honoringRetry-After, each bounded by a 10s timeout.Why
Client half of the Metronome integration. Pairs with elide-dev/cloud#7, which writes these records to
usage_eventsand forwards them to Metronome. Neither half bills anything alone.The event name and attribute keys are consumed verbatim by
workers/telemetry/meter.ts— renaming either silently stops metering, so treat both as frozen. The cloud side pins this with a golden-wire test.Not yet wired end-to-end. Nothing passes
InitTelemetryOptions.cloud, because the CLI has no cloud sign-in command yet (lands withcloud/auth.tsAuraAuthClient.manager). Without a signed-in session there's no token to attach, so Aura-tier exports go out unauthenticated and the edge 401s them — no usage is metered from the CLI until sign-in lands. Built-in and operator tiers are unaffected; Grafana telemetry keeps working.Testing
test/telemetry-authorized-exporters.test.ts(new, 11 tests): retry-then-success on 503 and on network rejection, exhaustion → FAILED, non-retryable 400 without retry,Retry-Afteroverriding backoff, per-attemptAbortSignal, happy path for all three exporters.sleepis injectable so these run instantly.test/otel-log-level-billing-probe.ts(new): out-of-process probe, matching the existingotel-*-probe.tspattern so the globalLoggerProvidersingleton doesn't leak into the runner. WithOTEL_LOG_LEVEL=error, an ordinary info log stays suppressed while the billing record still reaches the collector.Verified locally:
telemetry-authorized-exporters.test.ts11/11. The probe suites need thepi_nativesaddon built (bun --cwd=packages/natives run build); without it, untouched baseline tests fail identically, so that failure is environmental. CI builds natives.bun checkpasses