feat(dev): OTLP trace storage for local dev by tejaskash · Pull Request #2043 · aws/agentcore-cli

tejaskash · 2026-08-19T19:24:03Z

First slice of the project dev tracing work; #1980 stacks on this.

What this does

The storage half of local trace collection, filesystem I/O only:

otel/transforms.ts — OTLP wire handling: split one export batch into per-trace payloads, normalize protobuf/JSON ids to hex, shape spans/logs for display (attribute flattening, transport-noise filtering). SDKs batch by time, not by trace, so attributing a whole batch to its first trace id would corrupt trace identity — hence the per-trace partitioning.
otel/store.ts — append-only storage: one JSON Lines file per trace, read back raw. Malformed lines are skipped so a half-written line can't break a read.

Ported from the current CLI's operations/dev/otel/transforms.ts, with the per-trace batch partitioning as the one correctness fix over the original.

Verification

Per-layer tests (happy + unhappy paths), tsc/lint/format clean. End-to-end proof of the full pipeline is in #1980 (real project, real Bedrock invocation, traces on disk).

Replaces #2039, closed by a stacking-tool mishap (base flipped to main); identical branch and content.

codecov-commenter · 2026-08-19T19:30:38Z

Codecov Report

❌ Patch coverage is 98.97260% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.13%. Comparing base (71b0f0a) to head (5074354).

Files with missing lines	Patch %	Lines
src/core/dev/otel/transforms.ts	98.57%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2043      +/-   ##
============================================
+ Coverage     97.10%   97.13%   +0.02%     
============================================
  Files           384      386       +2     
  Lines         22683    22975     +292     
============================================
+ Hits          22027    22316     +289     
- Misses          656      659       +3

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

tejaskash · 2026-08-19T19:52:17Z

P2, trace filtering: extractTraceMeta now collects every participating service.name, so the filter matches any of them. Test added.

notgitika · 2026-08-19T20:45:14Z

+        timestamp: new Date(meta.lastSeen).toISOString(),
+        sessionId: meta.sessionId,
+        spanCount: String(meta.spanCount),
+        ...buildTraceDetail(trace.resourceSpans, trace.resourceLogs),


every summary carries the full detail, and there's no cap on trace count. and the inspector re-polls this after every invocation. Nothing prunes the dir either. Since ListTracesOptions is new here, a limit for newest-N seems like the cheap fix we should try.

Side note: get() ends up with no frontend caller because of this.

Added limit. Kept get() — the inspector reads detail through it. Can slim list() to summaries if youd

ah yes, you're right on get(), useTraces does call the detail endpoint, I was looking at useInvocationTraces mb

your reply was cut off here btw. what's still open there is that limit slices after every file is read and built + nothing prunes the dir. can we track that as a fast follow?

Good catch. Tracked as a fast follow: list() builds every file before slicing, and nothing prunes old files. Lands with the Inspector PR.

notgitika · 2026-08-19T20:53:57Z

+ * Normalize a trace/span id that may be base64 (protobuf JSON conversion) or
+ * already hex (JSON ingest) into lowercase hex.
+ */
+export function hexFromB64OrString(value: string | undefined): string {


unrelated, i thought this said B640r and was wondering what type of input it was :(

i thought the same too. Is it worth explicitly writing out Base64 to make this clearer?

Renamed hexFromB64OrString to hexFromBase64OrHex in #1980.

Hweinstock · 2026-08-19T21:54:11Z

+  value?: OtlpAttributeValue;
+}
+
+export type OtlpAttributes = OtlpAttribute[] | Record<string, unknown>;


why do we need to accept both formats?

Fixed to just OtlpAttribute[].

Hweinstock · 2026-08-19T21:55:57Z

+  spanId?: string;
+  parentSpanId?: string;
+  name?: string;
+  kind?: number | string;


same question here

Fixed, normalizeSpanKind handles both.

Pure OTLP wire handling (per-trace batch partitioning, id normalization, frontend shaping) and append-only per-trace JSONL storage. A batch routinely carries spans from several traces, so persistence partitions by trace id — writing whole batches under the first id corrupts trace identity. Consumed by the OTLP collector in #1980, which stacks on this.

A distributed trace spans several local agents whose exports append to the same trace file; filtering by any participant must find it, not only the first service seen.

- narrow OtlpAttributes to the key/value wire form (the flat-record variant had no producer); drop the dead passthrough branches - flatten array attributes through extractAnyValue so ints stay numeric and nested kvlists survive (was stringifying and dropping them) - count rendered (post-filter) spans for the list summary instead of raw records, so the count matches the waterfall the inspector shows - surface non-ENOENT fs errors from reads instead of masking them as empty - add newest-N limit to list() for the inspector's per-invocation poll

flattenAttributes hand-rolled a value branch per AnyValue kind and had no kvlistValue case, so an attribute whose value is a kvlist (or anything the chain didn't enumerate) silently vanished. Route every attribute value through extractAnyValue, which already unwraps all variants including kvlist — smaller and complete. Addresses Gitika's review on transforms.ts (reuse extractAnyValue; kvlist must not disappear).

Hweinstock · 2026-08-21T17:16:59Z

+ * Normalize a trace/span id that may be base64 (protobuf JSON conversion) or
+ * already hex (JSON ingest) into lowercase hex.
+ */
+export function hexFromB64OrString(value: string | undefined): string {


i thought the same too. Is it worth explicitly writing out Base64 to make this clearer?

notgitika

I replied to 2 of my earlier comments. they are non-blocking and can be follow ups.
@Hweinstock and I had a nit on a function name for readability if that could also be addressed in the follow up that would be great!

Follow-ups from #2043 review (Gitika, Harrison): - Add a TraceStore.list spanCount test with transport-noise spans (1 agent + 4 http-send -> "1"), guarding the post-filter count. - Rename hexFromB64OrString -> hexFromBase64OrHex; both reviewers misread B64.

* feat(dev): collect local OTEL traces in project dev An in-process OTLP/HTTP receiver (protobuf via the pinned otlp-transformer decoders, or JSON) persists agent traces through the storage layer. project dev starts it unless --no-traces or the runtime disables instrumentation, points every spawned agent at it (signal-specific OTEL env), rewrites the endpoint to host.docker.internal for containers — with an explicit host-gateway mapping so Linux Docker Engine resolves it — and keeps uvicorn --reload workers instrumented via sitecustomize on PYTHONPATH. Oversized collector requests get a 413 before the connection closes so exporters do not retry them as transient failures. Rebuilt from explicit paths: the previous tree-snapshot commit accidentally reverted unrelated merged work (cdk target guard, config-bundle TUI, error classification). * fix(dev): ack OTLP exports and report trace-persistence failures A batch that can't be persisted (disk full, permissions) was being turned into a 500, which the OTEL SDK exporter retries forever while the user sees nothing. Ack the export (200) so retries stop, and surface the failure once via an onError sink threaded from the collector to the dev handler, which owns the IO to warn the user. Addresses Gitika's review on store.ts:45 (catch in one place; don't let persistence faults read as a silent, retried loss). * test(dev): cover spanCount noise-filtering; clarify hex-id helper name Follow-ups from #2043 review (Gitika, Harrison): - Add a TraceStore.list spanCount test with transport-noise spans (1 agent + 4 http-send -> "1"), guarding the post-filter count. - Rename hexFromB64OrString -> hexFromBase64OrHex; both reviewers misread B64. * docs(dev): clarify collector/httpServer/flag comments per review Harrison review on #1980: - httpServer: generalize the answer-before-close comment (drop OTLP specificity) — the module is a shared io primitive (the Inspector server reuses it). - flags: explain why a default-true boolean is exposed as --no-<name>. - collector: reword the onError comment to state the collector's guarantee (ack + hand to onError) rather than the caller's report-once behavior; drop the volatile "matches the reference CLI" aside. * fix(dev): harden OTEL collector, container reachability, and dev lifecycle Address reviewer findings on the collector and dev wiring: - Validate top-level OTLP shape and return 400 instead of mislabeling a bad payload as a persistence error. - Guard the shared HTTP server against a client that disconnects mid-response so it can no longer crash project dev; add an optional bind host. - Bind the collector to 0.0.0.0 for container runtimes so a container can reach it over the host bridge on Linux. - Run the container template under opentelemetry-instrument so it emits traces. - Keep the collector alive through the child's shutdown grace so final spans are not lost. - Force the OTEL settings that would otherwise let shell or .env.local values disable or break local collection. - Make the uv sitecustomize discovery abortable and read its path from a marker rather than the last merged output line.

github-actions Bot added agentcore-harness-reviewing AgentCore Harness review in progress and removed agentcore-harness-reviewing AgentCore Harness review in progress labels Aug 19, 2026

notgitika reviewed Aug 19, 2026

View reviewed changes

Hweinstock reviewed Aug 19, 2026

View reviewed changes

tejaskash force-pushed the feat/dev-otel-storage branch 2 times, most recently from 0779350 to d72e3c9 Compare August 20, 2026 16:18

tejaskash mentioned this pull request Aug 20, 2026

feat(dev): collect local OTEL traces in project dev #1980

Merged

tejaskash force-pushed the feat/dev-otel-storage branch from d81a320 to 613ca5e Compare August 21, 2026 15:23

tejaskash added 3 commits August 21, 2026 11:37

fix(dev): match trace filters against every participating service

fad9c3b

A distributed trace spans several local agents whose exports append to the same trace file; filtering by any participant must find it, not only the first service seen.

tejaskash force-pushed the feat/dev-otel-storage branch from 613ca5e to 4035b7b Compare August 21, 2026 15:37

Hweinstock approved these changes Aug 21, 2026

View reviewed changes

notgitika approved these changes Aug 21, 2026

View reviewed changes

tejaskash merged commit b03cecb into refactor Aug 21, 2026
28 checks passed

tejaskash deleted the feat/dev-otel-storage branch August 21, 2026 17:51

Uh oh!

Conversation

tejaskash commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

Verification

Uh oh!

codecov-commenter commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tejaskash commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tejaskash Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tejaskash Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tejaskash Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tejaskash Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tejaskash Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

notgitika left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tejaskash commented Aug 19, 2026 •

edited

Loading

codecov-commenter commented Aug 19, 2026 •

edited

Loading

tejaskash commented Aug 19, 2026 •

edited

Loading

tejaskash Aug 21, 2026 •

edited

Loading

tejaskash Aug 21, 2026 •

edited

Loading

tejaskash Aug 21, 2026 •

edited

Loading

tejaskash Aug 21, 2026 •

edited

Loading

tejaskash Aug 21, 2026 •

edited

Loading