From f991058f1d9dfbec6a14b843be1a6e185beb1b74 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Mon, 1 Jun 2026 18:51:10 +0100 Subject: [PATCH 01/48] DRIVERS-3454: design for OTel trace-context propagation over OP_MSG Spec proposal (OP_MSG section kind 3, W3C traceparent, hello tracingSupport negotiation) plus driver-sync POC design to validate it. --- ...026-06-01-otel-opmsg-propagation-design.md | 221 ++++++++++++++++++ 1 file changed, 221 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md diff --git a/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md b/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md new file mode 100644 index 00000000000..415c31fbf73 --- /dev/null +++ b/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md @@ -0,0 +1,221 @@ +# Design: OpenTelemetry Trace-Context Propagation over OP_MSG + +- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) — Support trace context propagation to the server +- **Status:** Design (Investigating phase) +- **Author:** Nabil Hachicha +- **Date:** 2026-06-01 +- **Related:** [DRIVERS-719](https://jira.mongodb.org/browse/DRIVERS-719) (client-side OTel tracing), [SERVER-107128](https://jira.mongodb.org/browse/SERVER-107128) (define trace context in OP_MSG), server POC [10gen/mongo#49930](https://github.com/10gen/mongo/pull/49930) + +--- + +## 1. Summary + +DRIVERS-719 added client-side OpenTelemetry spans to the MongoDB drivers, but those +spans stop at the client boundary: the server starts a fresh, disconnected trace. This +work closes the gap by **propagating the client's active trace context to the server on +each wire message**, so the server can start a *child* span and produce one continuous +client → server timeline. + +Two deliverables, sequenced so the POC validates the spec: + +1. **Spec proposal** — defines the wire contract and negotiation mechanism, written so it + can become a `mongodb/specifications` PR shepherded through the DRIVERS process. +2. **Java driver POC** — a minimal, `driver-sync`-only implementation that exercises every + claim in the spec. Ambiguities in the spec should surface as failing tests or awkward + code; findings feed back into the spec. The POC is internal/throwaway quality and + touches no public API. + +--- + +## 2. Background & constraints + +- **Why transport layer, not command BSON.** Per SERVER-107128, the trace context is + carried in an OP_MSG *section* rather than inside the command document, so the server + can read it before parsing the command — enabling tracing of command-parse failures and + early network handling. +- **The server POC is a prototype.** PR #49930 is explicitly throwaway ("USING AI DO NOT + COPY"), intra-server focused, and unconditionally accepts the new section. It is the + authoritative reference for the **wire format only**, not for negotiation or production + behavior. +- **Process.** Spec changes must flow through DRIVERS tickets; the drivers team shepherds + them. This document is the basis for that proposal. +- **Status of dependencies (June 2026).** No OP_MSG-propagation spec exists yet. + SERVER-107128 is Open/Unassigned. The server `hello` capability flag described below is + **not** in the POC and must be added server-side for the negotiation to work end to end. + +--- + +## 3. Wire contract + +### 3.1 OP_MSG section + +A new OP_MSG section kind is defined: + +| Kind | Name | Status | +|------|-----------------------|------------| +| 0 | Body | existing | +| 1 | Document Sequence | existing | +| 2 | Security Token | existing | +| **3**| **OTel Trace Context**| **new** | + +This matches `kOtelTelemetryContext = 3` in the server POC +(`src/mongo/rpc/op_msg.cpp`). + +### 3.2 Payload format + +The section payload is a single **null-terminated C-string** containing a **W3C +`traceparent`** value: + +``` +00-<32 hex trace-id>-<16 hex span-id>-<2 hex trace-flags>[-] +``` + +- `00` — W3C version. +- trace-id — 16 bytes, 32 lowercase hex chars. +- span-id — 8 bytes, 16 lowercase hex chars (the client span that created the RPC). +- trace-flags — 1 byte, 2 hex chars (e.g. `01` = sampled). +- Base length is **55 chars**; an optional `-` suffix may follow + (server POC appends `span_context.trace_state()->ToHeader()`). + +The string is encoded with the BSON CString convention (UTF-8 bytes + trailing `\0`), +consistent with how the server reads it via `readCStr()`. + +### 3.3 Direction & optionality + +- **Request-only.** No trace data is added to responses; server spans are exported + independently via the OTel pipeline and correlated by trace-id. +- **Sparse.** The section is present only when the client has an **active, sampled** span. + When there is no span, or the span is not sampled, the section is omitted entirely. +- Server behavior (informational): if a trace context is present, the server starts a span + subject to its own external tracing rate limiter. + +--- + +## 4. Negotiation (the key addition over the POC) + +Older or mixed-version servers `uassert(40432)` ("Unknown section kind") on an +unrecognized OP_MSG section. The driver therefore must **not** send section kind 3 to a +server that does not understand it. + +**Mechanism:** the server advertises support in its `hello` response: + +``` +{ ..., "tracingSupport": true } +``` + +Rules: + +- The driver reads `tracingSupport` from each `hello`/handshake response and stores it as a + per-connection / per-server capability (default **false** when absent). +- The driver sends the section **only** on connections whose server advertised + `tracingSupport: true`. +- The capability is re-evaluated on reconnect/failover; a server that stops advertising it + (downgrade) stops receiving the section. + +> **Open question / dependency.** This flag is not in server POC #49930. SERVER-107128 must +> add it. Until then, the POC validates the section via a controlled negotiation +> (see §6). An alternative considered was gating on a `maxWireVersion` bump; rejected for +> the spec because it couples the feature to a wire-version release and is coarser than an +> explicit capability. To be confirmed with the server team. + +--- + +## 5. Edge cases (spec-level, noted; not all in POC scope) + +- **Invalid/empty traceparent.** If the driver cannot produce a valid 55-char traceparent, + it omits the section (never sends a malformed one). The server treats a sub-55-char or + malformed value as absent. +- **tracestate size.** Pass-through only in the POC; the spec should reference W3C limits + (512 chars) and define truncation/drop behavior — to be finalized with the server team. +- **Compression / OP_COMPRESSED.** The section is part of the OP_MSG body that gets + compressed; no special handling expected, but the spec calls this out for confirmation. +- **mongos / load-balanced passthrough.** Out of scope for this driver work; the server is + responsible for forwarding context to downstream services (noted only). + +--- + +## 6. Java POC (driver-sync only) + +Four focused changes in `driver-core`. All internal; no public API change. + +### 6.1 Expose the trace context + +`com.mongodb.internal.observability.micrometer.TraceContext` is currently an empty marker +interface, and `Span.context()` returns it. Extend it minimally: + +```java +public interface TraceContext { + TraceContext EMPTY = new TraceContext() { ... }; + @Nullable String traceParent(); // W3C traceparent, or null if unavailable/unsampled +} +``` + +- Implement in `MicrometerTraceContext`/`MicrometerSpan` by reading `traceId()`, + `spanId()`, and `sampled()` from the underlying Micrometer `io.micrometer.tracing.TraceContext` + and formatting the `00-…` string. Return `null` when the span is no-op or not sampled. +- `TraceContext.EMPTY` and `Span.EMPTY` return `null`. + +### 6.2 Read the capability + +- In `DescriptionHelper`, parse `tracingSupport` from the `hello` result. +- Store on `ServerDescription` (and `ConnectionDescription` as needed) with an + `isTracingSupport()` accessor, following the existing `helloOk` / `maxWireVersion` + patterns. + +### 6.3 Inject the section + +- In `CommandMessage`, add `PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT = 3`. +- In `writeOpMsg()`, after the body/sequence sections, write the new section **iff**: + 1. the connection's server advertised `tracingSupport`, **and** + 2. the active operation `Span` yields a non-null (sampled) `traceParent()`. +- The send path (`InternalStreamConnection.sendAndReceiveInternal`) already has both the + `Span` and the connection description; thread the capability + traceparent into + `CommandMessage` encoding via the existing plumbing. + +### 6.4 Validation — phased + +- **Phase 1 (in-repo, automated — the real proof):** + - Positive: build the OP_MSG bytes for a command with an active sampled span on a + tracing-capable connection; re-parse the sections and assert section kind 3 is present + and its traceparent round-trips and matches the span's trace-id/span-id. + - Negative: capability absent ⇒ no section; no/unsampled span ⇒ no section. +- **Phase 2 (optional, manual runbook — not CI):** build the server POC branch (#49930) + locally with an OTel collector, point the sync driver at it, and confirm the server + emits a child span linked to the client trace. + +### 6.5 Scope guards (YAGNI) + +Explicitly **out of scope** for the POC: reactive/async send path, any public API, +tracestate handling beyond pass-through, sampling rework, and `mongos`/load-balanced +propagation. + +--- + +## 7. Testing summary + +- Unit tests for traceparent formatting (`MicrometerTraceContext.traceParent()`), including + unsampled and no-op cases. +- Unit test for `tracingSupport` parsing in `DescriptionHelper`. +- OP_MSG encode/parse round-trip tests in `CommandMessage` (positive + negative), per §6.4. +- All new code follows `driver-core` conventions (Java 8 baseline, SLF4J, copyright header, + `@Nullable`/`@NonNull`). No reduction in coverage. + +--- + +## 8. Deliverable order + +1. Write & socialize this spec proposal (basis for the DRIVERS spec PR + SERVER-107128 + coordination on the `hello` flag). +2. Build the Java `driver-sync` POC (§6) and run Phase 1 validation. +3. Feed POC findings back into the spec; optionally run Phase 2 end-to-end. + +--- + +## 9. Open questions + +- Server `hello` `tracingSupport` flag ownership/timing (SERVER-107128 is unassigned). +- Final `tracestate` size/truncation policy. +- Whether this section should be designed as a special case or as the first user of the + broader "client-side telemetry section" effort (retry metadata + OTel + client config). +- Confirmation that the W3C `traceparent` (vs. a BSON-structured payload) is the final + on-wire encoding the server team will commit to. From 71944e99cb5e39cc71f4dfbf6abdd4a01ffb2424 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 2 Jun 2026 13:15:19 +0100 Subject: [PATCH 02/48] DRIVERS-3454: implementation plan for OP_MSG trace-context POC (driver-sync) --- .../2026-06-02-otel-opmsg-propagation-poc.md | 815 ++++++++++++++++++ 1 file changed, 815 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-02-otel-opmsg-propagation-poc.md diff --git a/docs/superpowers/plans/2026-06-02-otel-opmsg-propagation-poc.md b/docs/superpowers/plans/2026-06-02-otel-opmsg-propagation-poc.md new file mode 100644 index 00000000000..4c443b34720 --- /dev/null +++ b/docs/superpowers/plans/2026-06-02-otel-opmsg-propagation-poc.md @@ -0,0 +1,815 @@ +# OTel Trace-Context Propagation over OP_MSG (driver-sync POC) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the synchronous MongoDB Java driver inject the active client span's W3C `traceparent` into outgoing OP_MSG messages as a new section (kind 3), but only when the server advertised support — validating the wire contract from the DRIVERS-3454 spec. + +**Architecture:** The driver already builds OP_MSG sections in `CommandMessage.writeOpMsg()` and creates per-operation Micrometer spans accessible via `OperationContext.getTracingSpan()`. We (1) expose the W3C `traceparent` from the span's `TraceContext`, (2) read a `tracingSupport` capability from the `hello` handshake into `ConnectionDescription` → `MessageSettings`, and (3) write section kind 3 in `writeOpMsg()` gated on both the capability and a non-null sampled traceparent. Validation is phased: automated OP_MSG encode/parse round-trip tests (Phase 1), then an optional manual end-to-end runbook against the server POC (Phase 2). + +**Tech Stack:** Java 8 (driver-core baseline), Micrometer Observation (runtime) + Micrometer Tracing (extraction), JUnit 5 + Mockito, Gradle. + +**Spec:** `docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md` + +> **POC scope guards (do NOT exceed):** sync path only; no reactive/async; no `mongos`/load-balanced propagation; `tracestate` is pass-through only; no sampling rework. All code is internal — no breaking public-API change (only additive withers/getters). + +--- + +## File map + +| File | Responsibility | Change | +|---|---|---| +| `driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java` | Trace context abstraction | Add `@Nullable String traceParent()` | +| `driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java` | Micrometer impl | Implement `traceParent()` from the `Observation`'s tracing context | +| `driver-core/build.gradle.kts` | Module deps | Add `micrometer-tracing` as `optionalImplementation` | +| `gradle/libs.versions.toml` | Dependency catalog | Add `micrometer-tracing` library coordinate (main) | +| `driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java` | hello parsing | Parse `tracingSupport`, set on `ConnectionDescription` | +| `driver-core/src/main/com/mongodb/connection/ConnectionDescription.java` | Connection capabilities | Add `tracingSupport` field, `withTracingSupport()`, `isTracingSupport()` | +| `driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java` | Per-message settings | Add `tracingSupported` field + builder method + getter | +| `driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java` | Builds `MessageSettings` | Propagate `tracingSupport` from `ConnectionDescription` | +| `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` | OP_MSG encoding | Add section kind 3, write it gated on settings + traceparent | +| `docs/superpowers/runbooks/otel-opmsg-e2e.md` | Phase 2 manual validation | New runbook | + +--- + +## Task 1: Expose `traceParent()` on the `TraceContext` abstraction + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java` + +- [ ] **Step 1: Add the method to the interface and make `EMPTY` return null** + +Replace the interface body so `EMPTY` implements the new method: + +```java +@SuppressWarnings("InterfaceIsType") +public interface TraceContext { + TraceContext EMPTY = new TraceContext() { + @Override + public String traceParent() { + return null; + } + }; + + /** + * The W3C {@code traceparent} string for this context + * ({@code 00-<32hex traceId>-<16hex spanId>-<2hex flags>}), + * or {@code null} if unavailable or the span is not sampled. + */ + @Nullable + String traceParent(); +} +``` + +Add the import `import com.mongodb.lang.Nullable;` below the package statement. + +- [ ] **Step 2: Compile to verify the interface change is consistent** + +Run: `./gradlew :driver-core:compileJava` +Expected: SUCCESS. (`Span.EMPTY.context()` returns `TraceContext.EMPTY`, which now implements `traceParent()`.) If any other anonymous `TraceContext` implementers exist they will fail to compile — fix each by adding `traceParent()` returning `null`. + +- [ ] **Step 3: Commit** + +```bash +git add driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java +git commit -m "DRIVERS-3454: add traceParent() to TraceContext" +``` + +--- + +## Task 2: Implement `traceParent()` in the Micrometer tracer + +The driver only has the Micrometer **Observation** at runtime; the W3C trace/span IDs live in Micrometer **Tracing**, which attaches a `TracingObservationHandler.TracingContext` to the observation's context when a tracing bridge is configured. We read that. + +**Files:** +- Modify: `gradle/libs.versions.toml` +- Modify: `driver-core/build.gradle.kts` +- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java` +- Test: `driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java` + +- [ ] **Step 1: Add the main `micrometer-tracing` library coordinate** + +In `gradle/libs.versions.toml`, under `[libraries]` add (the `micrometer-tracing` version `1.6.0-M3` already exists under `[versions]`): + +```toml +micrometer-tracing = { module = "io.micrometer:micrometer-tracing", version.ref = "micrometer-tracing" } +``` + +- [ ] **Step 2: Add it as an optional dependency of driver-core** + +In `driver-core/build.gradle.kts`, directly after line 59 (`optionalImplementation(libs.micrometer.observation)`), add: + +```kotlin +optionalImplementation(libs.micrometer.tracing) +``` + +(The existing `"io.micrometer.*;resolution:=optional"` OSGi import on line 105 already covers the new package.) + +- [ ] **Step 3: Write the failing test** + +Create `driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java`: + +```java +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.observability.micrometer; + +import io.micrometer.observation.ObservationRegistry; +import io.micrometer.tracing.test.simple.SimpleTracer; +import io.micrometer.tracing.handler.DefaultTracingObservationHandler; +import com.mongodb.observability.micrometer.MongodbObservation; +import org.junit.jupiter.api.Test; + +import java.util.regex.Pattern; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class MicrometerTraceParentTest { + private static final Pattern TRACEPARENT = + Pattern.compile("00-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}"); + + @Test + void returnsTraceParentForSampledSpan() { + ObservationRegistry registry = ObservationRegistry.create(); + SimpleTracer tracer = new SimpleTracer(); + registry.observationConfig().observationHandler(new DefaultTracingObservationHandler(tracer)); + + MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); + Span span = micrometerTracer.nextSpan(MongodbObservation.COMMAND_OBSERVATION, "find", null, null); + span.openScope(); + try { + String traceParent = span.context().traceParent(); + assertNotNull(traceParent); + assertTrue(TRACEPARENT.matcher(traceParent).matches(), traceParent); + } finally { + span.closeScope(); + span.end(); + } + } + + @Test + void returnsNullWhenNoTracingBridgeConfigured() { + ObservationRegistry registry = ObservationRegistry.create(); + MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); + Span span = micrometerTracer.nextSpan(MongodbObservation.COMMAND_OBSERVATION, "find", null, null); + span.openScope(); + try { + assertNull(span.context().traceParent()); + } finally { + span.closeScope(); + span.end(); + } + } +} +``` + +> Note: confirm the enum constant name in `MongodbObservation` (e.g. `COMMAND_OBSERVATION`); if it differs, use the actual command-span constant. `SimpleTracer`/`DefaultTracingObservationHandler` come from the test-scoped `micrometer-tracing-integration-test` dependency already present. + +- [ ] **Step 4: Run the test to verify it fails to compile/fails** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.observability.micrometer.MicrometerTraceParentTest"` +Expected: FAIL — `MicrometerTraceContext` does not yet implement `traceParent()`. + +- [ ] **Step 5: Implement `traceParent()` in `MicrometerTraceContext`** + +In `MicrometerTracer.java`, add imports: + +```java +import io.micrometer.tracing.TraceContext; +import io.micrometer.tracing.handler.TracingObservationHandler; +``` + +> The driver's own type is also named `TraceContext`; reference the Micrometer one by its fully-qualified name in code to avoid the clash (do NOT add the conflicting import). Use `io.micrometer.tracing.TraceContext` inline. + +Replace the `MicrometerTraceContext` inner class with: + +```java + /** + * Represents a Micrometer-based trace context. + */ + private static class MicrometerTraceContext implements TraceContext { + @Nullable + private final Observation observation; + + MicrometerTraceContext(@Nullable final Observation observation) { + this.observation = observation; + } + + @Override + @Nullable + public String traceParent() { + if (observation == null) { + return null; + } + TracingObservationHandler.TracingContext tracingContext = + observation.getContextView().getOrNull(TracingObservationHandler.TracingContext.class); + if (tracingContext == null || tracingContext.getSpan() == null) { + return null; + } + io.micrometer.tracing.TraceContext ctx = tracingContext.getSpan().context(); + if (ctx == null || ctx.traceId() == null || ctx.spanId() == null) { + return null; + } + Boolean sampled = ctx.sampled(); + if (sampled == null || !sampled) { + return null; + } + return "00-" + ctx.traceId() + "-" + ctx.spanId() + "-01"; + } + } +``` + +> `traceId()`/`spanId()` from Micrometer Tracing are already lowercase hex of the correct length. We emit flags `01` (sampled) because we only propagate sampled spans, matching spec §3.3. + +Note: the `import io.micrometer.tracing.TraceContext;` added above is unused if you reference it fully-qualified — remove it and keep only the `TracingObservationHandler` import to satisfy the no-unused-import check. + +- [ ] **Step 6: Run the test to verify it passes** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.observability.micrometer.MicrometerTraceParentTest"` +Expected: PASS (both tests). + +- [ ] **Step 7: Commit** + +```bash +git add gradle/libs.versions.toml driver-core/build.gradle.kts \ + driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java \ + driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java +git commit -m "DRIVERS-3454: extract W3C traceparent from Micrometer span" +``` + +--- + +## Task 3: Read `tracingSupport` from `hello` into `ConnectionDescription` + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/connection/ConnectionDescription.java` +- Modify: `driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java` +- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java` + +> `ConnectionDescription` is public API. The change is **additive only** (new wither + getter; existing constructors keep delegating with `tracingSupport=false`), so it is binary-compatible. Do NOT change any existing public constructor signature. + +- [ ] **Step 1: Read the actual `ConnectionDescription` constructor chain and the `withServerType` wither** + +Run: `sed -n '40,210p' driver-core/src/main/com/mongodb/connection/ConnectionDescription.java` +Note the most-derived constructor (the one with `serviceId` + all fields) and how `withConnectionId`/`withServiceId`/`withServerType` build a copy. You will mirror that pattern exactly. + +- [ ] **Step 2: Write the failing test** + +Create `driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java`: + +```java +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.connection; + +import com.mongodb.ServerAddress; +import com.mongodb.connection.ClusterConnectionMode; +import com.mongodb.connection.ConnectionDescription; +import com.mongodb.connection.ConnectionId; +import com.mongodb.connection.ServerId; +import org.bson.BsonDocument; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class DescriptionHelperTracingTest { + private static final ConnectionId CONNECTION_ID = + new ConnectionId(new ServerId(new com.mongodb.connection.ClusterId(), new ServerAddress())); + + private static BsonDocument hello(final boolean tracing) { + BsonDocument doc = BsonDocument.parse( + "{ ok: 1, ismaster: true, maxWireVersion: 25, minWireVersion: 0," + + " maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, maxWriteBatchSize: 100000 }"); + if (tracing) { + doc.put("tracingSupport", org.bson.BsonBoolean.TRUE); + } + return doc; + } + + @Test + void parsesTracingSupportTrue() { + ConnectionDescription description = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(true)); + assertTrue(description.isTracingSupport()); + } + + @Test + void defaultsTracingSupportFalseWhenAbsent() { + ConnectionDescription description = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(false)); + assertFalse(description.isTracingSupport()); + } +} +``` + +> Confirm `createConnectionDescription`'s exact signature/visibility (Explore reported `static ConnectionDescription createConnectionDescription(ClusterConnectionMode, ConnectionId, BsonDocument)`). If the package-private method isn't visible, the test is already in the same `com.mongodb.internal.connection` package, so it is. + +- [ ] **Step 3: Run the test to verify it fails** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.DescriptionHelperTracingTest"` +Expected: FAIL — `isTracingSupport()` does not exist. + +- [ ] **Step 4: Add the field, wither, and getter to `ConnectionDescription`** + +Add the field next to the other `private final` fields (after `logicalSessionTimeoutMinutes`): + +```java + private final boolean tracingSupport; +``` + +In the most-derived constructor (the one with `serviceId` and all fields), add `, false` initialization by assigning `this.tracingSupport = false;` at the end of its body. For all other delegating constructors no change is needed (they call the most-derived one). + +Add a wither (mirror `withServerType`) — it constructs a copy via the most-derived constructor and then sets the flag. Since the constructor sets `tracingSupport=false`, implement the wither by copying through a private all-args path. Concretely, add: + +```java + /** + * Returns a copy of this {@code ConnectionDescription} with the given tracing-support capability. + * + * @param tracingSupport whether the server advertised OpenTelemetry trace-context support + * @return the new connection description + */ + public ConnectionDescription withTracingSupport(final boolean tracingSupport) { + ConnectionDescription copy = new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, + maxBatchCount, maxDocumentSize, maxMessageSize, compressors, saslSupportedMechanisms, + logicalSessionTimeoutMinutes); + copy.tracingSupportOverride = tracingSupport; + return copy; + } + + /** + * @return whether the server advertised OpenTelemetry trace-context support in its hello response + */ + public boolean isTracingSupport() { + return tracingSupportOverride != null ? tracingSupportOverride : tracingSupport; + } +``` + +Because `tracingSupport` is `final`, add a separate non-final override field to keep the change additive without rewriting every constructor: + +```java + @Nullable + private Boolean tracingSupportOverride; +``` + +(`@Nullable` import `com.mongodb.lang.Nullable` already present in this file; verify and add if missing.) Remove the now-unnecessary `this.tracingSupport = false;` line and instead initialize the field inline at declaration: `private final boolean tracingSupport = false;` is illegal for a constructor-set final — so declare it `private final boolean tracingSupport;` and assign `this.tracingSupport = false;` in the most-derived constructor only. + +> Rationale: this keeps all five public constructors source- and binary-compatible. The override field carries the capability set post-construction by `withTracingSupport`. + +- [ ] **Step 5: Set the capability in `DescriptionHelper.createConnectionDescription`** + +Add a parser near the other private getters: + +```java + private static boolean getTracingSupport(final BsonDocument helloResult) { + return helloResult.getBoolean("tracingSupport", org.bson.BsonBoolean.FALSE).getValue(); + } +``` + +In `createConnectionDescription`, change the returned value to apply the wither. Find the `return connectionDescription;` (or the `new ConnectionDescription(...)` return) and wrap it: + +```java + return connectionDescription.withTracingSupport(getTracingSupport(helloResult)); +``` + +(If the method returns the `new ConnectionDescription(...)` directly, assign it to a local `ConnectionDescription connectionDescription = new ConnectionDescription(...);` first, then return the wither call.) + +- [ ] **Step 6: Run the test to verify it passes** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.DescriptionHelperTracingTest"` +Expected: PASS (both tests). + +- [ ] **Step 7: Commit** + +```bash +git add driver-core/src/main/com/mongodb/connection/ConnectionDescription.java \ + driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java \ + driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java +git commit -m "DRIVERS-3454: parse hello tracingSupport into ConnectionDescription" +``` + +--- + +## Task 4: Carry `tracingSupported` through `MessageSettings` + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java` +- Modify: `driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java` + +- [ ] **Step 1: Add the field + builder method + getter to `MessageSettings`** + +Add next to `sessionSupported` (field after line 60): + +```java + private final boolean tracingSupported; +``` + +In the `Builder` (after `sessionSupported` field, line 82): + +```java + private boolean tracingSupported; +``` + +Add the builder setter (mirror `sessionSupported(...)`, after line 137): + +```java + public Builder tracingSupported(final boolean tracingSupported) { + this.tracingSupported = tracingSupported; + return this; + } +``` + +In the private `MessageSettings(final Builder builder)` constructor (line ~197), add: + +```java + this.tracingSupported = builder.tracingSupported; +``` + +Add the getter (mirror `getMaxWireVersion()`, near line 181): + +```java + public boolean isTracingSupported() { + return tracingSupported; + } +``` + +- [ ] **Step 2: Populate it in `ProtocolHelper`** + +In `ProtocolHelper.java` at the `MessageSettings.builder()` chain (line 231), directly after the existing `.sessionSupported(connectionDescription.getLogicalSessionTimeoutMinutes() != null)` line (line 237), add: + +```java + .tracingSupported(connectionDescription.isTracingSupport()) +``` + +- [ ] **Step 3: Compile** + +Run: `./gradlew :driver-core:compileJava` +Expected: SUCCESS. + +- [ ] **Step 4: Commit** + +```bash +git add driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java \ + driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java +git commit -m "DRIVERS-3454: thread tracingSupported into MessageSettings" +``` + +--- + +## Task 5: Write OP_MSG section kind 3 in `CommandMessage.writeOpMsg()` + +The section format matches the server POC's `kSecurityToken`/`kOtelTelemetryContext`: a single section-kind byte followed by a CString (no 4-byte length prefix). The server reads it with `readCStr()`. + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` + +- [ ] **Step 1: Add the section-kind constant** + +After `PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE` (line 82) add: + +```java + /** + * Specifies that the `OP_MSG` section payload is a W3C traceparent C-string (OpenTelemetry trace context). + */ + private static final byte PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT = 3; +``` + +- [ ] **Step 2: Add the import for the tracing Span** + +In the import block add: + +```java +import com.mongodb.internal.observability.micrometer.Span; +``` + +- [ ] **Step 3: Write the section in `writeOpMsg`, just before the flag bits are backpatched** + +In `writeOpMsg(...)`, immediately **before** the comment `// Write the flag bits` (line 280), insert: + +```java + writeOtelTraceContextSection(bsonOutput, operationContext); + +``` + +Then add the private method (place it after `writeOpMsg`, before `writeOpQuery`): + +```java + private void writeOtelTraceContextSection(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext) { + if (!getSettings().isTracingSupported()) { + return; + } + Span tracingSpan = operationContext.getTracingSpan(); + if (tracingSpan == null) { + return; + } + String traceParent = tracingSpan.context().traceParent(); + if (traceParent == null) { + return; + } + bsonOutput.writeByte(PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT); + bsonOutput.writeCString(traceParent); + } +``` + +> Rationale: at encode time only the operation-level span exists in `OperationContext` (the command span is created later, in `InternalStreamConnection`). The operation span is a valid parent for the server's RPC span. `context().traceParent()` returns `null` for no-op/unsampled spans, so a missing or unsampled span naturally omits the section (spec §3.3). Gating on `isTracingSupported()` ensures we never send the section to a server that would `uassert(40432)` (spec §4). + +- [ ] **Step 4: Compile** + +Run: `./gradlew :driver-core:compileJava` +Expected: SUCCESS. + +- [ ] **Step 5: Commit** + +```bash +git add driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +git commit -m "DRIVERS-3454: write OP_MSG otel trace-context section (kind 3)" +``` + +--- + +## Task 6 (Phase 1 validation): OP_MSG round-trip encode test + +This is the automated proof of the wire contract: encode a command with a tracing-capable connection + a sampled span, then scan the produced bytes for section kind 3 + the exact traceparent C-string; assert it is absent when the capability is off or the span yields no traceparent. + +**Files:** +- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` + +- [ ] **Step 1: Write the failing test** + +Create `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java`: + +```java +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.connection; + +import com.mongodb.MongoNamespace; +import com.mongodb.ReadConcern; +import com.mongodb.ReadPreference; +import com.mongodb.ServerApi; +import com.mongodb.connection.ClusterConnectionMode; +import com.mongodb.connection.ServerType; +import com.mongodb.internal.TimeoutContext; +import com.mongodb.internal.connection.MessageSequences.EmptyMessageSequences; +import com.mongodb.internal.observability.micrometer.Span; +import com.mongodb.internal.observability.micrometer.TraceContext; +import com.mongodb.internal.session.SessionContext; +import com.mongodb.internal.validator.NoOpFieldNameValidator; +import org.bson.BsonDocument; +import org.bson.BsonString; +import org.bson.ByteBuf; +import org.junit.jupiter.api.Test; + +import java.nio.charset.StandardCharsets; +import java.util.List; + +import static com.mongodb.internal.mockito.MongoMockito.mock; +import static com.mongodb.internal.operation.ServerVersionHelper.LATEST_WIRE_VERSION; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.when; + +class CommandMessageOtelTraceContextTest { + private static final MongoNamespace NAMESPACE = new MongoNamespace("db.test"); + private static final BsonDocument COMMAND = new BsonDocument("find", new BsonString(NAMESPACE.getCollectionName())); + private static final String TRACE_PARENT = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"; + + private static CommandMessage commandMessage(final boolean tracingSupported) { + return new CommandMessage(NAMESPACE.getDatabaseName(), COMMAND, NoOpFieldNameValidator.INSTANCE, + ReadPreference.primary(), + MessageSettings.builder() + .maxWireVersion(LATEST_WIRE_VERSION) + .serverType(ServerType.REPLICA_SET_PRIMARY) + .sessionSupported(true) + .tracingSupported(tracingSupported) + .build(), + true, EmptyMessageSequences.INSTANCE, ClusterConnectionMode.MULTIPLE, (ServerApi) null); + } + + private static OperationContext operationContextWithSpan(final String traceParentOrNull) { + SessionContext sessionContext = mock(SessionContext.class, mock -> { + when(mock.getClusterTime()).thenReturn(null); + when(mock.hasSession()).thenReturn(false); + when(mock.getReadConcern()).thenReturn(ReadConcern.DEFAULT); + when(mock.notifyMessageSent()).thenReturn(true); + when(mock.hasActiveTransaction()).thenReturn(false); + when(mock.isSnapshot()).thenReturn(false); + }); + TimeoutContext timeoutContext = mock(TimeoutContext.class); + TraceContext traceContext = () -> traceParentOrNull; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + return mock(OperationContext.class, mock -> { + when(mock.getSessionContext()).thenReturn(sessionContext); + when(mock.getTimeoutContext()).thenReturn(timeoutContext); + when(mock.getTracingSpan()).thenReturn(span); + }); + } + + private static byte[] encodeToBytes(final CommandMessage message, final OperationContext operationContext) { + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + List buffers = output.getByteBuffers(); + byte[] bytes = new byte[output.getSize()]; + int pos = 0; + for (ByteBuf buf : buffers) { + int remaining = buf.remaining(); + buf.get(bytes, pos, remaining); + pos += remaining; + } + buffers.forEach(ByteBuf::release); + return bytes; + } + } + + private static boolean containsTraceParentSection(final byte[] message) { + // Section kind 3 byte immediately followed by the null-terminated traceparent. + byte[] needle = new byte[1 + TRACE_PARENT.length() + 1]; + needle[0] = 3; + byte[] tp = TRACE_PARENT.getBytes(StandardCharsets.UTF_8); + System.arraycopy(tp, 0, needle, 1, tp.length); + needle[needle.length - 1] = 0; + outer: + for (int i = 0; i + needle.length <= message.length; i++) { + for (int j = 0; j < needle.length; j++) { + if (message[i + j] != needle[j]) { + continue outer; + } + } + return true; + } + return false; + } + + @Test + void writesSectionWhenSupportedAndSampledSpanPresent() { + byte[] bytes = encodeToBytes(commandMessage(true), operationContextWithSpan(TRACE_PARENT)); + assertTrue(containsTraceParentSection(bytes), "expected OP_MSG section kind 3 with traceparent"); + } + + @Test + void omitsSectionWhenCapabilityAbsent() { + byte[] bytes = encodeToBytes(commandMessage(false), operationContextWithSpan(TRACE_PARENT)); + assertFalse(containsTraceParentSection(bytes), "must not send section to non-tracing server"); + } + + @Test + void omitsSectionWhenSpanHasNoTraceParent() { + byte[] bytes = encodeToBytes(commandMessage(true), operationContextWithSpan(null)); + assertFalse(containsTraceParentSection(bytes), "must omit section when span yields no traceparent"); + } +} +``` + +> If `ByteBufferBsonOutput` exposes a different accessor than `getByteBuffers()`/`getSize()`, mirror whatever the existing `CommandMessageTest`/`CommandMessageSpecification` uses to read encoded bytes (e.g. `getByteBuffers()` then `ByteBufNIO`); adjust `encodeToBytes` accordingly. The `TraceContext traceContext = () -> traceParentOrNull;` lambda works because `TraceContext` is now a single-abstract-method interface returning the traceparent. + +- [ ] **Step 2: Run to verify it fails** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest"` +Expected: FAIL before Task 5 is implemented; after Task 5 it should pass. (If running tasks in order, this confirms PASS.) + +- [ ] **Step 3: Run to verify it passes** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest"` +Expected: PASS (all three). + +- [ ] **Step 4: Commit** + +```bash +git add driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +git commit -m "DRIVERS-3454: round-trip test for OP_MSG otel trace-context section" +``` + +--- + +## Task 7 (Phase 2, optional/manual): end-to-end runbook against the server POC + +Not automated / not CI. Documents how to confirm the server starts a linked child span. + +**Files:** +- Create: `docs/superpowers/runbooks/otel-opmsg-e2e.md` + +- [ ] **Step 1: Write the runbook** + +Create `docs/superpowers/runbooks/otel-opmsg-e2e.md`: + +```markdown +# Runbook: OTel OP_MSG propagation end-to-end (manual) + +Validates DRIVERS-3454 end to end: a sync-driver client span linked to a server child span. + +## Prerequisites +- A local build of the server POC branch (10gen/mongo PR #49930), which accepts OP_MSG + section kind 3 and starts a server span from it. NOTE: the POC does NOT yet advertise + `tracingSupport` in hello. Until SERVER-107128 adds it, temporarily force the driver + capability on for this manual run (see step 3). +- An OpenTelemetry collector / exporter the server POC is configured to export traces to + (e.g. Jaeger via OTLP), plus a Micrometer Tracing OTel bridge on the client. + +## Steps +1. Build & run the server POC `mongod` with tracing enabled and sampling at 100%. +2. Start a Jaeger all-in-one (or OTLP collector) and point both server and client exporters at it. +3. In a scratch sync-driver program, configure a `MongoClient` with an `ObservationRegistry` + that has an OTel-backed `DefaultTracingObservationHandler` (so `traceParent()` is non-null + and sampled). Run a `find`. + - Temporary capability override for the run (POC server lacks the hello flag): start an + OTel root span yourself, then run the command. Because the server POC accepts kind 3 + unconditionally, set the driver `tracingSupported` to true by connecting to a server + whose hello you patch to include `tracingSupport: true`, OR temporarily hardcode + `isTracingSupport()`/`tracingSupported` to `true` on the local branch for the manual run + only (revert before commit). +4. In Jaeger, confirm a single trace contains BOTH the client `find` span and a server span, + with the server span's parent = the client span id sent in the traceparent. + +## Pass criteria +- One trace, two+ spans, correct parent/child linkage across the client→server boundary. +- With the driver capability off (default), no server span is created (negative check). + +## Notes +- This proves the wire format end to end; the automated Phase 1 test + (`CommandMessageOtelTraceContextTest`) is the regression guard. +- Findings (any format mismatch, tracestate handling, flags) feed back into the spec. +``` + +- [ ] **Step 2: Commit** + +```bash +git add docs/superpowers/runbooks/otel-opmsg-e2e.md +git commit -m "DRIVERS-3454: add Phase 2 e2e validation runbook" +``` + +--- + +## Task 8: Full module verification + +- [ ] **Step 1: Format + static checks + tests for driver-core** + +Run: `./gradlew :driver-core:spotlessApply :driver-core:check` +Expected: BUILD SUCCESSFUL. Fix any spotless/checkstyle issues in the files you touched (copyright headers, import order, no unused imports — particularly the Micrometer `TraceContext` import note in Task 2). + +- [ ] **Step 2: Confirm the new tests ran and passed** + +Run: `./gradlew :driver-core:test --tests "*OtelTraceContext*" --tests "*MicrometerTraceParent*" --tests "*DescriptionHelperTracing*"` +Expected: PASS. + +- [ ] **Step 3: Final commit if spotless changed anything** + +```bash +git add -A && git commit -m "DRIVERS-3454: apply spotless formatting" || echo "nothing to commit" +``` + +--- + +## Self-Review (completed during planning) + +**Spec coverage:** +- §3.1 section kind 3 → Task 5. §3.2 W3C traceparent format → Task 2 (`00-…-01`). §3.3 request-only / sparse / sampled-only → Task 2 (null unless sampled) + Task 5 (omit when null). §4 `tracingSupport` negotiation → Tasks 3–5 (parse → MessageSettings → gate). §6.1 `TraceContext.traceParent()` → Task 1. §6.2 read capability → Task 3. §6.3 inject → Task 5. §6.4 Phase 1 automated → Task 6; Phase 2 manual → Task 7. §7 testing → Tasks 2,3,6 + Task 8 check. +- Out-of-scope items (reactive, mongos, tracestate beyond pass-through, sampling rework) intentionally have no task — matches spec §6.5. + +**Known follow-ups (not blockers for the POC):** +- Server `hello` `tracingSupport` flag does not exist yet (SERVER-107128). Phase 1 fully validates the driver without it; Phase 2 documents the temporary override. +- The `ConnectionDescription.tracingSupportOverride` approach (Task 3) keeps public constructors binary-compatible; a production implementation would likely fold the field into a new constructor + builder during the non-POC work. + +**Type consistency:** `traceParent()` (Task 1/2/5/6), `isTracingSupport()` on `ConnectionDescription` (Task 3/4), `isTracingSupported()`/`tracingSupported(...)` on `MessageSettings` (Task 4/5/6), `PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT` (Task 5) — names are used consistently across tasks. From c23e2c964e4d9a68d00c354db2773a2c31d194f7 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Thu, 4 Jun 2026 15:34:15 +0100 Subject: [PATCH 03/48] POC WIP --- .../2026-06-03-otel-span-linkage-followup.md | 80 +++ .../plans/2026-06-02-otel-e2e-jaeger.md | 507 ++++++++++++++++++ docs/superpowers/runbooks/otel-opmsg-e2e.md | 37 ++ .../2026-06-02-otel-e2e-jaeger-design.md | 218 ++++++++ driver-core/build.gradle.kts | 6 + .../connection/ConnectionDescription.java | 50 +- .../internal/connection/CommandMessage.java | 37 +- .../connection/DescriptionHelper.java | 5 + .../internal/connection/MessageSettings.java | 12 + .../internal/connection/ProtocolHelper.java | 1 + .../micrometer/MicrometerTracer.java | 24 + .../OtelTracePropagationTestToggle.java | 32 ++ .../micrometer/TraceContext.java | 14 + .../CommandMessageOtelTraceContextTest.java | 207 +++++++ .../DescriptionHelperTracingTest.java | 81 +++ .../micrometer/MicrometerTraceParentTest.java | 73 +++ gradle/libs.versions.toml | 1 + 17 files changed, 1380 insertions(+), 5 deletions(-) create mode 100644 docs/superpowers/followups/2026-06-03-otel-span-linkage-followup.md create mode 100644 docs/superpowers/plans/2026-06-02-otel-e2e-jaeger.md create mode 100644 docs/superpowers/runbooks/otel-opmsg-e2e.md create mode 100644 docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md create mode 100644 driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java create mode 100644 driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java create mode 100644 driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java create mode 100644 driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java diff --git a/docs/superpowers/followups/2026-06-03-otel-span-linkage-followup.md b/docs/superpowers/followups/2026-06-03-otel-span-linkage-followup.md new file mode 100644 index 00000000000..a1fd6d426e6 --- /dev/null +++ b/docs/superpowers/followups/2026-06-03-otel-span-linkage-followup.md @@ -0,0 +1,80 @@ +# Follow-up: server span links to the operation span, not the command span + +- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) +- **Status:** Known limitation of the POC — fix after the POC is validated. +- **Date:** 2026-06-03 +- **Related:** `docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md`, + `docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md` + +## Symptom (observed in Jaeger) + +For a single client operation, the exported trace nests as: + +``` +insert test.ping (client OPERATION span, id=4d5f339b) + ├─ insert (client COMMAND span, id=d49879c6, parent=4d5f339b) + └─ insert (mongod SERVER span, id=b89795e6, parent=4d5f339b) +``` + +The `mongod` server span is parented to the client **operation** span, making it a **sibling** of +the client **command** span. The expected/ideal hierarchy is +`operation span → command span → server span` (server span as a child of the command span). Sibling +spans render in Jaeger by start time, which is why the server span can appear "before" the command +span. + +## Root cause + +The OP_MSG trace-context section is written during message **encoding**, which happens **before** the +per-command span is created. At encode time the only span available in the `OperationContext` is the +operation span, so that is the context propagated on the wire. + +Code path (current `nabil_otel_context`): + +1. `InternalStreamConnection.sendAndReceiveInternal` (~line 445): `message.encode(bsonOutput, operationContext)` + runs first. `CommandMessage.writeOtelTraceContextSection` executes inside `encode` and reads + `operationContext.getTracingSpan()`. +2. `operationContext.getTracingSpan()` returns the **operation** span, set earlier by + `TracingManager.createOperationSpan` → `operationContext.setTracingSpan(span)` + (`TracingManager.java` ~line 284). +3. `InternalStreamConnection.sendAndReceiveInternal` (~lines 446–455): the **command** span is created + **after** encode via `createTracingSpan(...)`, parented to the operation span + (`TracingManager.java` ~lines 191–192: `addSpan(MONGODB_COMMAND, …, operationSpan.context())`). + +So the `traceparent` carries the operation span's id; the server attaches its span to the operation +span; the command span is a separate sibling under the same operation span. + +## Why the command span is the better parent + +The command span represents an individual wire RPC, and one operation can issue several commands +(retries, `getMore`, split bulk batches). Parenting each server span under the corresponding command +span ties it to the exact RPC that produced it. Under the operation span, multiple server spans pile +up as indistinguishable siblings. + +## Why it is not a trivial reorder (chicken-and-egg) + +`createTracingSpan` currently derives the command name from `message.getCommandDocument(bsonOutput)`, +i.e. it reads the **already-encoded** bytes. So the command span cannot simply be created before +`encode()` without changing how it obtains the command name. + +## Candidate fixes (after the POC) + +1. **Create the command span before `encode()` using the in-memory command name.** The command name + is available from `CommandMessage`'s in-memory command `BsonDocument` (its first key), without + re-parsing the encoded output. Create the command span first, then have + `writeOtelTraceContextSection` propagate the command span's context. Requires refactoring + `createTracingSpan` so the name/parent no longer depend on `getCommandDocument(bsonOutput)`. + *Preferred — keeps the section inside encoding and yields the correct parent.* + +2. **Append the section after the command span is created.** Move section-writing out of + `CommandMessage.writeOpMsg` into a post-`createTracingSpan` step in `InternalStreamConnection`, + appending the kind-3 section and re-backpatching the OP_MSG message length. More invasive to wire + framing; risks interacting with compression and `getCommandDocument` re-parsing. + +3. **Accept operation-span parenting for the POC.** The trace is still correctly linked end to end — + just one level shallower than ideal. No change. + +## Decision + +Defer. The POC's goal (end-to-end propagation visible in Jaeger) is met with operation-span +parenting. Revisit with option 1 when productionizing, alongside removing the test-only +`OtelTracePropagationTestToggle` and adding the real `hello` `tracingSupport` negotiation. diff --git a/docs/superpowers/plans/2026-06-02-otel-e2e-jaeger.md b/docs/superpowers/plans/2026-06-02-otel-e2e-jaeger.md new file mode 100644 index 00000000000..eb55898a016 --- /dev/null +++ b/docs/superpowers/plans/2026-06-02-otel-e2e-jaeger.md @@ -0,0 +1,507 @@ +# End-to-end OTel Trace Visualization (driver toggle + Spring Boot + Jaeger) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** See a single client→server trace in Jaeger: a Spring Boot app's MongoDB operation, the driver's propagated `traceparent` (forced on via a test toggle), and the POC server's child span, all exported to one Jaeger instance. + +**Architecture:** Add a test-only static toggle in the driver to bypass the server-capability gate; publish the driver to Maven Local; run Jaeger in Docker; reconnect the POC mongod to export OTLP to Jaeger; build a Spring Boot app (Spring Data MongoDB + Micrometer→OTel→Jaeger) that wires the driver's internal tracing via `MongoClientSettingsBuilderCustomizer`, flips the toggle, and exposes a REST trigger. + +**Tech Stack:** Java 8 driver-core (toggle), Gradle (publish), Docker (Jaeger + mongod), Spring Boot 3.3.x / Java 17 / Maven (app), Micrometer Tracing + OpenTelemetry OTLP. + +**Spec:** `docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md` + +> **Standing instruction:** the user wants driver changes STAGED, not committed. For driver tasks, replace any `git commit` with `git add` (stage only). The Spring Boot app lives OUTSIDE the driver repo (`~/MongoDB/otel-poc-server/otel-poc-client/`) and is not added to driver git at all. + +> **Environment facts (already true):** +> - POC mongod binaries extracted at `~/MongoDB/otel-poc-server/dist-test/`; Docker image `otel-poc-mongod` built; container `otel-poc-mongod-run` currently running on the default bridge with port 27017. +> - `featureFlagTracing` is enabled in that build; `opentelemetryHttpEndpoint` is a startup setParameter. +> - Driver OP_MSG kind-3 propagation POC is already staged (gate lives in `CommandMessage.writeOtelTraceContextSection`). + +--- + +## File map + +| File | Responsibility | Change | +|---|---|---| +| `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java` | Test-only force switch | Create | +| `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` | OP_MSG encoding gate | Modify (one clause) | +| `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` | Toggle behavior test | Modify (add 1 test) | +| `~/MongoDB/otel-poc-server/otel-poc-client/pom.xml` | App build, driver override, mavenLocal | Create | +| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/OtelPocApplication.java` | Main + toggle activation | Create | +| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/MongoTracingConfig.java` | `MongoClientSettingsBuilderCustomizer` wiring | Create | +| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/PingController.java` | REST trigger | Create | +| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/resources/application.properties` | URI, sampling, OTLP endpoint | Create | + +--- + +## Task 1: Driver test-only force toggle + +**Files:** +- Create: `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java` +- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` +- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` + +- [ ] **Step 1: Write the failing test** + +Add this method to `CommandMessageOtelTraceContextTest` (it reuses the existing `buildCommandMessage`, `buildOperationContext`, `encodeToBytes`, `containsOtelSection`, and `TRACEPARENT` helpers/constants already in the class). Add the import `import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle;` at the top. + +```java + @Test + void writesSectionWhenForcedEvenIfCapabilityAbsent() { + OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; + try { + CommandMessage message = buildCommandMessage(false); // server did NOT advertise tracingSupport + TraceContext traceContext = () -> TRACEPARENT; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertTrue(containsOtelSection(encoded, TRACEPARENT), + "With FORCE_PROPAGATION the section must be sent even when the server did not advertise support"); + } finally { + OtelTracePropagationTestToggle.FORCE_PROPAGATION = false; + } + } +``` + +- [ ] **Step 2: Run the test to verify it fails to compile** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest.writesSectionWhenForcedEvenIfCapabilityAbsent" -PskipCryptVerify=true` +Expected: FAIL — `OtelTracePropagationTestToggle` does not exist. + +- [ ] **Step 3: Create the toggle class** + +Create `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java`: + +```java +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.observability.micrometer; + +/** + * TEST-ONLY switch (DRIVERS-3454): when {@code true}, the driver writes the OP_MSG OpenTelemetry + * trace-context section even if the server did not advertise {@code tracingSupport} in its + * {@code hello} response. The sampled-{@code traceparent} requirement still applies. + * + *

This exists only to exercise end-to-end propagation against a server that does not yet advertise + * the capability. Remove before any production use.

+ */ +public final class OtelTracePropagationTestToggle { + public static volatile boolean FORCE_PROPAGATION = false; + + private OtelTracePropagationTestToggle() { + } +} +``` + +- [ ] **Step 4: Relax the capability gate in CommandMessage** + +In `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java`, add the import (alphabetically within the `com.mongodb.internal.observability.micrometer` group, next to the existing `Span` import): + +```java +import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; +``` + +In `writeOtelTraceContextSection`, change the first guard from: + +```java + if (!getSettings().isTracingSupported()) { + return; + } +``` + +to: + +```java + if (!getSettings().isTracingSupported() && !OtelTracePropagationTestToggle.FORCE_PROPAGATION) { + return; + } +``` + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest" -PskipCryptVerify=true` +Expected: PASS (all cases, including the new one and the existing `omitsSectionWhenCapabilityAbsent` which does NOT set the toggle, so it still passes). + +- [ ] **Step 6: Stage (do NOT commit)** + +```bash +git add driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java \ + driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java \ + driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +``` + +--- + +## Task 2: Publish the driver to Maven Local + +**Files:** none (build action) + +- [ ] **Step 1: Publish all modules as 5.9.0-SNAPSHOT** + +Run from the repo root: +```bash +./gradlew publishToMavenLocal -PskipCryptVerify=true +``` +Expected: BUILD SUCCESSFUL. + +- [ ] **Step 2: Verify the artifacts landed in the local repo** + +Run: +```bash +ls ~/.m2/repository/org/mongodb/mongodb-driver-sync/5.9.0-SNAPSHOT/ \ + ~/.m2/repository/org/mongodb/mongodb-driver-core/5.9.0-SNAPSHOT/ \ + ~/.m2/repository/org/mongodb/bson/5.9.0-SNAPSHOT/ +``` +Expected: each directory contains a `*-5.9.0-SNAPSHOT.jar`. If `bson-record-codec` is referenced transitively, confirm it too: +```bash +ls ~/.m2/repository/org/mongodb/bson-record-codec/5.9.0-SNAPSHOT/ 2>/dev/null || echo "(bson-record-codec not published — fine unless the app fails to resolve it)" +``` + +--- + +## Task 3: Start Jaeger in Docker + +**Files:** none (infra) + +- [ ] **Step 1: Create the shared network and run Jaeger** + +```bash +docker network create otel-poc-net 2>/dev/null || true +docker rm -f jaeger 2>/dev/null || true +docker run -d --name jaeger --network otel-poc-net \ + -e COLLECTOR_OTLP_ENABLED=true \ + -p 16686:16686 -p 4317:4317 -p 4318:4318 \ + jaegertracing/all-in-one:1.62.0 +``` + +- [ ] **Step 2: Verify the UI is up** + +```bash +until curl -sf http://localhost:16686/ >/dev/null; do sleep 1; done; echo "Jaeger UI reachable" +``` +Expected: `Jaeger UI reachable`. + +--- + +## Task 4: Reconnect mongod to export OTLP to Jaeger + +**Files:** none (infra) + +- [ ] **Step 1: Recreate the mongod container on the shared network with the OTLP endpoint** + +```bash +cd ~/MongoDB/otel-poc-server +docker rm -f otel-poc-mongod-run 2>/dev/null || true +docker run -d --name otel-poc-mongod-run --platform linux/arm64 --network otel-poc-net \ + -v "$PWD/dist-test:/opt/mongo:ro" \ + -v "$PWD/data:/data/db" \ + -p 27017:27017 \ + otel-poc-mongod \ + /opt/mongo/bin/mongod --dbpath /data/db --bind_ip_all --port 27017 \ + --setParameter opentelemetryHttpEndpoint=http://jaeger:4318/v1/traces \ + --setParameter openTelemetryExportIntervalMillis=1000 +``` + +- [ ] **Step 2: Wait for readiness and confirm the endpoint was accepted** + +```bash +until docker logs otel-poc-mongod-run 2>&1 | grep -q "Waiting for connections" || ! docker ps -q --filter name=otel-poc-mongod-run | grep -q .; do sleep 1; done +docker ps --filter name=otel-poc-mongod-run --format '{{.Status}}' +docker logs otel-poc-mongod-run 2>&1 | grep -iE "opentelemetry|otel|trace|export|endpoint" | tail -15 +``` +Expected: container `Up`; no fatal OTLP/endpoint parse error in logs. + +- [ ] **Step 3: Fallback if the endpoint form is rejected** + +If mongod refuses to start or logs an OTLP endpoint error, re-run Step 1 replacing the endpoint with the file exporter (already proven working): +``` + --setParameter opentelemetryTraceDirectory=/data/db/otel-traces +``` +and note in the final report that server spans are inspected as JSONL under `~/MongoDB/otel-poc-server/data/otel-traces/` rather than in the Jaeger UI. Then continue. + +--- + +## Task 5: Scaffold the Spring Boot app (pom + properties + main) + +**Files (all under `~/MongoDB/otel-poc-server/otel-poc-client/`, OUTSIDE the driver repo):** +- Create: `pom.xml` +- Create: `src/main/resources/application.properties` +- Create: `src/main/java/com/example/otelpoc/OtelPocApplication.java` + +- [ ] **Step 1: Create the project directory and Maven wrapper** + +```bash +mkdir -p ~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc +mkdir -p ~/MongoDB/otel-poc-server/otel-poc-client/src/main/resources +``` + +- [ ] **Step 2: Create `pom.xml`** + +```xml + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 3.3.5 + + + + com.example + otel-poc-client + 0.0.1-SNAPSHOT + + + 17 + + 5.9.0-SNAPSHOT + + + + + org.springframework.boot + spring-boot-starter-web + + + org.springframework.boot + spring-boot-starter-data-mongodb + + + org.springframework.boot + spring-boot-starter-actuator + + + io.micrometer + micrometer-tracing-bridge-otel + + + io.opentelemetry + opentelemetry-exporter-otlp + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + + +``` + +> Maven consults the local `~/.m2` repository by default, so the `5.9.0-SNAPSHOT` artifacts published in Task 2 resolve without any extra `` entry. + +- [ ] **Step 3: Create `src/main/resources/application.properties`** + +```properties +spring.application.name=otel-poc-client +server.port=8080 +spring.data.mongodb.uri=mongodb://localhost:27017/test +management.tracing.sampling.probability=1.0 +management.otlp.tracing.endpoint=http://localhost:4318/v1/traces +# Suppress Spring's auto Mongo command instrumentation: the driver's internal tracer is the ONLY +# source of Mongo command spans (no duplicate/competing spans). +management.metrics.enable.mongodb=false +spring.autoconfigure.exclude=org.springframework.boot.actuate.autoconfigure.metrics.mongo.MongoMetricsAutoConfiguration +``` + +- [ ] **Step 4: Create the main class with toggle activation** + +`src/main/java/com/example/otelpoc/OtelPocApplication.java`: + +```java +package com.example.otelpoc; + +import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; + +@SpringBootApplication +public class OtelPocApplication { + public static void main(String[] args) { + // TEST-ONLY: force the driver to send the OP_MSG trace-context section even though this POC + // server does not advertise tracingSupport in hello. Set before any MongoClient is used. + OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; + SpringApplication.run(OtelPocApplication.class, args); + } +} +``` + +- [ ] **Step 5: Generate the Maven wrapper** + +```bash +cd ~/MongoDB/otel-poc-server/otel-poc-client +mvn -N wrapper:wrapper -Dmaven=3.9.9 2>&1 | tail -3 || echo "(if 'mvn' is absent, install it or run with a system Maven in Task 7)" +``` +Expected: `mvnw` and `.mvn/` created. If no system `mvn` exists, skip — Task 7 notes the alternative. + +--- + +## Task 6: Tracing wiring bean + REST trigger + +**Files (under `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/`):** +- Create: `MongoTracingConfig.java` +- Create: `PingController.java` + +- [ ] **Step 1: Create the `MongoClientSettingsBuilderCustomizer` wiring** + +`MongoTracingConfig.java`: + +```java +package com.example.otelpoc; + +import com.mongodb.observability.micrometer.MicrometerObservabilitySettings; +import io.micrometer.observation.ObservationRegistry; +import org.springframework.boot.autoconfigure.mongo.MongoClientSettingsBuilderCustomizer; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class MongoTracingConfig { + + /** + * Activates the driver's INTERNAL tracing (the path that sets operationContext.tracingSpan, which + * traceParent() reads for OP_MSG propagation) and routes its spans through Spring's OTel-bridged + * ObservationRegistry so they export to Jaeger. + */ + @Bean + public MongoClientSettingsBuilderCustomizer tracingCustomizer(final ObservationRegistry registry) { + return builder -> builder.observabilitySettings( + MicrometerObservabilitySettings.builder() + .observationRegistry(registry) + .build()); + } +} +``` + +- [ ] **Step 2: Create the REST trigger** + +`PingController.java`: + +```java +package com.example.otelpoc; + +import org.bson.Document; +import org.springframework.data.mongodb.core.MongoTemplate; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.RestController; + +import java.util.Date; + +@RestController +public class PingController { + + private final MongoTemplate mongoTemplate; + + public PingController(final MongoTemplate mongoTemplate) { + this.mongoTemplate = mongoTemplate; + } + + @GetMapping("/ping") + public String ping() { + mongoTemplate.getCollection("ping").insertOne(new Document("at", new Date())); + long count = mongoTemplate.getCollection("ping").countDocuments(); + return "ok, count=" + count + "\n"; + } +} +``` + +- [ ] **Step 3: Compile the app to verify wiring resolves** + +```bash +cd ~/MongoDB/otel-poc-server/otel-poc-client +./mvnw -q -DskipTests compile 2>&1 | tail -20 || mvn -q -DskipTests compile 2>&1 | tail -20 +``` +Expected: BUILD SUCCESS. If compilation fails to resolve `MicrometerObservabilitySettings` or `OtelTracePropagationTestToggle`, confirm Task 2 published `mongodb-driver-core` (the toggle and settings live there) and that `mongodb.version` is `5.9.0-SNAPSHOT`. + +--- + +## Task 7: End-to-end run and Jaeger verification + +**Files:** none (run + verify). Prereqs: Tasks 1–6 done; Jaeger (Task 3) and mongod-on-network (Task 4) running. + +- [ ] **Step 1: Start the Spring Boot app** + +```bash +cd ~/MongoDB/otel-poc-server/otel-poc-client +./mvnw spring-boot:run 2>&1 | tee /tmp/otel-poc-client.log & +# wait until it is listening on 8080 +until curl -sf http://localhost:8080/ping >/dev/null 2>&1; do sleep 2; done +echo "app up" +``` +(If there is no `mvnw`, use `mvn spring-boot:run`.) Expected: `app up`. + +- [ ] **Step 2: Generate a traced operation** + +```bash +curl -s http://localhost:8080/ping +``` +Expected: `ok, count=`. + +- [ ] **Step 3: Confirm the client exported a trace to Jaeger** + +```bash +sleep 3 +curl -s "http://localhost:16686/api/services" | tr ',' '\n' | grep -i "otel-poc-client" && echo "client service present in Jaeger" +``` +Expected: `otel-poc-client` appears in the services list. + +- [ ] **Step 4: Confirm a linked server span exists in the same trace** + +```bash +TRACE_JSON=$(curl -s "http://localhost:16686/api/traces?service=otel-poc-client&limit=1") +echo "$TRACE_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); \ +spans=d['data'][0]['spans']; procs=d['data'][0]['processes']; \ +print('services in trace:', sorted({procs[s['processID']]['serviceName'] for s in spans})); \ +print('span names:', [s['operationName'] for s in spans])" +``` +Expected: the services set includes BOTH `otel-poc-client` and the mongod service (e.g. `mongod`), proving the client span and the server span share one trace. + +- [ ] **Step 5: Visual confirmation** + +Open **http://localhost:16686**, select service `otel-poc-client`, open the most recent trace. Confirm a span tree: HTTP `GET /ping` → driver mongo command span → `mongod` server span (same traceId; server span's parent is the client mongo span). + +- [ ] **Step 6: Negative check (capability gate still works)** + +Stop the app, set the toggle off by editing `OtelPocApplication.main` to `FORCE_PROPAGATION = false` (or comment the line), `./mvnw spring-boot:run` again, `curl /ping`, and confirm in Jaeger the new client trace has **no** `mongod` span (server starts its own unrelated trace). Restore the line afterward. + +- [ ] **Step 7: Record the result** + +If server spans appear in Jaeger: success — note it. If Task 4 used the file-exporter fallback, instead show the server span and matching traceId from the latest `~/MongoDB/otel-poc-server/data/otel-traces/*.jsonl` (grep for the client's traceId) and note that server spans are file-based rather than in the Jaeger UI. + +--- + +## Self-Review + +**Spec coverage:** +- §3 toggle → Task 1 (class + gate + test). §4 publish → Task 2. §5 Jaeger → Task 3. §6 mongod OTLP (+ file fallback) → Task 4 (Steps 1–3). §7 app: deps/version override → Task 5 Step 2; properties → Task 5 Step 3; toggle activation → Task 5 Step 4; `MongoClientSettingsBuilderCustomizer` wiring → Task 6 Step 1; REST trigger → Task 6 Step 2. §8 run & verification → Task 7. §10 risks: OTLP endpoint form → Task 4 Step 3 fallback; duplicate Spring command spans → acceptable, see note below; driver-version override → Task 5 Step 2 + Task 6 Step 3 troubleshooting. + +**Duplicate-span suppression (spec §7, firm):** Spring's auto Mongo command instrumentation is suppressed in `application.properties` (Task 5 Step 3) via `spring.autoconfigure.exclude=…MongoMetricsAutoConfiguration` + `management.metrics.enable.mongodb=false`, so the driver's internal tracer is the only source of Mongo command spans. If startup logs show the excluded class still contributing (version drift), the implementer confirms the exact actuator Mongo autoconfig class for Spring Boot 3.3.5 and excludes that instead. + +**Placeholder scan:** no TBD/TODO; every code/file step has full content; the only conditional is the documented OTLP-endpoint fallback (Task 4 Step 3) and the no-`mvnw` alternative (system `mvn`). + +**Type/name consistency:** `OtelTracePropagationTestToggle.FORCE_PROPAGATION` (Tasks 1, 5); `MicrometerObservabilitySettings.builder().observationRegistry(...)` and `.observabilitySettings(...)` (Task 6, matches driver API verified in spec §7); package `com.example.otelpoc` and class names match the file map; `mongodb.version=5.9.0-SNAPSHOT` consistent (Tasks 2, 5, 6). + +**Staging vs commit:** Task 1 stages only (driver repo); Tasks 5–6 create files outside the driver repo (never added to driver git). diff --git a/docs/superpowers/runbooks/otel-opmsg-e2e.md b/docs/superpowers/runbooks/otel-opmsg-e2e.md new file mode 100644 index 00000000000..ac41c43ba79 --- /dev/null +++ b/docs/superpowers/runbooks/otel-opmsg-e2e.md @@ -0,0 +1,37 @@ +# Runbook: OTel OP_MSG propagation end-to-end (manual) + +Validates DRIVERS-3454 end to end: a sync-driver client span linked to a server child span. + +## Prerequisites +- A local build of the server POC branch (10gen/mongo PR #49930), which accepts OP_MSG + section kind 3 and starts a server span from it. NOTE: the POC does NOT yet advertise + `tracingSupport` in hello. Until SERVER-107128 adds it, temporarily force the driver + capability on for this manual run (see step 3). +- An OpenTelemetry collector / exporter the server POC is configured to export traces to + (e.g. Jaeger via OTLP), plus a Micrometer Tracing OTel bridge on the client. + +## Steps +1. Build & run the server POC `mongod` with tracing enabled and sampling at 100%. +2. Start a Jaeger all-in-one (or OTLP collector) and point both server and client exporters at it. +3. In a scratch sync-driver program, configure a `MongoClient` with an `ObservationRegistry` + that has an OTel-backed `DefaultTracingObservationHandler` (so `traceParent()` is non-null + and sampled). Run a `find`. + - Temporary capability override for the run (POC server lacks the hello flag): start an + OTel root span yourself, then run the command. Because the server POC accepts kind 3 + unconditionally, set the driver `tracingSupported` to true by connecting to a server + whose hello you patch to include `tracingSupport: true`, OR temporarily hardcode + `isTracingSupport()`/`tracingSupported` to `true` on the local branch for the manual run + only (revert before staging). +4. In Jaeger, confirm a single trace contains BOTH the client `find` span and a server span, + with the server span's parent = the client span id sent in the traceparent. + +## Pass criteria +- One trace, two+ spans, correct parent/child linkage across the client→server boundary. +- With the driver capability off (default), no server span is created (negative check). + +## Notes +- This proves the wire format end to end; the automated Phase 1 test + (`CommandMessageOtelTraceContextTest`) is the regression guard. In particular, + `getCommandDocumentIgnoresOtelSection` guards against the trailing kind-3 section + corrupting command-document reconstruction on the send path. +- Findings (any format mismatch, tracestate handling, flags) feed back into the spec. diff --git a/docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md b/docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md new file mode 100644 index 00000000000..7bd19fe5562 --- /dev/null +++ b/docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md @@ -0,0 +1,218 @@ +# Design: End-to-end OTel trace visualization (driver toggle + Spring Boot + Jaeger) + +- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) — trace context propagation to the server +- **Status:** Design +- **Author:** Nabil Hachicha +- **Date:** 2026-06-02 +- **Builds on:** `docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md` (the OP_MSG kind-3 POC, already staged) and the running server POC (`~/MongoDB/otel-poc-server/`, mongod `gitVersion b2cb2bf`). + +--- + +## 1. Goal + +Make a client→server trace **visible end to end in Jaeger**: a Spring Boot app issues a MongoDB +operation; the driver propagates its sampled W3C `traceparent` to the server via the OP_MSG kind-3 +section; the server starts a child span; both client and server spans export to one Jaeger instance +and appear under a single trace. + +The POC server does **not** advertise `tracingSupport` in `hello`, so the driver (which gates on +that capability) needs a temporary, removable switch to send the section anyway. + +## 2. Components + +1. **Driver test-only force toggle** — bypasses the server-capability gate only. +2. **Local Maven publish** — `5.9.0-SNAPSHOT` consumable by the app. +3. **Jaeger** (Docker) — single OTLP collector + UI. +4. **mongod re-config** — export server spans to Jaeger. +5. **Spring Boot app** (host JVM) — Spring Data MongoDB + Micrometer→OTel→Jaeger, wired to the + driver's internal tracing, with a REST trigger. + +--- + +## 3. Driver: force-propagation toggle + +New file `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java`: + +```java +public final class OtelTracePropagationTestToggle { + // TEST-ONLY (DRIVERS-3454): force sending the OP_MSG OTel trace-context section even when the + // server did not advertise tracingSupport in hello. Remove before any production use. + public static volatile boolean FORCE_PROPAGATION = false; + + private OtelTracePropagationTestToggle() { + } +} +``` + +`CommandMessage.writeOtelTraceContextSection` gate changes from: + +```java +if (!getSettings().isTracingSupported()) { + return; +} +``` + +to: + +```java +if (!getSettings().isTracingSupported() && !OtelTracePropagationTestToggle.FORCE_PROPAGATION) { + return; +} +``` + +The remaining gates are unchanged: a `null` tracing span or a `null`/unsampled `traceParent()` still +omits the section. So the toggle only relaxes the capability check. + +**Test** (`CommandMessageOtelTraceContextTest`): new case — `tracingSupported=false` + +`FORCE_PROPAGATION=true` + sampled span ⇒ section IS written. Set/reset `FORCE_PROPAGATION` in a +`try/finally` so the static does not leak into other tests. + +**Removal path:** delete the toggle class and revert the one `&&` clause. + +## 4. Publish to Maven Local + +From the repo root: + +```bash +./gradlew publishToMavenLocal -PskipCryptVerify=true +``` + +Publishes all modules (`bson`, `bson-record-codec`, `driver-core`, `driver-sync`, …) as +`org.mongodb:*:5.9.0-SNAPSHOT`. The app consumes `5.9.0-SNAPSHOT` from `mavenLocal()`. + +## 5. Jaeger (Docker) + +`jaegertracing/all-in-one:1.62.0` on a dedicated network `otel-poc-net`, OTLP enabled, host ports: + +| Port | Purpose | +|------|---------| +| 16686 | Jaeger UI | +| 4317 | OTLP gRPC | +| 4318 | OTLP HTTP | + +Run: + +```bash +docker network create otel-poc-net 2>/dev/null || true +docker run -d --name jaeger --network otel-poc-net \ + -e COLLECTOR_OTLP_ENABLED=true \ + -p 16686:16686 -p 4317:4317 -p 4318:4318 \ + jaegertracing/all-in-one:1.62.0 +``` + +## 6. mongod: export server spans to Jaeger + +Restart the existing POC mongod container **on `otel-poc-net`** so it can reach Jaeger by name, and +point its OTLP exporter at Jaeger: + +```bash +docker rm -f otel-poc-mongod-run +docker run -d --name otel-poc-mongod-run --platform linux/arm64 --network otel-poc-net \ + -v "$HOME/MongoDB/otel-poc-server/dist-test:/opt/mongo:ro" \ + -v "$HOME/MongoDB/otel-poc-server/data:/data/db" \ + -p 27017:27017 \ + otel-poc-mongod \ + /opt/mongo/bin/mongod --dbpath /data/db --bind_ip_all --port 27017 \ + --setParameter opentelemetryHttpEndpoint=http://jaeger:4318/v1/traces \ + --setParameter openTelemetryExportIntervalMillis=1000 +``` + +`featureFlagTracing` is already enabled in this build. **Verification & fallback:** confirm via +server logs that the OTLP endpoint is accepted and exports succeed; if the exact endpoint form is +rejected, fall back to the already-working file exporter +(`opentelemetryTraceDirectory=/data/db/otel-traces`) and note that server spans are then inspected as +JSONL rather than in the Jaeger UI. + +## 7. Spring Boot app (host JVM) + +Maven project at `~/MongoDB/otel-poc-server/otel-poc-client/` — kept outside the driver repo so it is +not entangled with the driver build. + +**Stack:** Spring Boot 3.3.x, Java 17, Maven wrapper. + +**Dependencies:** `spring-boot-starter-web`, `spring-boot-starter-data-mongodb`, +`spring-boot-starter-actuator`, `micrometer-tracing-bridge-otel`, `opentelemetry-exporter-otlp`. +`pom.xml` adds `mavenLocal()` (via a ``) and overrides `5.9.0-SNAPSHOT`. + +**Critical wiring bean** — activates the driver's *internal* tracing (the path that sets +`operationContext.tracingSpan`, which `traceParent()` reads): + +```java +@Bean +MongoClientSettingsBuilderCustomizer tracingCustomizer(ObservationRegistry registry) { + return builder -> builder.observabilitySettings( + MicrometerObservabilitySettings.builder() + .observationRegistry(registry) + .build()); +} +``` + +**Toggle activation** — set before any operation runs: + +```java +@PostConstruct +void enableForcedPropagation() { + OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; +} +``` + +**Trigger** — a `@RestController`: + +```java +@GetMapping("/ping") +String ping() { + mongoTemplate.getCollection("ping").insertOne(new Document("at", new Date())); + long n = mongoTemplate.getCollection("ping").countDocuments(); + return "ok, count=" + n; +} +``` + +Hitting `/ping` creates an HTTP server span (root, sampled), under which the driver creates a mongo +command span (exported to Jaeger; its context is the propagated `traceparent`), and the server +creates its child span (exported to Jaeger). + +**`application.properties`:** + +```properties +spring.application.name=otel-poc-client +spring.data.mongodb.uri=mongodb://localhost:27017/test +management.tracing.sampling.probability=1.0 +management.otlp.tracing.endpoint=http://localhost:4318/v1/traces +# Suppress Spring's auto Mongo command instrumentation so the driver's internal tracer is the +# ONLY source of Mongo command spans (avoids duplicate/competing spans). +management.metrics.enable.mongodb=false +spring.autoconfigure.exclude=org.springframework.boot.actuate.autoconfigure.metrics.mongo.MongoMetricsAutoConfiguration +``` + +**Suppressing Spring's auto Mongo instrumentation (firm decision).** Spring Boot's actuator +auto-registers a Mongo command listener via `MongoMetricsAutoConfiguration`. To guarantee the +driver's own internal tracer is the single source of Mongo command spans (and the one performing +propagation), we **exclude** `org.springframework.boot.actuate.autoconfigure.metrics.mongo.MongoMetricsAutoConfiguration` +and set `management.metrics.enable.mongodb=false`. Only our `MongoClientSettingsBuilderCustomizer` +(§7 wiring) then contributes Mongo observability. + +## 8. End-to-end run & verification + +1. Start Jaeger (§5). 2. Restart mongod on the network with OTLP (§6). 3. Publish the driver (§4). +4. `./mvnw spring-boot:run` (§7). 5. `curl localhost:8080/ping`. 6. Open **http://localhost:16686**, +select service `otel-poc-client`, open the latest trace. + +**Pass criteria:** one trace contains the HTTP span, the driver mongo command span, **and** a +`mongod` server span; the server span's `traceId` equals the client trace, and its parent is the +client mongo span's id. Negative check: with `FORCE_PROPAGATION=false`, the server span is absent +from the client trace (server starts its own unrelated trace, if any). + +## 9. Scope guards + +- Test-only toggle; not a public API; removed before any real merge. +- Single mongod (no replica set/sharding), `find`/`insert` only. +- No auth/TLS on the local mongod. +- The app lives outside the driver repo; it is not committed to the driver repo. + +## 10. Open questions / risks + +- Exact accepted form of `opentelemetryHttpEndpoint` (full `/v1/traces` URL vs base) — verify via + logs; file-exporter fallback documented (§6). +- (Resolved) Spring's auto Mongo instrumentation is suppressed via autoconfigure-exclude + metric disable (§7). +- Spring Boot 3.3.x pins an older driver; overriding `mongodb.version` to `5.9.0-SNAPSHOT` assumes API + compatibility (it is, the API is unchanged by this POC). diff --git a/driver-core/build.gradle.kts b/driver-core/build.gradle.kts index 047b3a43a63..5efda24e3a3 100644 --- a/driver-core/build.gradle.kts +++ b/driver-core/build.gradle.kts @@ -57,9 +57,15 @@ dependencies { optionalImplementation(platform(libs.micrometer.observation.bom)) optionalImplementation(libs.micrometer.observation) + optionalImplementation(libs.micrometer.tracing) testImplementation(project(path = ":bson", configuration = "testArtifacts")) testImplementation(libs.reflections) + + // Tracing testing + testImplementation(platform(libs.micrometer.tracing.integration.test.bom)) + testImplementation(libs.micrometer.tracing.integration.test) { exclude(group = "org.junit.jupiter") } + testImplementation(libs.netty.tcnative.boringssl.static) listOf("linux-x86_64", "linux-aarch_64", "osx-x86_64", "osx-aarch_64", "windows-x86_64").forEach { arch -> testImplementation("${libs.netty.tcnative.boringssl.static.get()}::$arch") diff --git a/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java b/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java index c8c213398b7..b62cc018595 100644 --- a/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java +++ b/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java @@ -48,6 +48,7 @@ public class ConnectionDescription { private final List compressors; private final BsonArray saslSupportedMechanisms; private final Integer logicalSessionTimeoutMinutes; + private final boolean tracingSupport; private static final int DEFAULT_MAX_MESSAGE_SIZE = 0x2000000; // 32MB private static final int DEFAULT_MAX_WRITE_BATCH_SIZE = 512; @@ -150,6 +151,15 @@ private ConnectionDescription(@Nullable final ObjectId serviceId, final Connecti final ServerType serverType, final int maxBatchCount, final int maxDocumentSize, final int maxMessageSize, final List compressors, @Nullable final BsonArray saslSupportedMechanisms, @Nullable final Integer logicalSessionTimeoutMinutes) { + this(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, maxMessageSize, compressors, + saslSupportedMechanisms, logicalSessionTimeoutMinutes, false); + } + + private ConnectionDescription(@Nullable final ObjectId serviceId, final ConnectionId connectionId, final int maxWireVersion, + final ServerType serverType, final int maxBatchCount, final int maxDocumentSize, + final int maxMessageSize, final List compressors, + @Nullable final BsonArray saslSupportedMechanisms, @Nullable final Integer logicalSessionTimeoutMinutes, + final boolean tracingSupport) { this.serviceId = serviceId; this.connectionId = connectionId; this.serverType = serverType; @@ -160,6 +170,7 @@ private ConnectionDescription(@Nullable final ObjectId serviceId, final Connecti this.compressors = notNull("compressors", Collections.unmodifiableList(new ArrayList<>(compressors))); this.saslSupportedMechanisms = saslSupportedMechanisms; this.logicalSessionTimeoutMinutes = logicalSessionTimeoutMinutes; + this.tracingSupport = tracingSupport; } /** * Creates a new connection description with the set connection id @@ -171,7 +182,7 @@ private ConnectionDescription(@Nullable final ObjectId serviceId, final Connecti public ConnectionDescription withConnectionId(final ConnectionId connectionId) { notNull("connectionId", connectionId); return new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, - maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes); + maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes, tracingSupport); } /** @@ -184,7 +195,22 @@ public ConnectionDescription withConnectionId(final ConnectionId connectionId) { public ConnectionDescription withServiceId(final ObjectId serviceId) { notNull("serviceId", serviceId); return new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, - maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes); + maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes, tracingSupport); + } + + /** + * Creates a new connection description with the given tracing support flag. + * + *

A value of {@code true} indicates that the server advertised OpenTelemetry trace-context support in its {@code hello} response + * and that the driver may send trace context to the server over OP_MSG.

+ * + * @param tracingSupport whether the server supports OpenTelemetry trace-context propagation + * @return the new connection description + * @since 5.9 + */ + public ConnectionDescription withTracingSupport(final boolean tracingSupport) { + return new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, + maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes, tracingSupport); } /** @@ -293,6 +319,21 @@ public BsonArray getSaslSupportedMechanisms() { public Integer getLogicalSessionTimeoutMinutes() { return logicalSessionTimeoutMinutes; } + + /** + * Returns whether the server advertised OpenTelemetry trace-context support in its {@code hello} response. + * + *

When {@code true}, the driver may propagate trace context to the server over OP_MSG. + * When {@code false} (the default), the server did not advertise this capability and sending unknown OP_MSG sections would cause + * the server to reject the message.

+ * + * @return {@code true} if the server supports OpenTelemetry trace-context propagation + * @since 5.9 + */ + public boolean isTracingSupported() { + return tracingSupport; + } + /** * Get the default maximum message size. * @@ -350,6 +391,9 @@ public boolean equals(final Object o) { if (!Objects.equals(logicalSessionTimeoutMinutes, that.logicalSessionTimeoutMinutes)) { return false; } + if (tracingSupport != that.tracingSupport) { + return false; + } return Objects.equals(saslSupportedMechanisms, that.saslSupportedMechanisms); } @@ -365,6 +409,7 @@ public int hashCode() { result = 31 * result + (serviceId != null ? serviceId.hashCode() : 0); result = 31 * result + (saslSupportedMechanisms != null ? saslSupportedMechanisms.hashCode() : 0); result = 31 * result + (logicalSessionTimeoutMinutes != null ? logicalSessionTimeoutMinutes.hashCode() : 0); + result = 31 * result + (tracingSupport ? 1 : 0); return result; } @@ -380,6 +425,7 @@ public String toString() { + ", compressors=" + compressors + ", logicialSessionTimeoutMinutes=" + logicalSessionTimeoutMinutes + ", serviceId=" + serviceId + + ", tracingSupport=" + tracingSupport + '}'; } } diff --git a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java index 348349fd18c..d3d779ac984 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +++ b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java @@ -25,6 +25,8 @@ import com.mongodb.internal.MongoNamespaceHelper; import com.mongodb.internal.TimeoutContext; import com.mongodb.internal.connection.MessageSequences.EmptyMessageSequences; +import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; +import com.mongodb.internal.observability.micrometer.Span; import com.mongodb.internal.session.SessionContext; import com.mongodb.lang.Nullable; import org.bson.BsonArray; @@ -80,6 +82,12 @@ public final class CommandMessage extends RequestMessage { * Specifies that the `OP_MSG` section payload is a sequence of BSON documents. */ private static final byte PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE = 1; + /** + * Specifies that the `OP_MSG` section payload is a W3C traceparent C-string (OpenTelemetry trace context). + * + *

Mirrors the server's {@code kOtelTelemetryContext = 3} section kind (see DRIVERS-3454).

+ */ + private static final byte PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT = 3; private static final int UNINITIALIZED_POSITION = -1; @@ -156,15 +164,20 @@ BsonDocument getCommandDocument(final ByteBufferBsonOutput bsonOutput) { byteBuf.position(firstDocumentPosition); ByteBufBsonDocument byteBufBsonDocument = createOne(byteBuf); - // If true, it means there is at least one `PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE` section in the OP_MSG + // If true, there are more sections after the `PAYLOAD_TYPE_0_DOCUMENT` section: either one or more + // `PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE` sections, and/or a trailing `PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT` section. if (byteBuf.hasRemaining()) { BsonDocument commandBsonDocument = byteBufBsonDocument.toBaseBsonDocument(); // Each loop iteration processes one Document Sequence // When there are no more bytes remaining, there are no more Document Sequences while (byteBuf.hasRemaining()) { - // skip reading the payload type, we know it is `PAYLOAD_TYPE_1` - byteBuf.position(byteBuf.position() + 1); + byte payloadType = byteBuf.get(); + // Document-sequence sections always precede any trailing non-sequence section (e.g. + // PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT), and such sections carry no command-document fields, so stop here. + if (payloadType != PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE) { + break; + } int sequenceStart = byteBuf.position(); int sequenceSizeInBytes = byteBuf.getInt(); int sectionEnd = sequenceStart + sequenceSizeInBytes; @@ -277,11 +290,29 @@ private int writeOpMsg(final ByteBufferBsonOutput bsonOutput, final OperationCon fail(sequences.toString()); } + writeOtelTraceContextSection(bsonOutput, operationContext); + // Write the flag bits bsonOutput.writeInt32(flagPosition, getOpMsgFlagBits()); return commandStartPosition; } + private void writeOtelTraceContextSection(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext) { + if (!getSettings().isTracingSupported() && !OtelTracePropagationTestToggle.FORCE_PROPAGATION) { + return; + } + Span tracingSpan = operationContext.getTracingSpan(); + if (tracingSpan == null) { + return; + } + String traceParent = tracingSpan.context().traceParent(); + if (traceParent == null) { + return; + } + bsonOutput.writeByte(PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT); + bsonOutput.writeCString(traceParent); + } + private int writeOpQuery(final ByteBufferBsonOutput bsonOutput) { bsonOutput.writeInt32(0); bsonOutput.writeCString(new MongoNamespace(getDatabase(), MongoNamespaceHelper.COMMAND_COLLECTION_NAME).getFullName()); diff --git a/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java b/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java index 26f73bcee9c..a070d8bbdf4 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java +++ b/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java @@ -69,6 +69,7 @@ static ConnectionDescription createConnectionDescription(final ClusterConnection getMaxWireVersion(helloResult), getServerType(helloResult), getMaxWriteBatchSize(helloResult), getMaxBsonObjectSize(helloResult), getMaxMessageSizeBytes(helloResult), getCompressors(helloResult), helloResult.getArray("saslSupportedMechs", null), getLogicalSessionTimeoutMinutes(helloResult)); + connectionDescription = connectionDescription.withTracingSupport(getTracingSupport(helloResult)); if (helloResult.containsKey("connectionId")) { ConnectionId newConnectionId = connectionDescription.getConnectionId().withServerValue(helloResult.getNumber("connectionId").longValue()); @@ -170,6 +171,10 @@ private static Integer getLogicalSessionTimeoutMinutes(final BsonDocument helloR ? helloResult.getNumber("logicalSessionTimeoutMinutes").intValue() : null; } + private static boolean getTracingSupport(final BsonDocument helloResult) { + return helloResult.getBoolean("tracingSupport", BsonBoolean.FALSE).getValue(); + } + @Nullable private static String getString(final BsonDocument response, final String key) { if (response.containsKey(key)) { diff --git a/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java b/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java index 51587e8f91d..6eaad0ea379 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java +++ b/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java @@ -58,6 +58,7 @@ public final class MessageSettings { private final int maxWireVersion; private final ServerType serverType; private final boolean sessionSupported; + private final boolean tracingSupported; private final boolean cryptd; /** @@ -80,6 +81,7 @@ public static final class Builder { private int maxWireVersion = UNKNOWN_WIRE_VERSION; private ServerType serverType; private boolean sessionSupported; + private boolean tracingSupported; private boolean cryptd; /** @@ -139,6 +141,11 @@ public Builder sessionSupported(final boolean sessionSupported) { return this; } + public Builder tracingSupported(final boolean tracingSupported) { + this.tracingSupported = tracingSupported; + return this; + } + /** * Set whether the server is a mongocryptd. * @@ -193,6 +200,10 @@ public boolean isSessionSupported() { return sessionSupported; } + public boolean isTracingSupported() { + return tracingSupported; + } + private MessageSettings(final Builder builder) { this.maxDocumentSize = builder.maxDocumentSize; @@ -201,6 +212,7 @@ private MessageSettings(final Builder builder) { this.maxWireVersion = builder.maxWireVersion; this.serverType = builder.serverType; this.sessionSupported = builder.sessionSupported; + this.tracingSupported = builder.tracingSupported; this.cryptd = builder.cryptd; } } diff --git a/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java b/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java index c6ad5f451a0..a519252dfed 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java +++ b/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java @@ -235,6 +235,7 @@ static MessageSettings getMessageSettings(final ConnectionDescription connection .maxWireVersion(connectionDescription.getMaxWireVersion()) .serverType(connectionDescription.getServerType()) .sessionSupported(connectionDescription.getLogicalSessionTimeoutMinutes() != null) + .tracingSupported(connectionDescription.isTracingSupported()) .cryptd(serverDescription.isCryptd()) .build(); } diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java index d0d306de2c4..a75bdaa0140 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java @@ -24,6 +24,7 @@ import io.micrometer.observation.Observation; import io.micrometer.observation.ObservationConvention; import io.micrometer.observation.ObservationRegistry; +import io.micrometer.tracing.handler.TracingObservationHandler; import org.bson.BsonDocument; import org.bson.BsonReader; import org.bson.json.JsonMode; @@ -116,6 +117,29 @@ private static class MicrometerTraceContext implements TraceContext { MicrometerTraceContext(@Nullable final Observation observation) { this.observation = observation; } + + @Override + @Nullable + public String traceParent() { + if (observation == null) { + return null; + } + TracingObservationHandler.TracingContext tracingContext = + observation.getContextView().get(TracingObservationHandler.TracingContext.class); + if (tracingContext == null || tracingContext.getSpan() == null) { + return null; + } + // Fully qualified to avoid a name clash with this package's own TraceContext interface. + io.micrometer.tracing.TraceContext ctx = tracingContext.getSpan().context(); + if (ctx == null || ctx.traceId() == null || ctx.spanId() == null) { + return null; + } + Boolean sampled = ctx.sampled(); + if (sampled == null || !sampled) { + return null; + } + return "00-" + ctx.traceId() + "-" + ctx.spanId() + "-01"; + } } /** diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java new file mode 100644 index 00000000000..96d5b767121 --- /dev/null +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java @@ -0,0 +1,32 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.observability.micrometer; + +/** + * TEST-ONLY switch (DRIVERS-3454): when {@code true}, the driver writes the OP_MSG OpenTelemetry + * trace-context section even if the server did not advertise {@code tracingSupport} in its + * {@code hello} response. The sampled-{@code traceparent} requirement still applies. + * + *

This exists only to exercise end-to-end propagation against a server that does not yet advertise + * the capability. Remove before any production use.

+ */ +public final class OtelTracePropagationTestToggle { + public static volatile boolean FORCE_PROPAGATION = false; + + private OtelTracePropagationTestToggle() { + } +} diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java index 5ca248db59d..c7f78b727e6 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java @@ -16,8 +16,22 @@ package com.mongodb.internal.observability.micrometer; +import com.mongodb.lang.Nullable; + @SuppressWarnings("InterfaceIsType") public interface TraceContext { TraceContext EMPTY = new TraceContext() { + @Override + public String traceParent() { + return null; + } }; + + /** + * The W3C {@code traceparent} string for this context + * ({@code 00-<32hex traceId>-<16hex spanId>-<2hex flags>}), + * or {@code null} if unavailable or the span is not sampled. + */ + @Nullable + String traceParent(); } diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java new file mode 100644 index 00000000000..5f941327699 --- /dev/null +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java @@ -0,0 +1,207 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.connection; + +import com.mongodb.MongoNamespace; +import com.mongodb.ReadConcern; +import com.mongodb.ReadPreference; +import com.mongodb.connection.ClusterConnectionMode; +import com.mongodb.connection.ServerType; +import com.mongodb.internal.TimeoutContext; +import com.mongodb.internal.TimeoutSettings; +import com.mongodb.internal.connection.MessageSequences.EmptyMessageSequences; +import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; +import com.mongodb.internal.observability.micrometer.Span; +import com.mongodb.internal.observability.micrometer.TraceContext; +import com.mongodb.internal.session.SessionContext; +import com.mongodb.internal.validator.NoOpFieldNameValidator; +import org.bson.BsonDocument; +import org.bson.BsonString; +import org.junit.jupiter.api.Test; + +import java.nio.charset.StandardCharsets; + +import static com.mongodb.internal.mockito.MongoMockito.mock; +import static com.mongodb.internal.operation.ServerVersionHelper.LATEST_WIRE_VERSION; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.when; + +class CommandMessageOtelTraceContextTest { + + private static final String TRACEPARENT = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"; + private static final MongoNamespace NAMESPACE = new MongoNamespace("db.test"); + private static final BsonDocument COMMAND = new BsonDocument("find", new BsonString(NAMESPACE.getCollectionName())); + + @Test + void writesSectionWhenSupportedAndSampledSpanPresent() { + CommandMessage message = buildCommandMessage(true); + TraceContext traceContext = () -> TRACEPARENT; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertTrue(containsOtelSection(encoded, TRACEPARENT), + "Encoded message should contain the OTel trace context section (kind byte 3 + traceparent C-string)"); + } + + @Test + void getCommandDocumentIgnoresOtelSection() { + // Regression guard: InternalStreamConnection calls getCommandDocument() on every send (logging/monitoring/ + // compression). The trailing kind-3 section must not corrupt command-document reconstruction. + CommandMessage message = buildCommandMessage(true); + TraceContext traceContext = () -> TRACEPARENT; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + BsonDocument commandDocument = message.getCommandDocument(output); + assertEquals("test", commandDocument.getString("find").getValue()); + assertEquals("db", commandDocument.getString("$db").getValue()); + // The reconstructed command is exactly the body section (find + $db); the trailing + // kind-3 section must not leak any extra fields into it. + assertEquals(2, commandDocument.size()); + } + } + + @Test + void omitsSectionWhenCapabilityAbsent() { + CommandMessage message = buildCommandMessage(false); + TraceContext traceContext = () -> TRACEPARENT; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertFalse(containsOtelSection(encoded, TRACEPARENT), + "Encoded message should NOT contain the OTel trace context section when tracingSupported=false"); + } + + @Test + void omitsSectionWhenSpanHasNoTraceParent() { + CommandMessage message = buildCommandMessage(true); + TraceContext traceContext = () -> null; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertFalse(containsOtelSection(encoded, TRACEPARENT), + "Encoded message should NOT contain the OTel trace context section when traceParent is null"); + } + + @Test + void omitsSectionWhenSpanIsNull() { + CommandMessage message = buildCommandMessage(true); + OperationContext operationContext = buildOperationContext(null); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertFalse(containsOtelSection(encoded, TRACEPARENT), + "Encoded message should NOT contain the OTel trace context section when there is no tracing span"); + } + + @Test + void writesSectionWhenForcedEvenIfCapabilityAbsent() { + OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; + try { + CommandMessage message = buildCommandMessage(false); // server did NOT advertise tracingSupport + TraceContext traceContext = () -> TRACEPARENT; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertTrue(containsOtelSection(encoded, TRACEPARENT), + "With FORCE_PROPAGATION the section must be sent even when the server did not advertise support"); + } finally { + OtelTracePropagationTestToggle.FORCE_PROPAGATION = false; + } + } + + // --- helpers --- + + private static CommandMessage buildCommandMessage(final boolean tracingSupported) { + return new CommandMessage( + NAMESPACE.getDatabaseName(), + COMMAND, + NoOpFieldNameValidator.INSTANCE, + ReadPreference.primary(), + MessageSettings.builder() + .maxWireVersion(LATEST_WIRE_VERSION) + .serverType(ServerType.REPLICA_SET_PRIMARY) + .sessionSupported(true) + .tracingSupported(tracingSupported) + .build(), + true, + EmptyMessageSequences.INSTANCE, + ClusterConnectionMode.MULTIPLE, + null); + } + + private static OperationContext buildOperationContext(final Span span) { + SessionContext sessionContext = mock(SessionContext.class, mock -> { + when(mock.getClusterTime()).thenReturn(null); + when(mock.hasSession()).thenReturn(false); + when(mock.getReadConcern()).thenReturn(ReadConcern.DEFAULT); + when(mock.notifyMessageSent()).thenReturn(true); + when(mock.hasActiveTransaction()).thenReturn(false); + when(mock.isSnapshot()).thenReturn(false); + }); + TimeoutContext timeoutContext = new TimeoutContext(TimeoutSettings.DEFAULT); + return mock(OperationContext.class, mock -> { + when(mock.getSessionContext()).thenReturn(sessionContext); + when(mock.getTimeoutContext()).thenReturn(timeoutContext); + when(mock.getTracingSpan()).thenReturn(span); + }); + } + + private static byte[] encodeToBytes(final CommandMessage message, final OperationContext operationContext) { + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + return output.toByteArray(); + } + } + + /** + * Searches for the needle: kind byte {@code 3} immediately followed by the UTF-8 bytes of + * {@code traceparent} and a trailing null byte (C-string terminator). + */ + private static boolean containsOtelSection(final byte[] encoded, final String traceparent) { + byte[] traceparentBytes = traceparent.getBytes(StandardCharsets.UTF_8); + // needle = [0x03, tp[0], tp[1], ..., tp[n-1], 0x00] + int needleLen = 1 + traceparentBytes.length + 1; + outer: + for (int i = 0; i <= encoded.length - needleLen; i++) { + if (encoded[i] != 3) { + continue; + } + for (int j = 0; j < traceparentBytes.length; j++) { + if (encoded[i + 1 + j] != traceparentBytes[j]) { + continue outer; + } + } + if (encoded[i + 1 + traceparentBytes.length] == 0) { + return true; + } + } + return false; + } +} diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java new file mode 100644 index 00000000000..1af0ca854fa --- /dev/null +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java @@ -0,0 +1,81 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.connection; + +import com.mongodb.ServerAddress; +import com.mongodb.connection.ClusterConnectionMode; +import com.mongodb.connection.ClusterId; +import com.mongodb.connection.ConnectionDescription; +import com.mongodb.connection.ConnectionId; +import com.mongodb.connection.ServerId; +import org.bson.BsonBoolean; +import org.bson.BsonDocument; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class DescriptionHelperTracingTest { + private static final ConnectionId CONNECTION_ID = + new ConnectionId(new ServerId(new ClusterId(), new ServerAddress())); + + private static BsonDocument hello(final boolean tracing) { + BsonDocument doc = BsonDocument.parse( + "{ ok: 1, ismaster: true, maxWireVersion: 25, minWireVersion: 0," + + " maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, maxWriteBatchSize: 100000 }"); + if (tracing) { + doc.put("tracingSupport", BsonBoolean.TRUE); + } + return doc; + } + + @Test + void parsesTracingSupportTrue() { + ConnectionDescription description = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(true)); + assertTrue(description.isTracingSupported()); + } + + @Test + void defaultsTracingSupportFalseWhenAbsent() { + ConnectionDescription description = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(false)); + assertFalse(description.isTracingSupported()); + } + + @Test + void parsesTracingSupportFalseWhenExplicitlyFalse() { + BsonDocument helloResult = hello(false); + helloResult.put("tracingSupport", BsonBoolean.FALSE); + ConnectionDescription description = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, helloResult); + assertFalse(description.isTracingSupported()); + } + + @Test + void withTracingSupportPreservesOtherFields() { + ConnectionDescription original = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(false)); + ConnectionDescription updated = original.withTracingSupport(true); + + assertTrue(updated.isTracingSupported()); + assertEquals(original.getConnectionId(), updated.getConnectionId()); + assertEquals(original.getMaxWireVersion(), updated.getMaxWireVersion()); + assertEquals(original.getServerType(), updated.getServerType()); + } +} diff --git a/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java new file mode 100644 index 00000000000..99324750344 --- /dev/null +++ b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java @@ -0,0 +1,73 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.observability.micrometer; + +import io.micrometer.observation.ObservationRegistry; +import io.micrometer.tracing.test.simple.SimpleTracer; +import io.micrometer.tracing.test.simple.SimpleTraceContext; +import io.micrometer.tracing.handler.DefaultTracingObservationHandler; +import com.mongodb.observability.micrometer.MongodbObservation; +import org.junit.jupiter.api.Test; + +import java.util.regex.Pattern; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class MicrometerTraceParentTest { + // SimpleTracer generates 8-byte (16-hex-char) trace IDs; real OTel uses 16-byte (32-hex-char). + // Accept either length in the unit test — the format check is what matters here. + private static final Pattern TRACEPARENT = + Pattern.compile("00-[0-9a-f]{16,32}-[0-9a-f]{16}-[0-9a-f]{2}"); + + @Test + void returnsTraceParentForSampledSpan() { + ObservationRegistry registry = ObservationRegistry.create(); + SimpleTracer tracer = new SimpleTracer(); + registry.observationConfig().observationHandler(new DefaultTracingObservationHandler(tracer)); + + MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); + Span span = micrometerTracer.nextSpan(MongodbObservation.MONGODB_COMMAND, "find", null, null); + span.openScope(); + // SimpleTracer creates spans with sampled=false by default; mark as sampled so + // traceParent() emits the header (flags=01). + ((SimpleTraceContext) tracer.lastSpan().context()).setSampled(true); + try { + String traceParent = span.context().traceParent(); + assertNotNull(traceParent); + assertTrue(TRACEPARENT.matcher(traceParent).matches(), traceParent); + } finally { + span.closeScope(); + span.end(); + } + } + + @Test + void returnsNullWhenNoTracingBridgeConfigured() { + ObservationRegistry registry = ObservationRegistry.create(); + MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); + Span span = micrometerTracer.nextSpan(MongodbObservation.MONGODB_COMMAND, "find", null, null); + span.openScope(); + try { + assertNull(span.context().traceParent()); + } finally { + span.closeScope(); + span.end(); + } + } +} diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 7686cc15c41..2e67d80b43b 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -186,6 +186,7 @@ project-reactor-test = { module = "io.projectreactor:reactor-test" } reactive-streams-tck = { module = " org.reactivestreams:reactive-streams-tck", version.ref = "reactive-streams" } reflections = { module = "org.reflections:reflections", version.ref = "reflections" } +micrometer-tracing = { module = "io.micrometer:micrometer-tracing", version.ref = "micrometer-tracing" } micrometer-tracing-integration-test-bom = { module = " io.micrometer:micrometer-tracing-bom", version.ref = "micrometer-tracing" } micrometer-tracing-integration-test = { module = " io.micrometer:micrometer-tracing-integration-test" } From b7def2e3f875d4a82e0e5d90547256e748a1073d Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Mon, 13 Jul 2026 18:14:08 +0100 Subject: [PATCH 04/48] DRIVERS-3454: design for reference implementation (BSON telemetry section, wire-version gate) --- ...-08-mongos-version-skew-and-negotiation.md | 102 +++++++++++ ...telemetry-section-reference-impl-design.md | 167 ++++++++++++++++++ 2 files changed, 269 insertions(+) create mode 100644 docs/superpowers/followups/2026-06-08-mongos-version-skew-and-negotiation.md create mode 100644 docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md diff --git a/docs/superpowers/followups/2026-06-08-mongos-version-skew-and-negotiation.md b/docs/superpowers/followups/2026-06-08-mongos-version-skew-and-negotiation.md new file mode 100644 index 00000000000..e2a4cd9bcbd --- /dev/null +++ b/docs/superpowers/followups/2026-06-08-mongos-version-skew-and-negotiation.md @@ -0,0 +1,102 @@ +# Follow-up: mongos/mongod version skew and why mongos-`hello` negotiation is sufficient + +- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) +- **Date:** 2026-06-08 +- **Context:** Confirms that gating OP_MSG trace-context propagation on the **mongos** `hello` + capability is safe across sharded-cluster version skew. Pairs with the stock-server rejection + finding (`2026-06-08-stock-server-rejection.md`) and the propagation spec. + +## Experiment that motivated this + +Forcing the kind-3 section (`OtelTracePropagationTestToggle.FORCE_PROPAGATION=true`) against a stock +**MongoDB 7.0 sharded cluster** (config RS + shard RS + mongos) made the mongos reject **every** +command with `error 40432 (Location40432): "Unknown section kind 3"` — a clean, recoverable command +error (server stayed healthy), but a 100% app outage for those operations. This is the exact failure +the `hello`-capability gate must prevent, so it matters *which* node the driver negotiates with. + +## Are mongos and mongod the same version? + +Yes. `mongos` and `mongod` ship as one MongoDB Server release (same version stream), upgraded +together. The only standard divergence is **during a rolling sharded-cluster upgrade**, and the skew +direction is fixed by the required order: + +``` +disable balancer → config servers → shards → mongos (LAST) → re-enable balancer → bump FCV +``` + +So during the window the **mongos is the OLDEST** component (upgraded last); shards/config are already +newer. + +### Hard constraints that bound the skew (from official docs + server source) + +- *"The mongos binary cannot connect to mongod instances whose **feature compatibility version (FCV) + is greater than that of the mongos**."* (5.0→8.0 sharded-upgrade docs.) ⇒ mongos is **never behind + the cluster FCV**. +- During upgrade/downgrade the **internal-cluster `hello`** clamps `minWireVersion == maxWireVersion + == LATEST` so an out-of-range internal peer fails to connect + (`src/mongo/db/commands/feature_compatibility_version.cpp`, + `jstests/noPassthrough/.../hello_feature_compatibility_version.js`). This clamp is for + `internalClient` only; external app drivers still receive the full `minWireVersion=0..max` range. +- Internal TSE knowledge lists *"mongos version skew"* as a known upgrade failure mode + (`10gen/tse-strategy-backtest-scoreboard` upgrade-paths SKILL). + +**Net:** mongos may be one major behind the *binaries* transiently, but never behind the *FCV*, and +mongos ≤ shard versions during the upgrade. + +## Why mongos-`hello` negotiation is sufficient + +1. The driver connects to the **mongos** and reads its `hello`. Because mongos is upgraded **last**, + it's the **lowest common denominator** the driver talks to — too-old mongos ⇒ no `tracingSupport` + advertised ⇒ driver doesn't send the section. Conservative and safe. +2. The **mongos→shard** hop is covered: shards are upgraded **before** mongos, so once the mongos + advertises support, the shards already support it (no risk of forwarding kind-3 to a shard that + can't — *assuming the server re-propagates the section onward*). +3. **FCV-gating** closes the remaining gap: if advertisement is FCV-gated, the mongos won't advertise + until FCV is bumped, which only happens after every component (incl. mongos) is upgraded ⇒ uniform + support. + +Conclusion: the driver does **not** need to probe each shard; negotiating once at the mongos is +sufficient given the documented upgrade order + FCV rule. + +## Atlas Proxy / mongonet — the third (and most important) parsing surface + +In Atlas the driver connects to the **Atlas Proxy**, not to mongos/mongod directly, so the proxy is +the surface that matters most. + +- The Atlas Proxy uses **mongonet** for all wire-protocol parse/serialize/forward (Atlas Proxy + Resources wiki). It fully decodes each `OP_MSG`, iterates `msg.Sections`, and re-serializes before + forwarding. +- mongonet models sections as an **enumerated set of typed structs** (`BodySection`, + `DocumentSequenceSection`, `SecurityToken`). The security-token section (**kind 2**) had to be + **explicitly added to mongonet** (CLOUDP-167278 era: *"Add UnsignedSecurityToken section to + mongonet OP_MSG"*). mongonet does **not** generically pass through arbitrary section kinds. +- **Consequence:** an un-updated Atlas Proxy will not understand **kind 3** — it will either reject it + (most likely, mirroring mongod's `40432` → hard outage) or fail to forward it (silently broken + propagation, section dropped before the backend). Either way Atlas needs **explicit mongonet + + Atlas Proxy work** to parse-and-forward the section, even if mongos/mongod already support it. +- **Negotiation still works at the endpoint:** the driver's `hello` in Atlas comes *through the + proxy*, so the proxy is the natural capability gate — it won't advertise `tracingSupport` until + mongonet can handle the section. Same lowest-common-denominator logic as mongos-upgraded-last. + +This is already flagged internally: James Kovacs's DRIVERS-3454 sync notes have an action item to +*"investigate impact on drivers and other software that parses OP_MSG… unknown section type, +especially for existing drivers/services."* Noah Stapp's **"OP_MSG Telemetry Scope"** wants the +section processable *without* server changes and notes the server must support it *"present in +potentially any message"* — plus there is an effort to **consolidate OTel context + retry telemetry + +backpressure into one generic OP_MSG telemetry section**, which may reshape this whole approach. + +## Open confirmations for the server / Atlas team + +1. **`hello` capability** — confirm the plan to advertise trace-context support in `hello` (e.g. + `tracingSupport: true`); this is the production gate replacing the test toggle. +2. **Binary-gated vs FCV-gated** — will `tracingSupport` be advertised based on binary version or on + FCV? (Affects whether the gate is uniform during the upgrade window.) +3. **mongos onward propagation** — does the mongos re-propagate the kind-3 section on the + mongos→shard hop (so shard-side spans link), or is server-side propagation handled separately? +4. **mongonet / Atlas Proxy support** — will mongonet parse *and forward* the new section (and the + proxy re-propagate it to the backend), so Atlas-connected drivers work rather than break? +5. **Proxy `hello` advertisement** — will the Atlas Proxy advertise `tracingSupport` (reflecting proxy + + backend support together), so the driver's capability gate covers the Atlas path? + +Items 2–5 determine whether endpoint-only negotiation (mongos or Atlas Proxy) is fully sufficient end +to end. The consolidation into a single generic telemetry section (above) may also change the design. diff --git a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md new file mode 100644 index 00000000000..a3b2be7d103 --- /dev/null +++ b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md @@ -0,0 +1,167 @@ +# Design: DRIVERS-3454 Reference Implementation — OTel Trace-Context Propagation (BSON telemetry section) + +- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) — Support trace context propagation to the server +- **Status:** Approved design (reference implementation) +- **Author:** Nabil Hachicha +- **Date:** 2026-07-13 +- **Supersedes:** `2026-06-01-otel-opmsg-propagation-design.md` (POC design; wire contract and negotiation have since changed) +- **Related:** [DRIVERS-719](https://jira.mongodb.org/browse/DRIVERS-719) (client-side OTel tracing), + [SERVER-107128](https://jira.mongodb.org/browse/SERVER-107128), + server implementation [10gen/mongo#56646](https://github.com/10gen/mongo/pull/56646) (merged 2026-07-06), + [Technical Design: Drivers OTEL Trace Context Propagation to the Server](https://docs.google.com/document/d/1mLFw3yvLW3GPq8GLU82tNIHmsNiS5UvDmMJ4FjB80R4/edit) + +--- + +## 1. Summary + +Turn the existing POC on this branch into the production-quality **reference implementation** +for propagating the client's OTel trace context to the server, matching the final +driver/server contract that the server has now shipped (10gen/mongo#56646). The final +contract differs from the POC in three material ways: + +1. **Payload is a BSON document**, not a bare traceparent C-string: + `{ otel: { traceparent: "<55-char W3C traceparent>" } }`. +2. **Gating is by `maxWireVersion >= 29`** (server 9.0), not a `hello` capability flag. + The hello-flag approach was explicitly rejected (meeting 2026-06-16, Nabil/Jeff/Didier). +3. The server **strictly validates** the traceparent and enforces a 4 KB section size cap. + +Deliverables: production internal implementation in `driver-core` (shared by sync and +reactive), prose-test definitions upstreamable to the future DRIVERS spec PR, unit tests, +and an automated end-to-end test that asserts server-span linkage via the server's OTLP +file exporter. + +## 2. Server contract (authoritative, from merged PR #56646) + +- OP_MSG section kind **3** (`kTelemetry`), payload is a single BSON document. +- Schema (IDL `TelemetryContextSection`, `strict: false` at both levels — unknown fields + ignored): + + ``` + { otel: { traceparent: } } + ``` + +- Max section size **4096 bytes** (`kMaxTelemetrySectionSize`); larger payloads are + rejected with `BSONObjectTooLarge`. +- `traceparent` is validated by `otel::traces::validateW3CTraceparent`: + - exactly **55 characters**: `"---"`; + - lowercase hex only; `-` delimiters at fixed positions; + - trace-id and parent-id must **not** be all zeroes; + - version `ff` is forbidden; + - **no tracestate suffix is accepted** (a >55-char value is rejected — this differs from + the earlier server POC #49930, which appended tracestate). +- Multiple telemetry sections in one message are rejected. +- The server writes the section before the body when serializing, but the **parser is + order-agnostic**; the driver may write it after the body. +- Consumption: `service_entry_point_shard_role.cpp` starts server spans from the received + context; mongos forwards it on downstream remote calls. +- Server spans are exported via startup parameters + `opentelemetryTraceDirectory` (OTLP JSON files) or `opentelemetryHttpEndpoint` + (OTLP/HTTP), subject to the server tracing feature flag and sampling parameters. + +## 3. Driver-side design + +### 3.1 Gating rule + +Attach the telemetry section to an outgoing OP_MSG **iff**: + +1. `MessageSettings.getMaxWireVersion() >= ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION` + (new constant, value **29**, matching `WIRE_VERSION_90 = 29` on the server); and +2. the `OperationContext` carries an active tracing `Span` whose context yields a + **valid** W3C traceparent (non-null). + +Tracing-disabled clients, monitoring connections, and auth commands never have an active +span (per the DRIVERS-719 implementation), so those exclusions fall out of condition 2 +without extra plumbing. `MessageSettings` already carries `maxWireVersion`, so no new +capability plumbing is needed. + +### 3.2 Wire encoding (`CommandMessage`) + +In `writeOpMsg()`, after the body and any document-sequence sections, when the gating rule +passes, write: + +- one byte `PAYLOAD_TYPE_3_TELEMETRY = 3`; +- the BSON document `{ otel: { traceparent: "" } }` (raw BSON, no length prefix + beyond the document's own). + +Placement after the body is kept (simpler for our encoder; server parser is +order-agnostic). The existing POC change to `getCommandDocument()` (tolerating a trailing +non-sequence section when re-parsing our own message) is retained and adapted. + +### 3.3 Traceparent production (`MicrometerTracer` / `TraceContext`) + +`TraceContext.traceParent()` remains the internal accessor, tightened to guarantee it +never returns a value the server would reject: + +- format `00-<32 lowercase hex>-<16 lowercase hex>-<02 flags>`, exactly 55 chars; +- return `null` for no-op spans, zero trace-id or span-id, or any malformed component + (section is then omitted entirely — the driver never sends a malformed traceparent); +- unsampled contexts are propagated with flags `00` (the server applies its own + sampling); no tracestate is ever appended. + +### 3.4 Removals (POC artifacts) + +- `OtelTracePropagationTestToggle` (force-send toggle) — deleted. +- `MessageSettings.tracingSupported` and its builder — deleted. +- `ConnectionDescription` / `DescriptionHelper` `tracingSupport` hello parsing — reverted. +- POC-era tests reworked to the new contract. + +No public API changes; everything stays under `com.mongodb.internal.*`. Sync and reactive +drivers both inherit the behavior through the shared `driver-core` send path. + +## 4. Testing + +### 4.1 Unit tests (run everywhere) + +- **Traceparent formatting matrix** (`MicrometerTracer`): valid sampled/unsampled spans, + no-op span, zero ids, length exactly 55, lowercase hex. +- **`CommandMessage` OP_MSG round-trip** (rework of `CommandMessageOtelTraceContextTest`): + - positive: wire version ≥ 29 + active span ⇒ section kind 3 present; payload BSON + parses to `{otel: {traceparent: ...}}` matching the span's trace-id/span-id; + - negative: wire version < 29 ⇒ no section; no span ⇒ no section; invalid traceparent ⇒ + no section; interaction with document-sequence sections preserved. + +### 4.2 Prose tests (upstreamable) + +A markdown prose-test definition (structured for the future `mongodb/specifications` +tracing spec PR) covering the gating matrix and the e2e linkage test, plus Java +implementations following the `AbstractMicrometerProseTest` pattern. + +### 4.3 End-to-end server-span linkage test + +- New integration test (sync, `AbstractMicrometerProseTest`-style) using + `InMemoryOtelSetup` to capture the client command span (expected trace-id/span-id). +- Enabled only when `-Dorg.mongodb.test.otel.trace.dir=` is set; skipped otherwise. +- Requires the test's MongoDB deployment (same host, as with mongo-orchestration in CI) to + be a **server 9.0 build started with** + `--setParameter opentelemetryTraceDirectory=` plus the tracing feature flag and + sampling parameters. +- The test runs a traced CRUD operation, polls the directory past the export interval + (`openTelemetryTracingBatchExportIntervalMillis`, default 1 s), parses the OTLP JSON + span files, and asserts a server span exists with `traceId` equal to the client span's + trace-id and `parentSpanId` equal to the client command span's span-id. +- The local runbook (`docs/superpowers/runbooks/otel-opmsg-e2e.md`) is updated for the + Evergreen 9.0 artifact and file-exporter workflow (Jaeger flow kept as an optional + visual check). + +## 5. Branch strategy + +Merge `origin/main` into `nabil_otel_context` (currently 11 commits behind), keeping POC +history; implement the production version as new, logically separated commits. Remove +unrelated local noise (e.g. `prompt.txt`, stray modified test files) before starting. + +## 6. Out of scope + +- `baggage` / `tracestate` propagation (future payload evolution; requires wire bump). +- Unified test format runner extensions. +- Evergreen YAML task wiring for the e2e test (follow-up). +- Public API surface changes and mongos/load-balancer forwarding (server-side concern). + +## 7. Risks / open items + +- **Wire version N = 29** matches server master today; if the final DRIVERS spec picks a + different gate before 9.0 GA, only the `ServerVersionHelper` constant changes. +- The server deploys parsing behind a feature flag (design Decision 5); a 9.0 server with + the flag off still parses the section (parse path is unconditional in #56646), so the + driver needs no flag awareness. +- Intermediaries (Atlas Proxy, Envoy, mongobetween) compatibility is tracked server-side + under SERVER-128017; not a driver concern for this implementation. From 10a7e4847ccc43a3a36be208130bb17b1ec39878 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Mon, 13 Jul 2026 18:33:00 +0100 Subject: [PATCH 05/48] DRIVERS-3454: implementation plan for reference implementation --- ...3-otel-telemetry-section-reference-impl.md | 695 ++++++++++++++++++ 1 file changed, 695 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-13-otel-telemetry-section-reference-impl.md diff --git a/docs/superpowers/plans/2026-07-13-otel-telemetry-section-reference-impl.md b/docs/superpowers/plans/2026-07-13-otel-telemetry-section-reference-impl.md new file mode 100644 index 00000000000..74cd117ccf8 --- /dev/null +++ b/docs/superpowers/plans/2026-07-13-otel-telemetry-section-reference-impl.md @@ -0,0 +1,695 @@ +# DRIVERS-3454 Reference Implementation Plan — BSON Telemetry Section + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Convert the DRIVERS-3454 POC into the production-quality reference implementation matching the shipped server contract: OP_MSG section kind 3 carrying `{otel: {traceparent: "<55-char W3C>"}}` as BSON, gated on `maxWireVersion >= 29`. + +**Architecture:** All changes live in `driver-core` internal packages (shared by sync/reactive). The hello-capability plumbing from the POC is deleted; the gate becomes the existing `MessageSettings.getMaxWireVersion()` compared to a new `ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION = 29`. `MicrometerTracer` is tightened so `TraceContext.traceParent()` never yields a value the server would reject. Prose tests + an orchestration-wired e2e test verify server-span linkage. + +**Tech Stack:** Java 8 (driver-core baseline), JUnit 5, Micrometer Tracing test kit (`InMemoryOtelSetup`), Gradle Kotlin DSL. + +**Spec:** `docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md` (read it first). + +## Global Constraints + +- **LOCAL ONLY: never `git push`.** All commits stay on local branch `nabil_otel_context`. +- Java 8 source baseline in driver-core — no `var`, records, `Stream.toList()`, text blocks, etc. +- No public API changes. Everything under `com.mongodb.internal.*` except the *removal* of the not-yet-released POC additions to `com.mongodb.connection.ConnectionDescription` (added on this branch only, never released — safe to delete). +- Copyright header `Copyright 2008-present MongoDB, Inc.` on new files. No `System.out.println`; SLF4J only. +- Run `./gradlew spotlessApply` before each commit if formatting-sensitive files changed. +- Server contract (from merged 10gen/mongo#56646): section kind 3; payload BSON `{otel: {traceparent: }}`; traceparent exactly 55 chars `00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>`, non-zero trace-id/span-id, version != `ff`, **no tracestate**; max section 4096 bytes; at most one telemetry section per message. + +--- + +### Task 1: Branch prep — park unrelated noise, merge origin/main + +**Files:** +- No source files; git operations only. + +**Interfaces:** +- Produces: a working tree at `origin/main`-merged state with only DRIVERS-3454 content, on which all later tasks build. + +- [ ] **Step 1: Park unrelated local modifications (do NOT delete)** + +The working tree has unrelated noise. Stash tracked modifications with a label; leave untracked files (`prompt.txt`, `bson-kotlin/.../ReproJava6230Test.kt`) in place — they are ignored by later tasks and must not be committed. + +```bash +git status --porcelain +# reset the accidentally staged unrelated file, then stash all unrelated tracked changes +git restore --staged bson/src/test/unit/org/bson/BsonDocumentTest.java +git stash push -m "unrelated-local-noise-before-DRIVERS-3454" \ + bson/src/test/unit/org/bson/BsonDocumentTest.java \ + driver-scala/src/integrationTest/scala/tour/QuickTour.scala \ + driver-scala/src/test/scala/org/mongodb/scala/MongoClientSpec.scala \ + mongodb-crypt/build.gradle.kts +``` + +Expected: `git status --porcelain` shows only the untracked `prompt.txt`, `ReproJava6230Test.kt`, and the followups doc if still staged (`docs/superpowers/followups/...` — commit that separately if staged: `git commit -m "DRIVERS-3454: follow-up notes on mongos version skew" docs/superpowers/followups/2026-06-08-mongos-version-skew-and-negotiation.md`). + +- [ ] **Step 2: Merge origin/main** + +```bash +git fetch origin +git merge origin/main --no-edit +``` + +If conflicts arise, they will most likely be in `driver-core` files the POC touched (`ConnectionDescription.java`, `CommandMessage.java`, `gradle/libs.versions.toml`). Resolve keeping BOTH the upstream changes and the POC changes (the POC changes get reworked/deleted in later tasks, so a mechanically correct merge is enough). Commit the merge. + +- [ ] **Step 3: Sanity-build driver-core** + +```bash +./gradlew :driver-core:compileJava :driver-core:compileTestJava -q +``` + +Expected: BUILD SUCCESSFUL. If the merge broke compilation, fix and `git commit --amend` the merge resolution. + +--- + +### Task 2: `ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION = 29` + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java` + +**Interfaces:** +- Produces: `public static final int NINE_DOT_ZERO_WIRE_VERSION = 29;` — consumed by Task 4's gate in `CommandMessage`. + +- [ ] **Step 1: Add the constant** + +In `ServerVersionHelper`, immediately after `EIGHT_DOT_ZERO_WIRE_VERSION = 25` (line ~33): + +```java + // Server 9.0 (WIRE_VERSION_90 = 29). Minimum wire version for the OP_MSG telemetry + // section (OTel trace-context propagation, DRIVERS-3454). + public static final int NINE_DOT_ZERO_WIRE_VERSION = 29; +``` + +Do NOT change `LATEST_WIRE_VERSION` (it deliberately tracks the newest wire version the driver fully supports). + +- [ ] **Step 2: Compile and commit** + +```bash +./gradlew :driver-core:compileJava -q +git add driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java +git commit -m "DRIVERS-3454: add NINE_DOT_ZERO_WIRE_VERSION (29) constant" +``` + +--- + +### Task 3: Tighten `MicrometerTracer` traceparent production + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java` (inner class `MicrometerTraceContext.traceParent()`, ~line 122) +- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java` (javadoc only) +- Test: `driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java` (exists from POC — rework) + +**Interfaces:** +- Consumes: nothing from other tasks. +- Produces: `TraceContext.traceParent()` returns either `null` or a guaranteed-server-valid 55-char string. Behavior change vs POC: **unsampled contexts now return a traceparent with flags `00`** (previously `null`); malformed/zero ids return `null`. + +- [ ] **Step 1: Rework the unit test to the new contract** + +Replace the POC assertions in `MicrometerTraceParentTest` with a matrix (keep the file's existing test scaffolding/mocks for building Micrometer contexts; adjust to these cases): + +```java + @Test + void shouldFormatSampledTraceParent() { + // given a context with traceId "0af7651916cd43dd8448eb211c80319c", spanId "b7ad6b7169203331", sampled=true + assertEquals("00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01", traceParent); + assertEquals(55, traceParent.length()); + } + + @Test + void shouldFormatUnsampledTraceParentWithZeroFlags() { + // same ids, sampled=false (and also sampled=null) + assertEquals("00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-00", traceParent); + } + + @Test + void shouldReturnNullForInvalidIds() { + // each of these must yield null: + // traceId all zeros: "00000000000000000000000000000000" + // spanId all zeros: "0000000000000000" + // traceId wrong length: "abc" + // spanId wrong length: "abc" + // traceId uppercase hex: "0AF7651916CD43DD8448EB211C80319C" + // traceId null / spanId null + assertNull(traceParent); + } +``` + +- [ ] **Step 2: Run to verify the new cases fail** + +```bash +./gradlew :driver-core:test --tests 'com.mongodb.internal.observability.micrometer.MicrometerTraceParentTest' -q +``` + +Expected: FAIL (unsampled currently returns null; invalid ids currently pass through). + +- [ ] **Step 3: Implement** + +In `MicrometerTracer.MicrometerTraceContext`, replace the body of `traceParent()` from the `io.micrometer.tracing.TraceContext ctx` null-check down, and add the helper: + +```java + io.micrometer.tracing.TraceContext ctx = tracingContext.getSpan().context(); + if (ctx == null) { + return null; + } + String traceId = ctx.traceId(); + String spanId = ctx.spanId(); + if (!isValidNonZeroLowercaseHex(traceId, 32) || !isValidNonZeroLowercaseHex(spanId, 16)) { + return null; + } + Boolean sampled = ctx.sampled(); + return "00-" + traceId + "-" + spanId + (sampled != null && sampled ? "-01" : "-00"); + } + + /** + * The server ({@code validateW3CTraceparent}) rejects ids that are not exactly the expected + * length of lowercase hex, or that are all zeroes. Never emit a traceparent it would reject. + */ + private static boolean isValidNonZeroLowercaseHex(@Nullable final String value, final int expectedLength) { + if (value == null || value.length() != expectedLength) { + return false; + } + boolean nonZero = false; + for (int i = 0; i < value.length(); i++) { + char c = value.charAt(i); + if (!((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f'))) { + return false; + } + if (c != '0') { + nonZero = true; + } + } + return nonZero; + } +``` + +Update the javadoc of `TraceContext.traceParent()` to say: returns the 55-char W3C traceparent (`00-<32 hex>-<16 hex>-`; flags `01` sampled / `00` unsampled), or `null` when there is no valid context (no-op span, missing/zero/malformed ids). Never includes tracestate. + +- [ ] **Step 4: Run tests, verify pass** + +```bash +./gradlew :driver-core:test --tests 'com.mongodb.internal.observability.micrometer.MicrometerTraceParentTest' -q +``` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add driver-core/src/main/com/mongodb/internal/observability/micrometer/ driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/ +git commit -m "DRIVERS-3454: guarantee traceParent() is server-valid; propagate unsampled with flags 00" +``` + +--- + +### Task 4: `CommandMessage` — BSON telemetry section, wire-version gate + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` +- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` (exists from POC — rework) + +**Interfaces:** +- Consumes: `ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION` (Task 2); `TraceContext.traceParent()` contract (Task 3); existing `OperationContext.getTracingSpan()` (returns `@Nullable Span`). +- Produces: OP_MSG bytes with trailing section kind 3 whose payload is the BSON document `{otel: {traceparent: }}`. `getCommandDocument()` keeps ignoring the trailing non-sequence section (POC behavior retained). + +- [ ] **Step 1: Rework the round-trip test** + +Rework `CommandMessageOtelTraceContextTest` (keep its existing harness for encoding a `CommandMessage` and re-reading the produced buffer; the POC test asserted a C-string payload). New assertions: + +```java + @Test + void shouldWriteTelemetrySectionWhenWireVersionAtLeast29AndSpanActive() { + // settings: MessageSettings.builder().maxWireVersion(NINE_DOT_ZERO_WIRE_VERSION).build() + // operationContext with a tracing span whose context().traceParent() returns + // "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01" + // after encode(): scan sections after the body; expect one byte 0x03 followed by a BSON document + BsonDocument telemetry = readTelemetrySectionDocument(buffer); + assertEquals(new BsonDocument("otel", new BsonDocument("traceparent", + new BsonString("00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"))), telemetry); + } + + @Test + void shouldNotWriteTelemetrySectionWhenWireVersionBelow29() { + // identical setup but maxWireVersion = EIGHT_DOT_ZERO_WIRE_VERSION (25) => no section kind 3 + } + + @Test + void shouldNotWriteTelemetrySectionWhenNoSpan() { + // wire version 29, operationContext.getTracingSpan() == null => no section kind 3 + } + + @Test + void shouldNotWriteTelemetrySectionWhenTraceParentNull() { + // wire version 29, span present but context().traceParent() == null => no section kind 3 + } + + @Test + void getCommandDocumentIgnoresTelemetrySection() { + // encode with section present; CommandMessage.getCommandDocument(...) still returns the command + // (retains the POC regression test, renamed) + } + + @Test + void shouldWriteTelemetrySectionAfterDocumentSequences() { + // encode a command with a document-sequence payload AND an active span at wire version 29; + // assert sequence section(s) precede the kind-3 section and the command re-parses correctly + } +``` + +`readTelemetrySectionDocument` helper: walk sections after the body exactly like the production parser (kind byte, then for kind 1 skip `int32` size bytes, for kind 3 decode one BSON document via `new BsonDocumentCodec().decode(new BsonBinaryReader(...))`), fail if kind 3 absent. + +- [ ] **Step 2: Run to verify failures** + +```bash +./gradlew :driver-core:test --tests 'com.mongodb.internal.connection.CommandMessageOtelTraceContextTest' -q +``` + +Expected: FAIL (payload is still a C-string; gate is still `isTracingSupported()`). + +- [ ] **Step 3: Implement in `CommandMessage`** + +1. Rename the constant (~line 89) and update its javadoc: + +```java + /** + * Specifies that the `OP_MSG` section payload is a BSON telemetry document + * ({@code {otel: {traceparent: }}}). Mirrors the server's {@code kTelemetry = 3} + * section kind (DRIVERS-3454, 10gen/mongo#56646). + */ + private static final byte PAYLOAD_TYPE_3_TELEMETRY = 3; +``` + +2. Replace `writeOtelTraceContextSection` with: + +```java + private void writeTelemetryContextSection(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext) { + if (getSettings().getMaxWireVersion() < NINE_DOT_ZERO_WIRE_VERSION) { + return; + } + Span tracingSpan = operationContext.getTracingSpan(); + if (tracingSpan == null) { + return; + } + String traceParent = tracingSpan.context().traceParent(); + if (traceParent == null) { + return; + } + bsonOutput.writeByte(PAYLOAD_TYPE_3_TELEMETRY); + BsonDocument telemetry = new BsonDocument("otel", + new BsonDocument("traceparent", new BsonString(traceParent))); + new BsonDocumentCodec().encode(new BsonBinaryWriter(bsonOutput), telemetry, + EncoderContext.builder().build()); + } +``` + +Update the call site in `writeOpMsg()` (`writeOtelTraceContextSection(...)` → `writeTelemetryContextSection(...)`). Add imports: `org.bson.BsonString`, `org.bson.codecs.BsonDocumentCodec`, `org.bson.codecs.EncoderContext`, `static com.mongodb.internal.operation.ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION`. Remove imports of `OtelTracePropagationTestToggle`. Keep the `getCommandDocument()` handling of trailing non-sequence sections (update its comment to say `PAYLOAD_TYPE_3_TELEMETRY`). + +- [ ] **Step 4: Run tests, verify pass** + +```bash +./gradlew :driver-core:test --tests 'com.mongodb.internal.connection.CommandMessageOtelTraceContextTest' -q +``` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +git commit -m "DRIVERS-3454: write BSON telemetry section gated on maxWireVersion >= 29" +``` + +--- + +### Task 5: Delete POC capability plumbing + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/connection/ConnectionDescription.java` (revert POC additions: `tracingSupport` field, constructor param, `withTracingSupport`, `isTracingSupported`, equals/hashCode/toString entries) +- Modify: `driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java` (remove `withTracingSupport(...)` call at ~line 72 and `getTracingSupport` helper at ~line 175) +- Modify: `driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java` (remove `.tracingSupported(...)` at ~line 238) +- Modify: `driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java` (remove `tracingSupported` field, builder method, getter) +- Delete: `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java` +- Delete: `driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java` + +**Interfaces:** +- Consumes: Task 4 must be done first (CommandMessage no longer references the toggle or `isTracingSupported()`). +- Produces: clean tree — `git grep -i tracingSupport` and `git grep OtelTracePropagationTestToggle` return nothing. + +- [ ] **Step 1: Remove all plumbing** + +Use `git diff origin/main -- ` on each of the four modified files to see exactly what the POC added, and revert those hunks (the cleanest method: `git checkout origin/main -- driver-core/src/main/com/mongodb/connection/ConnectionDescription.java driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java` — valid because the POC changes are the ONLY branch changes to these files; verify with `git diff origin/main --stat` first, and re-apply by hand if upstream merge conflicts made the files diverge otherwise). Then: + +```bash +git rm driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java +git rm driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java +``` + +- [ ] **Step 2: Verify nothing references the removed API** + +```bash +git grep -il 'tracingSupport' -- '*.java' ; git grep -l 'OtelTracePropagationTestToggle' -- '*.java' +./gradlew :driver-core:compileJava :driver-core:compileTestJava -q +``` + +Expected: no grep hits; BUILD SUCCESSFUL. Fix any leftover references (e.g. `DescriptionHelperTest`, `ConnectionDescription` unit tests touched by the POC). + +- [ ] **Step 3: Run the affected test suites** + +```bash +./gradlew :driver-core:test --tests '*CommandMessage*' --tests '*ConnectionDescription*' --tests '*DescriptionHelper*' --tests '*MessageSettings*' -q +``` + +Expected: PASS. + +- [ ] **Step 4: Commit** + +```bash +git add -A driver-core +git commit -m "DRIVERS-3454: remove hello-capability plumbing and test toggle (wire-version gate supersedes)" +``` + +--- + +### Task 6: Full driver-core validation + +**Files:** none (verification gate). + +- [ ] **Step 1: Static checks + full driver-core unit tests** + +```bash +./gradlew spotlessApply -q +./gradlew :driver-core:check -q +``` + +Expected: BUILD SUCCESSFUL. Fix anything that fails (checkstyle on new code, unused imports, etc.). + +- [ ] **Step 2: Commit any fixups** + +```bash +git status --porcelain # if formatting changed files: +git add -A && git commit -m "DRIVERS-3454: formatting/static-check fixups" +``` + +--- + +### Task 7: Prose-test definitions document + +**Files:** +- Create: `docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md` + +**Interfaces:** +- Produces: the prose-test text that Task 8's Java test implements (test names must match the prose numbering). + +- [ ] **Step 1: Write the document** + +Content (verbatim; structured like the existing OTel spec prose tests README so it can be lifted into the future `mongodb/specifications` PR): + +```markdown +# Trace-Context Propagation Prose Tests (DRIVERS-3454) + +These tests complement the OpenTelemetry spec prose tests. Tests 1–4 are unit-level and +MUST NOT require a server. Test 5 requires a MongoDB 9.0+ server (maxWireVersion >= 29) +started with OTel tracing enabled and the OTLP file exporter +(`--setParameter opentelemetryTraceDirectory=` plus the tracing feature flag and +sampling parameters), running on the same host as the test. + +## 1. Telemetry section is attached when supported and traced + +With tracing enabled and an active operation span, encode a command targeting a +connection whose `maxWireVersion` is >= 29. Assert the resulting OP_MSG contains exactly +one section of kind 3 whose payload is a BSON document of the form +`{otel: {traceparent: }}`, where `traceparent` is 55 characters, +`00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>`, and the trace-id/span-id match +the active span's context. + +## 2. Telemetry section is omitted for older servers + +Same as (1) but with `maxWireVersion` < 29. Assert no section of kind 3 is present. + +## 3. Telemetry section is omitted without an active span + +With tracing disabled (or no active span, e.g. monitoring/auth commands), encode a +command at `maxWireVersion` >= 29. Assert no section of kind 3 is present. + +## 4. Malformed trace context is never sent + +With an active span whose context cannot produce a valid W3C traceparent (zero trace-id, +zero span-id, wrong-length or non-lowercase-hex ids), assert no section of kind 3 is +present (drivers MUST omit the section rather than send an invalid traceparent). + +## 5. End-to-end server span linkage + +With tracing enabled, run a CRUD operation (e.g. `find`) against the configured 9.0+ +server. Capture the driver's finished command span (client side). Then read the server's +exported OTLP JSON from the trace directory and assert a server span exists whose +`traceId` equals the client command span's trace-id and whose `parentSpanId` equals the +client command span's span-id. Allow for the server's batch export interval when polling. +``` + +- [ ] **Step 2: Commit** + +```bash +git add docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md +git commit -m "DRIVERS-3454: prose-test definitions for trace-context propagation" +``` + +Note: prose tests 1–4 are implemented by `CommandMessageOtelTraceContextTest` (Task 4) and `MicrometerTraceParentTest` (Task 3); add a javadoc line to each test class referencing this document and the prose test numbers. + +--- + +### Task 8: E2E server-span linkage test + +**Files:** +- Create: `driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java` + +**Interfaces:** +- Consumes: `AbstractMicrometerProseTest` patterns (`InMemoryOtelSetup`, `getMongoClientSettingsBuilder()`, `ObservabilitySettings.micrometerBuilder()`), env-var helper `setEnv` if needed for enabling tracing. +- Produces: prose test 5 implementation, gated on system property `org.mongodb.test.otel.trace.dir`. + +- [ ] **Step 1: Write the test** + +```java +/* + * Copyright 2008-present MongoDB, Inc. + * ... (standard Apache header, copy from AbstractMicrometerProseTest) + */ +package com.mongodb.client.observability; + +import com.mongodb.MongoClientSettings; +import com.mongodb.client.MongoClient; +import com.mongodb.client.MongoClients; +import com.mongodb.client.MongoCollection; +import com.mongodb.observability.ObservabilitySettings; +import io.micrometer.observation.ObservationRegistry; +import io.micrometer.tracing.exporter.FinishedSpan; +import io.micrometer.tracing.test.reporter.inmemory.InMemoryOtelSetup; +import org.bson.Document; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import static com.mongodb.ClusterFixture.getDefaultDatabaseName; +import static com.mongodb.client.Fixture.getMongoClientSettingsBuilder; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; + +/** + * Prose test 5 (docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md): + * end-to-end server span linkage via the server's OTLP file exporter. + * + *

Requires a MongoDB 9.0+ (maxWireVersion >= 29) server on the same host, started with + * {@code --setParameter opentelemetryTraceDirectory=

} (plus tracing feature flag and + * sampling parameters), and the test run with + * {@code -Dorg.mongodb.test.otel.trace.dir=}. Skipped otherwise.

+ */ +@EnabledIfSystemProperty(named = "org.mongodb.test.otel.trace.dir", matches = ".+") +public class ServerSpanLinkageProseTest { + + private static final long EXPORT_POLL_TIMEOUT_MS = 30_000; + private static final long EXPORT_POLL_INTERVAL_MS = 1_000; + + private final ObservationRegistry observationRegistry = ObservationRegistry.create(); + private InMemoryOtelSetup memoryOtelSetup; + private InMemoryOtelSetup.Builder.OtelBuildingBlocks inMemoryOtel; + + @BeforeEach + void setUp() { + memoryOtelSetup = InMemoryOtelSetup.builder().register(observationRegistry); + inMemoryOtel = memoryOtelSetup.getBuildingBlocks(); + } + + @AfterEach + void tearDown() { + memoryOtelSetup.close(); + } + + @Test + @DisplayName("Prose test 5: server emits a child span of the driver command span") + void testServerSpanLinkage() throws Exception { + MongoClientSettings clientSettings = getMongoClientSettingsBuilder() + .observabilitySettings(ObservabilitySettings.micrometerBuilder() + .observationRegistry(observationRegistry) + .build()) + .build(); + + try (MongoClient client = MongoClients.create(clientSettings)) { + MongoCollection collection = + client.getDatabase(getDefaultDatabaseName()).getCollection("serverSpanLinkage"); + collection.find().first(); + } + + List clientSpans = inMemoryOtel.getFinishedSpans(); + assertTrue(clientSpans.size() >= 2, "expected operation + command client spans, got: " + clientSpans); + // command span is the innermost (first finished) span; see AbstractMicrometerProseTest ordering + FinishedSpan commandSpan = clientSpans.get(0); + String traceId = commandSpan.getTraceId(); + String commandSpanId = commandSpan.getSpanId(); + + Path traceDir = Paths.get(System.getProperty("org.mongodb.test.otel.trace.dir")); + long deadline = System.currentTimeMillis() + EXPORT_POLL_TIMEOUT_MS; + while (System.currentTimeMillis() < deadline) { + if (serverSpanLinked(traceDir, traceId, commandSpanId)) { + return; + } + Thread.sleep(EXPORT_POLL_INTERVAL_MS); + } + fail("no server span found in " + traceDir + " with traceId=" + traceId + + " and parentSpanId=" + commandSpanId); + } + + /** + * Scans OTLP JSON export files for a span with the given traceId whose parentSpanId is the + * driver command span. OTLP file exports contain resourceSpans[].scopeSpans[].spans[] objects + * with hex-encoded traceId/spanId/parentSpanId fields; a simple containment check on the two + * hex ids in the same file line is sufficient and avoids a protobuf/JSON-schema dependency. + */ + private static boolean serverSpanLinked(final Path traceDir, final String traceId, final String parentSpanId) + throws IOException { + if (!Files.isDirectory(traceDir)) { + return false; + } + try (Stream files = Files.walk(traceDir)) { + List exportFiles = files.filter(Files::isRegularFile).collect(Collectors.toList()); + for (Path file : exportFiles) { + for (String line : Files.readAllLines(file, StandardCharsets.UTF_8)) { + if (line.contains(traceId) && line.contains("\"parentSpanId\":\"" + parentSpanId + "\"")) { + return true; + } + } + } + } + return false; + } +} +``` + +Adaptation note for the implementer: verify the exact accessor names on `FinishedSpan` (`getTraceId()`/`getSpanId()`) and the client-span ordering against `AbstractMicrometerProseTest` (it asserts `get(0)` is the command span, e.g. `"find"`); also check how that class enables tracing — if tracing is env-var-gated (`ENV_OBSERVABILITY_ENABLED`), reuse its `setEnv` helper in `@BeforeEach`/`@AfterEach` the same way existing tests do. Match whatever compiles against the merged main. + +- [ ] **Step 2: Verify compile + skip behavior** + +```bash +./gradlew :driver-sync:compileFunctionalJava -q 2>/dev/null || ./gradlew :driver-sync:compileTestJava -q +# run without the property; the test must be SKIPPED, not failed (check the task's test report) +``` + +Expected: compiles; test skipped when property absent. (Full e2e execution happens in Task 9's runbook flow — it needs the special server.) + +- [ ] **Step 3: Commit** + +```bash +git add driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java +git commit -m "DRIVERS-3454: e2e server-span linkage prose test (OTLP file exporter)" +``` + +--- + +### Task 9: Runbook update + manual e2e run + +**Files:** +- Modify: `docs/superpowers/runbooks/otel-opmsg-e2e.md` + +**Interfaces:** +- Consumes: Task 8's test and its `org.mongodb.test.otel.trace.dir` property. + +- [ ] **Step 1: Rewrite the runbook** + +Replace its POC content with the current workflow: + +1. Download a server 9.0 binary containing 10gen/mongo#56646 (merged 2026-07-06) from Evergreen (any master waterfall compile task after that date; note the artifact URL used). +2. Start it locally, e.g.: + +```bash +mkdir -p /tmp/otel-e2e/{db,traces} +/mongod --dbpath /tmp/otel-e2e/db --port 27017 \ + --setParameter opentelemetryTraceDirectory=/tmp/otel-e2e/traces \ + --setParameter featureFlagOtelTracing=true \ + --setParameter 'opentelemetrySamplingRates={"default": 1.0}' +``` + +(Implementer: confirm the exact feature-flag and sampling parameter names from the downloaded binary via `mongod --help` / `getParameter '*'` — they live in `src/mongo/otel/traces/tracing_feature_flags.idl` and `trace_sampling_parameters.idl` on the server; record the working invocation in the runbook.) + +3. Run the e2e test: + +```bash +./gradlew :driver-sync:test --tests 'com.mongodb.client.observability.ServerSpanLinkageProseTest' \ + -Dorg.mongodb.test.uri="mongodb://localhost:27017" \ + -Dorg.mongodb.test.otel.trace.dir=/tmp/otel-e2e/traces +``` + +4. Keep the previous Jaeger flow as an optional "visual verification" appendix (`opentelemetryHttpEndpoint=http://localhost:4318/v1/traces`), removing all references to `OtelTracePropagationTestToggle` and `tracingSupport`. + +- [ ] **Step 2: Execute the runbook end-to-end** + +Actually perform steps 1–3 and record results. Expected: `ServerSpanLinkageProseTest` PASSES against the real server. If the server rejects or ignores the section, debug with `superpowers:systematic-debugging` before touching driver code — likely causes: feature flag name, sampling off, wire version reported < 29, or traceparent validation mismatch. + +- [ ] **Step 3: Commit** + +```bash +git add docs/superpowers/runbooks/otel-opmsg-e2e.md +git commit -m "DRIVERS-3454: update e2e runbook for 9.0 server and OTLP file exporter" +``` + +--- + +### Task 10: Supersede old spec + final validation + +**Files:** +- Modify: `docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md` (status header only) + +- [ ] **Step 1: Mark the old design superseded** + +Change its `- **Status:**` line to: + +```markdown +- **Status:** SUPERSEDED by `2026-07-13-otel-telemetry-section-reference-impl-design.md` (payload is now a BSON document; gating is by maxWireVersion >= 29, not a hello flag) +``` + +- [ ] **Step 2: Final full validation** + +```bash +./gradlew spotlessApply docs check -q +``` + +Expected: BUILD SUCCESSFUL. (Skip `scalaCheck` unless Scala files were touched; integration tests only with a running server per AGENTS.md.) + +- [ ] **Step 3: Commit** + +```bash +git add -A +git commit -m "DRIVERS-3454: mark POC design superseded; final validation fixups" +``` + +**Do not push. Everything stays local until Nabil says otherwise.** From eb27cfa49906fca2df94cdc3c3f601789da62705 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Mon, 13 Jul 2026 18:58:53 +0100 Subject: [PATCH 06/48] DRIVERS-3454: add NINE_DOT_ZERO_WIRE_VERSION (29) constant --- .../com/mongodb/internal/operation/ServerVersionHelper.java | 3 +++ 1 file changed, 3 insertions(+) diff --git a/driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java b/driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java index 093d48e3781..655ab416490 100644 --- a/driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java +++ b/driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java @@ -31,6 +31,9 @@ public final class ServerVersionHelper { public static final int SIX_DOT_ZERO_WIRE_VERSION = 17; public static final int SEVEN_DOT_ZERO_WIRE_VERSION = 21; public static final int EIGHT_DOT_ZERO_WIRE_VERSION = 25; + // Server 9.0 (WIRE_VERSION_90 = 29). Minimum wire version for the OP_MSG telemetry + // section (OTel trace-context propagation, DRIVERS-3454). + public static final int NINE_DOT_ZERO_WIRE_VERSION = 29; public static final int LATEST_WIRE_VERSION = EIGHT_DOT_ZERO_WIRE_VERSION; public static boolean serverIsAtLeastVersionFourDotFour(final ConnectionDescription description) { From 44fd4ca740bfa4e899c6e2173cff3f08d684b610 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 14 Jul 2026 13:27:58 +0100 Subject: [PATCH 07/48] DRIVERS-3454: guarantee traceParent() is server-valid; propagate unsampled with flags 00 --- .../micrometer/MicrometerTracer.java | 31 ++++++++-- .../micrometer/TraceContext.java | 7 ++- .../micrometer/MicrometerTraceParentTest.java | 61 ++++++++++--------- 3 files changed, 64 insertions(+), 35 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java index e2a68cbc2b3..78caf6ae6ef 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java @@ -131,14 +131,37 @@ public String traceParent() { } // Fully qualified to avoid a name clash with this package's own TraceContext interface. io.micrometer.tracing.TraceContext ctx = tracingContext.getSpan().context(); - if (ctx == null || ctx.traceId() == null || ctx.spanId() == null) { + if (ctx == null) { return null; } - Boolean sampled = ctx.sampled(); - if (sampled == null || !sampled) { + String traceId = ctx.traceId(); + String spanId = ctx.spanId(); + if (!isValidNonZeroLowercaseHex(traceId, 32) || !isValidNonZeroLowercaseHex(spanId, 16)) { return null; } - return "00-" + ctx.traceId() + "-" + ctx.spanId() + "-01"; + Boolean sampled = ctx.sampled(); + return "00-" + traceId + "-" + spanId + (sampled != null && sampled ? "-01" : "-00"); + } + + /** + * The server ({@code validateW3CTraceparent}) rejects ids that are not exactly the expected + * length of lowercase hex, or that are all zeroes. Never emit a traceparent it would reject. + */ + private static boolean isValidNonZeroLowercaseHex(@Nullable final String value, final int expectedLength) { + if (value == null || value.length() != expectedLength) { + return false; + } + boolean nonZero = false; + for (int i = 0; i < value.length(); i++) { + char c = value.charAt(i); + if (!((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f'))) { + return false; + } + if (c != '0') { + nonZero = true; + } + } + return nonZero; } } diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java index c7f78b727e6..66199bf13ea 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java @@ -28,9 +28,10 @@ public String traceParent() { }; /** - * The W3C {@code traceparent} string for this context - * ({@code 00-<32hex traceId>-<16hex spanId>-<2hex flags>}), - * or {@code null} if unavailable or the span is not sampled. + * The 55-char W3C {@code traceparent} string for this context + * ({@code 00-<32 hex traceId>-<16 hex spanId>-}; flags {@code 01} sampled / {@code 00} unsampled), + * or {@code null} when there is no valid context (no-op span, missing/zero/malformed ids). + * Never includes tracestate. */ @Nullable String traceParent(); diff --git a/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java index 99324750344..72dc47482f7 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java @@ -23,48 +23,53 @@ import com.mongodb.observability.micrometer.MongodbObservation; import org.junit.jupiter.api.Test; -import java.util.regex.Pattern; - -import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertNull; -import static org.junit.jupiter.api.Assertions.assertTrue; class MicrometerTraceParentTest { - // SimpleTracer generates 8-byte (16-hex-char) trace IDs; real OTel uses 16-byte (32-hex-char). - // Accept either length in the unit test — the format check is what matters here. - private static final Pattern TRACEPARENT = - Pattern.compile("00-[0-9a-f]{16,32}-[0-9a-f]{16}-[0-9a-f]{2}"); + private static final String VALID_TRACE_ID = "0af7651916cd43dd8448eb211c80319c"; + private static final String VALID_SPAN_ID = "b7ad6b7169203331"; @Test - void returnsTraceParentForSampledSpan() { - ObservationRegistry registry = ObservationRegistry.create(); - SimpleTracer tracer = new SimpleTracer(); - registry.observationConfig().observationHandler(new DefaultTracingObservationHandler(tracer)); + void shouldFormatSampledTraceParent() { + String traceParent = traceParentFor(VALID_TRACE_ID, VALID_SPAN_ID, true); + assertEquals("00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01", traceParent); + assertEquals(55, traceParent.length()); + } - MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); - Span span = micrometerTracer.nextSpan(MongodbObservation.MONGODB_COMMAND, "find", null, null); - span.openScope(); - // SimpleTracer creates spans with sampled=false by default; mark as sampled so - // traceParent() emits the header (flags=01). - ((SimpleTraceContext) tracer.lastSpan().context()).setSampled(true); - try { - String traceParent = span.context().traceParent(); - assertNotNull(traceParent); - assertTrue(TRACEPARENT.matcher(traceParent).matches(), traceParent); - } finally { - span.closeScope(); - span.end(); - } + @Test + void shouldFormatUnsampledTraceParentWithZeroFlags() { + assertEquals("00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-00", + traceParentFor(VALID_TRACE_ID, VALID_SPAN_ID, false)); + assertEquals("00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-00", + traceParentFor(VALID_TRACE_ID, VALID_SPAN_ID, null)); } @Test - void returnsNullWhenNoTracingBridgeConfigured() { + void shouldReturnNullForInvalidIds() { + assertNull(traceParentFor("00000000000000000000000000000000", VALID_SPAN_ID, true)); + assertNull(traceParentFor(VALID_TRACE_ID, "0000000000000000", true)); + assertNull(traceParentFor("abc", VALID_SPAN_ID, true)); + assertNull(traceParentFor(VALID_TRACE_ID, "abc", true)); + assertNull(traceParentFor("0AF7651916CD43DD8448EB211C80319C", VALID_SPAN_ID, true)); + assertNull(traceParentFor(null, VALID_SPAN_ID, true)); + assertNull(traceParentFor(VALID_TRACE_ID, null, true)); + } + + private static String traceParentFor(final String traceId, final String spanId, final Boolean sampled) { ObservationRegistry registry = ObservationRegistry.create(); + SimpleTracer tracer = new SimpleTracer(); + registry.observationConfig().observationHandler(new DefaultTracingObservationHandler(tracer)); + MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); Span span = micrometerTracer.nextSpan(MongodbObservation.MONGODB_COMMAND, "find", null, null); span.openScope(); try { - assertNull(span.context().traceParent()); + SimpleTraceContext context = (SimpleTraceContext) tracer.lastSpan().context(); + context.setTraceId(traceId); + context.setSpanId(spanId); + context.setSampled(sampled); + return span.context().traceParent(); } finally { span.closeScope(); span.end(); From 9bd5c65513eeea9fe5c776888a33762edc4eb7c2 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 14 Jul 2026 13:49:14 +0100 Subject: [PATCH 08/48] DRIVERS-3454: write BSON telemetry section gated on maxWireVersion >= 29 --- .../internal/connection/CommandMessage.java | 29 ++- .../CommandMessageOtelTraceContextTest.java | 210 ++++++++++-------- 2 files changed, 138 insertions(+), 101 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java index d3d779ac984..81ac10536aa 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +++ b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java @@ -25,7 +25,6 @@ import com.mongodb.internal.MongoNamespaceHelper; import com.mongodb.internal.TimeoutContext; import com.mongodb.internal.connection.MessageSequences.EmptyMessageSequences; -import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; import com.mongodb.internal.observability.micrometer.Span; import com.mongodb.internal.session.SessionContext; import com.mongodb.lang.Nullable; @@ -38,6 +37,8 @@ import org.bson.BsonString; import org.bson.ByteBuf; import org.bson.FieldNameValidator; +import org.bson.codecs.BsonDocumentCodec; +import org.bson.codecs.EncoderContext; import org.bson.io.BsonOutput; import java.io.ByteArrayOutputStream; @@ -66,6 +67,7 @@ import static com.mongodb.internal.connection.ByteBufBsonDocument.createList; import static com.mongodb.internal.connection.ByteBufBsonDocument.createOne; import static com.mongodb.internal.connection.ReadConcernHelper.getReadConcernDocument; +import static com.mongodb.internal.operation.ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION; import static com.mongodb.internal.operation.ServerVersionHelper.UNKNOWN_WIRE_VERSION; /** @@ -83,11 +85,11 @@ public final class CommandMessage extends RequestMessage { */ private static final byte PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE = 1; /** - * Specifies that the `OP_MSG` section payload is a W3C traceparent C-string (OpenTelemetry trace context). - * - *

Mirrors the server's {@code kOtelTelemetryContext = 3} section kind (see DRIVERS-3454).

+ * Specifies that the `OP_MSG` section payload is a BSON telemetry document + * ({@code {otel: {traceparent: }}}). Mirrors the server's {@code kTelemetry = 3} + * section kind (DRIVERS-3454, 10gen/mongo#56646). */ - private static final byte PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT = 3; + private static final byte PAYLOAD_TYPE_3_TELEMETRY = 3; private static final int UNINITIALIZED_POSITION = -1; @@ -165,7 +167,7 @@ BsonDocument getCommandDocument(final ByteBufferBsonOutput bsonOutput) { ByteBufBsonDocument byteBufBsonDocument = createOne(byteBuf); // If true, there are more sections after the `PAYLOAD_TYPE_0_DOCUMENT` section: either one or more - // `PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE` sections, and/or a trailing `PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT` section. + // `PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE` sections, and/or a trailing `PAYLOAD_TYPE_3_TELEMETRY` section. if (byteBuf.hasRemaining()) { BsonDocument commandBsonDocument = byteBufBsonDocument.toBaseBsonDocument(); @@ -174,7 +176,7 @@ BsonDocument getCommandDocument(final ByteBufferBsonOutput bsonOutput) { while (byteBuf.hasRemaining()) { byte payloadType = byteBuf.get(); // Document-sequence sections always precede any trailing non-sequence section (e.g. - // PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT), and such sections carry no command-document fields, so stop here. + // PAYLOAD_TYPE_3_TELEMETRY), and such sections carry no command-document fields, so stop here. if (payloadType != PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE) { break; } @@ -290,15 +292,15 @@ private int writeOpMsg(final ByteBufferBsonOutput bsonOutput, final OperationCon fail(sequences.toString()); } - writeOtelTraceContextSection(bsonOutput, operationContext); + writeTelemetryContextSection(bsonOutput, operationContext); // Write the flag bits bsonOutput.writeInt32(flagPosition, getOpMsgFlagBits()); return commandStartPosition; } - private void writeOtelTraceContextSection(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext) { - if (!getSettings().isTracingSupported() && !OtelTracePropagationTestToggle.FORCE_PROPAGATION) { + private void writeTelemetryContextSection(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext) { + if (getSettings().getMaxWireVersion() < NINE_DOT_ZERO_WIRE_VERSION) { return; } Span tracingSpan = operationContext.getTracingSpan(); @@ -309,8 +311,11 @@ private void writeOtelTraceContextSection(final ByteBufferBsonOutput bsonOutput, if (traceParent == null) { return; } - bsonOutput.writeByte(PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT); - bsonOutput.writeCString(traceParent); + bsonOutput.writeByte(PAYLOAD_TYPE_3_TELEMETRY); + BsonDocument telemetry = new BsonDocument("otel", + new BsonDocument("traceparent", new BsonString(traceParent))); + new BsonDocumentCodec().encode(new BsonBinaryWriter(bsonOutput), telemetry, + EncoderContext.builder().build()); } private int writeOpQuery(final ByteBufferBsonOutput bsonOutput) { diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java index 5f941327699..d1baf2f201c 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java @@ -23,23 +23,30 @@ import com.mongodb.connection.ServerType; import com.mongodb.internal.TimeoutContext; import com.mongodb.internal.TimeoutSettings; +import com.mongodb.internal.bulk.InsertRequest; +import com.mongodb.internal.bulk.WriteRequestWithIndex; import com.mongodb.internal.connection.MessageSequences.EmptyMessageSequences; -import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; import com.mongodb.internal.observability.micrometer.Span; import com.mongodb.internal.observability.micrometer.TraceContext; import com.mongodb.internal.session.SessionContext; import com.mongodb.internal.validator.NoOpFieldNameValidator; +import org.bson.BsonBinaryReader; import org.bson.BsonDocument; import org.bson.BsonString; +import org.bson.ByteBuf; +import org.bson.ByteBufNIO; +import org.bson.codecs.BsonDocumentCodec; +import org.bson.codecs.DecoderContext; import org.junit.jupiter.api.Test; -import java.nio.charset.StandardCharsets; +import java.nio.ByteBuffer; +import java.util.Collections; import static com.mongodb.internal.mockito.MongoMockito.mock; -import static com.mongodb.internal.operation.ServerVersionHelper.LATEST_WIRE_VERSION; +import static com.mongodb.internal.operation.ServerVersionHelper.EIGHT_DOT_ZERO_WIRE_VERSION; +import static com.mongodb.internal.operation.ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION; import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertFalse; -import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.assertNull; import static org.mockito.Mockito.when; class CommandMessageOtelTraceContextTest { @@ -49,109 +56,124 @@ class CommandMessageOtelTraceContextTest { private static final BsonDocument COMMAND = new BsonDocument("find", new BsonString(NAMESPACE.getCollectionName())); @Test - void writesSectionWhenSupportedAndSampledSpanPresent() { - CommandMessage message = buildCommandMessage(true); - TraceContext traceContext = () -> TRACEPARENT; - Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + void shouldWriteTelemetrySectionWhenWireVersionAtLeast29AndSpanActive() { + CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); + Span span = spanWithTraceParent(TRACEPARENT); OperationContext operationContext = buildOperationContext(span); - byte[] encoded = encodeToBytes(message, operationContext); + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + byte[] buffer = output.toByteArray(); - assertTrue(containsOtelSection(encoded, TRACEPARENT), - "Encoded message should contain the OTel trace context section (kind byte 3 + traceparent C-string)"); + BsonDocument telemetry = readTelemetrySectionDocument(buffer); + assertEquals(new BsonDocument("otel", new BsonDocument("traceparent", new BsonString(TRACEPARENT))), telemetry); + } } @Test - void getCommandDocumentIgnoresOtelSection() { - // Regression guard: InternalStreamConnection calls getCommandDocument() on every send (logging/monitoring/ - // compression). The trailing kind-3 section must not corrupt command-document reconstruction. - CommandMessage message = buildCommandMessage(true); - TraceContext traceContext = () -> TRACEPARENT; - Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + void shouldNotWriteTelemetrySectionWhenWireVersionBelow29() { + CommandMessage message = buildCommandMessage(EIGHT_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); + Span span = spanWithTraceParent(TRACEPARENT); OperationContext operationContext = buildOperationContext(span); try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { message.encode(output, operationContext); - BsonDocument commandDocument = message.getCommandDocument(output); - assertEquals("test", commandDocument.getString("find").getValue()); - assertEquals("db", commandDocument.getString("$db").getValue()); - // The reconstructed command is exactly the body section (find + $db); the trailing - // kind-3 section must not leak any extra fields into it. - assertEquals(2, commandDocument.size()); + byte[] buffer = output.toByteArray(); + + assertNull(findTelemetrySectionDocument(buffer)); } } @Test - void omitsSectionWhenCapabilityAbsent() { - CommandMessage message = buildCommandMessage(false); - TraceContext traceContext = () -> TRACEPARENT; - Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); - OperationContext operationContext = buildOperationContext(span); + void shouldNotWriteTelemetrySectionWhenNoSpan() { + CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); + OperationContext operationContext = buildOperationContext(null); - byte[] encoded = encodeToBytes(message, operationContext); + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + byte[] buffer = output.toByteArray(); - assertFalse(containsOtelSection(encoded, TRACEPARENT), - "Encoded message should NOT contain the OTel trace context section when tracingSupported=false"); + assertNull(findTelemetrySectionDocument(buffer)); + } } @Test - void omitsSectionWhenSpanHasNoTraceParent() { - CommandMessage message = buildCommandMessage(true); - TraceContext traceContext = () -> null; - Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + void shouldNotWriteTelemetrySectionWhenTraceParentNull() { + CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); + Span span = spanWithTraceParent(null); OperationContext operationContext = buildOperationContext(span); - byte[] encoded = encodeToBytes(message, operationContext); + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + byte[] buffer = output.toByteArray(); - assertFalse(containsOtelSection(encoded, TRACEPARENT), - "Encoded message should NOT contain the OTel trace context section when traceParent is null"); + assertNull(findTelemetrySectionDocument(buffer)); + } } @Test - void omitsSectionWhenSpanIsNull() { - CommandMessage message = buildCommandMessage(true); - OperationContext operationContext = buildOperationContext(null); - - byte[] encoded = encodeToBytes(message, operationContext); + void getCommandDocumentIgnoresTelemetrySection() { + // Regression guard: InternalStreamConnection calls getCommandDocument() on every send (logging/monitoring/ + // compression). The trailing kind-3 section must not corrupt command-document reconstruction. + CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); + Span span = spanWithTraceParent(TRACEPARENT); + OperationContext operationContext = buildOperationContext(span); - assertFalse(containsOtelSection(encoded, TRACEPARENT), - "Encoded message should NOT contain the OTel trace context section when there is no tracing span"); + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + BsonDocument commandDocument = message.getCommandDocument(output); + assertEquals("test", commandDocument.getString("find").getValue()); + assertEquals("db", commandDocument.getString("$db").getValue()); + // The reconstructed command is exactly the body section (find + $db); the trailing + // kind-3 section must not leak any extra fields into it. + assertEquals(2, commandDocument.size()); + } } @Test - void writesSectionWhenForcedEvenIfCapabilityAbsent() { - OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; - try { - CommandMessage message = buildCommandMessage(false); // server did NOT advertise tracingSupport - TraceContext traceContext = () -> TRACEPARENT; - Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); - OperationContext operationContext = buildOperationContext(span); - - byte[] encoded = encodeToBytes(message, operationContext); - - assertTrue(containsOtelSection(encoded, TRACEPARENT), - "With FORCE_PROPAGATION the section must be sent even when the server did not advertise support"); - } finally { - OtelTracePropagationTestToggle.FORCE_PROPAGATION = false; + void shouldWriteTelemetrySectionAfterDocumentSequences() { + SplittablePayload payload = new SplittablePayload(SplittablePayload.Type.INSERT, + Collections.singletonList(new WriteRequestWithIndex( + new InsertRequest(new BsonDocument("_id", new BsonString("1"))), 0)), + true, NoOpFieldNameValidator.INSTANCE); + CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, payload); + Span span = spanWithTraceParent(TRACEPARENT); + OperationContext operationContext = buildOperationContext(span); + + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + byte[] buffer = output.toByteArray(); + + // Sequence section(s) precede the kind-3 telemetry section; scanning left-to-right and taking + // the first kind-3 occurrence therefore validates ordering as well as content. + BsonDocument telemetry = readTelemetrySectionDocument(buffer); + assertEquals(new BsonDocument("otel", new BsonDocument("traceparent", new BsonString(TRACEPARENT))), telemetry); + + BsonDocument commandDocument = message.getCommandDocument(output); + assertEquals("test", commandDocument.getString("find").getValue()); } } // --- helpers --- - private static CommandMessage buildCommandMessage(final boolean tracingSupported) { + private static Span spanWithTraceParent(final String traceParent) { + TraceContext traceContext = () -> traceParent; + return mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + } + + private static CommandMessage buildCommandMessage(final int maxWireVersion, final MessageSequences sequences) { return new CommandMessage( NAMESPACE.getDatabaseName(), COMMAND, NoOpFieldNameValidator.INSTANCE, ReadPreference.primary(), MessageSettings.builder() - .maxWireVersion(LATEST_WIRE_VERSION) + .maxWireVersion(maxWireVersion) .serverType(ServerType.REPLICA_SET_PRIMARY) .sessionSupported(true) - .tracingSupported(tracingSupported) .build(), true, - EmptyMessageSequences.INSTANCE, + sequences, ClusterConnectionMode.MULTIPLE, null); } @@ -173,35 +195,45 @@ private static OperationContext buildOperationContext(final Span span) { }); } - private static byte[] encodeToBytes(final CommandMessage message, final OperationContext operationContext) { - try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { - message.encode(output, operationContext); - return output.toByteArray(); - } - } - /** - * Searches for the needle: kind byte {@code 3} immediately followed by the UTF-8 bytes of - * {@code traceparent} and a trailing null byte (C-string terminator). + * Walks the OP_MSG sections exactly like the production parser (see {@code CommandMessage#getCommandDocument}): + * skip the type-0 body document, then for each subsequent section read the kind byte; for kind 1 + * (document sequence) skip past the {@code int32} section size, and for kind 3 (telemetry) decode and + * return the BSON document payload. Returns {@code null} if no kind-3 section is found. */ - private static boolean containsOtelSection(final byte[] encoded, final String traceparent) { - byte[] traceparentBytes = traceparent.getBytes(StandardCharsets.UTF_8); - // needle = [0x03, tp[0], tp[1], ..., tp[n-1], 0x00] - int needleLen = 1 + traceparentBytes.length + 1; - outer: - for (int i = 0; i <= encoded.length - needleLen; i++) { - if (encoded[i] != 3) { - continue; - } - for (int j = 0; j < traceparentBytes.length; j++) { - if (encoded[i + 1 + j] != traceparentBytes[j]) { - continue outer; + private static BsonDocument findTelemetrySectionDocument(final byte[] buffer) { + ByteBuf byteBuf = new ByteBufNIO(ByteBuffer.wrap(buffer)); + // MsgHeader (16 bytes) + flagBits (4 bytes) + payload type byte (1 byte) for the body section. + int position = 16 + 4 + 1; + byteBuf.position(position); + int bodyLength = byteBuf.getInt(byteBuf.position()); + byteBuf.position(byteBuf.position() + bodyLength); + + while (byteBuf.hasRemaining()) { + byte kind = byteBuf.get(); + if (kind == 1) { + int sectionStart = byteBuf.position(); + int sectionSize = byteBuf.getInt(); + byteBuf.position(sectionStart + sectionSize); + } else if (kind == 3) { + BsonBinaryReader reader = new BsonBinaryReader(byteBuf.asNIO()); + try { + return new BsonDocumentCodec().decode(reader, DecoderContext.builder().build()); + } finally { + reader.close(); } + } else { + throw new AssertionError("Unexpected section kind byte: " + kind); } - if (encoded[i + 1 + traceparentBytes.length] == 0) { - return true; - } } - return false; + return null; + } + + private static BsonDocument readTelemetrySectionDocument(final byte[] buffer) { + BsonDocument telemetry = findTelemetrySectionDocument(buffer); + if (telemetry == null) { + throw new AssertionError("Expected a kind-3 telemetry section but none was found"); + } + return telemetry; } } From 9931e8932561873948dc54a9b02a6c6542b6044f Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 14 Jul 2026 13:52:12 +0100 Subject: [PATCH 09/48] DRIVERS-3454: remove hello-capability plumbing and test toggle (wire-version gate supersedes) --- .../connection/ConnectionDescription.java | 50 +----------- .../connection/DescriptionHelper.java | 5 -- .../internal/connection/MessageSettings.java | 12 --- .../internal/connection/ProtocolHelper.java | 1 - .../OtelTracePropagationTestToggle.java | 32 -------- .../DescriptionHelperTracingTest.java | 81 ------------------- 6 files changed, 2 insertions(+), 179 deletions(-) delete mode 100644 driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java delete mode 100644 driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java diff --git a/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java b/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java index b62cc018595..c8c213398b7 100644 --- a/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java +++ b/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java @@ -48,7 +48,6 @@ public class ConnectionDescription { private final List compressors; private final BsonArray saslSupportedMechanisms; private final Integer logicalSessionTimeoutMinutes; - private final boolean tracingSupport; private static final int DEFAULT_MAX_MESSAGE_SIZE = 0x2000000; // 32MB private static final int DEFAULT_MAX_WRITE_BATCH_SIZE = 512; @@ -151,15 +150,6 @@ private ConnectionDescription(@Nullable final ObjectId serviceId, final Connecti final ServerType serverType, final int maxBatchCount, final int maxDocumentSize, final int maxMessageSize, final List compressors, @Nullable final BsonArray saslSupportedMechanisms, @Nullable final Integer logicalSessionTimeoutMinutes) { - this(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, maxMessageSize, compressors, - saslSupportedMechanisms, logicalSessionTimeoutMinutes, false); - } - - private ConnectionDescription(@Nullable final ObjectId serviceId, final ConnectionId connectionId, final int maxWireVersion, - final ServerType serverType, final int maxBatchCount, final int maxDocumentSize, - final int maxMessageSize, final List compressors, - @Nullable final BsonArray saslSupportedMechanisms, @Nullable final Integer logicalSessionTimeoutMinutes, - final boolean tracingSupport) { this.serviceId = serviceId; this.connectionId = connectionId; this.serverType = serverType; @@ -170,7 +160,6 @@ private ConnectionDescription(@Nullable final ObjectId serviceId, final Connecti this.compressors = notNull("compressors", Collections.unmodifiableList(new ArrayList<>(compressors))); this.saslSupportedMechanisms = saslSupportedMechanisms; this.logicalSessionTimeoutMinutes = logicalSessionTimeoutMinutes; - this.tracingSupport = tracingSupport; } /** * Creates a new connection description with the set connection id @@ -182,7 +171,7 @@ private ConnectionDescription(@Nullable final ObjectId serviceId, final Connecti public ConnectionDescription withConnectionId(final ConnectionId connectionId) { notNull("connectionId", connectionId); return new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, - maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes, tracingSupport); + maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes); } /** @@ -195,22 +184,7 @@ public ConnectionDescription withConnectionId(final ConnectionId connectionId) { public ConnectionDescription withServiceId(final ObjectId serviceId) { notNull("serviceId", serviceId); return new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, - maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes, tracingSupport); - } - - /** - * Creates a new connection description with the given tracing support flag. - * - *

A value of {@code true} indicates that the server advertised OpenTelemetry trace-context support in its {@code hello} response - * and that the driver may send trace context to the server over OP_MSG.

- * - * @param tracingSupport whether the server supports OpenTelemetry trace-context propagation - * @return the new connection description - * @since 5.9 - */ - public ConnectionDescription withTracingSupport(final boolean tracingSupport) { - return new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, - maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes, tracingSupport); + maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes); } /** @@ -319,21 +293,6 @@ public BsonArray getSaslSupportedMechanisms() { public Integer getLogicalSessionTimeoutMinutes() { return logicalSessionTimeoutMinutes; } - - /** - * Returns whether the server advertised OpenTelemetry trace-context support in its {@code hello} response. - * - *

When {@code true}, the driver may propagate trace context to the server over OP_MSG. - * When {@code false} (the default), the server did not advertise this capability and sending unknown OP_MSG sections would cause - * the server to reject the message.

- * - * @return {@code true} if the server supports OpenTelemetry trace-context propagation - * @since 5.9 - */ - public boolean isTracingSupported() { - return tracingSupport; - } - /** * Get the default maximum message size. * @@ -391,9 +350,6 @@ public boolean equals(final Object o) { if (!Objects.equals(logicalSessionTimeoutMinutes, that.logicalSessionTimeoutMinutes)) { return false; } - if (tracingSupport != that.tracingSupport) { - return false; - } return Objects.equals(saslSupportedMechanisms, that.saslSupportedMechanisms); } @@ -409,7 +365,6 @@ public int hashCode() { result = 31 * result + (serviceId != null ? serviceId.hashCode() : 0); result = 31 * result + (saslSupportedMechanisms != null ? saslSupportedMechanisms.hashCode() : 0); result = 31 * result + (logicalSessionTimeoutMinutes != null ? logicalSessionTimeoutMinutes.hashCode() : 0); - result = 31 * result + (tracingSupport ? 1 : 0); return result; } @@ -425,7 +380,6 @@ public String toString() { + ", compressors=" + compressors + ", logicialSessionTimeoutMinutes=" + logicalSessionTimeoutMinutes + ", serviceId=" + serviceId - + ", tracingSupport=" + tracingSupport + '}'; } } diff --git a/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java b/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java index a070d8bbdf4..26f73bcee9c 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java +++ b/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java @@ -69,7 +69,6 @@ static ConnectionDescription createConnectionDescription(final ClusterConnection getMaxWireVersion(helloResult), getServerType(helloResult), getMaxWriteBatchSize(helloResult), getMaxBsonObjectSize(helloResult), getMaxMessageSizeBytes(helloResult), getCompressors(helloResult), helloResult.getArray("saslSupportedMechs", null), getLogicalSessionTimeoutMinutes(helloResult)); - connectionDescription = connectionDescription.withTracingSupport(getTracingSupport(helloResult)); if (helloResult.containsKey("connectionId")) { ConnectionId newConnectionId = connectionDescription.getConnectionId().withServerValue(helloResult.getNumber("connectionId").longValue()); @@ -171,10 +170,6 @@ private static Integer getLogicalSessionTimeoutMinutes(final BsonDocument helloR ? helloResult.getNumber("logicalSessionTimeoutMinutes").intValue() : null; } - private static boolean getTracingSupport(final BsonDocument helloResult) { - return helloResult.getBoolean("tracingSupport", BsonBoolean.FALSE).getValue(); - } - @Nullable private static String getString(final BsonDocument response, final String key) { if (response.containsKey(key)) { diff --git a/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java b/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java index 6eaad0ea379..51587e8f91d 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java +++ b/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java @@ -58,7 +58,6 @@ public final class MessageSettings { private final int maxWireVersion; private final ServerType serverType; private final boolean sessionSupported; - private final boolean tracingSupported; private final boolean cryptd; /** @@ -81,7 +80,6 @@ public static final class Builder { private int maxWireVersion = UNKNOWN_WIRE_VERSION; private ServerType serverType; private boolean sessionSupported; - private boolean tracingSupported; private boolean cryptd; /** @@ -141,11 +139,6 @@ public Builder sessionSupported(final boolean sessionSupported) { return this; } - public Builder tracingSupported(final boolean tracingSupported) { - this.tracingSupported = tracingSupported; - return this; - } - /** * Set whether the server is a mongocryptd. * @@ -200,10 +193,6 @@ public boolean isSessionSupported() { return sessionSupported; } - public boolean isTracingSupported() { - return tracingSupported; - } - private MessageSettings(final Builder builder) { this.maxDocumentSize = builder.maxDocumentSize; @@ -212,7 +201,6 @@ private MessageSettings(final Builder builder) { this.maxWireVersion = builder.maxWireVersion; this.serverType = builder.serverType; this.sessionSupported = builder.sessionSupported; - this.tracingSupported = builder.tracingSupported; this.cryptd = builder.cryptd; } } diff --git a/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java b/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java index a519252dfed..c6ad5f451a0 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java +++ b/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java @@ -235,7 +235,6 @@ static MessageSettings getMessageSettings(final ConnectionDescription connection .maxWireVersion(connectionDescription.getMaxWireVersion()) .serverType(connectionDescription.getServerType()) .sessionSupported(connectionDescription.getLogicalSessionTimeoutMinutes() != null) - .tracingSupported(connectionDescription.isTracingSupported()) .cryptd(serverDescription.isCryptd()) .build(); } diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java deleted file mode 100644 index 96d5b767121..00000000000 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java +++ /dev/null @@ -1,32 +0,0 @@ -/* - * Copyright 2008-present MongoDB, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.mongodb.internal.observability.micrometer; - -/** - * TEST-ONLY switch (DRIVERS-3454): when {@code true}, the driver writes the OP_MSG OpenTelemetry - * trace-context section even if the server did not advertise {@code tracingSupport} in its - * {@code hello} response. The sampled-{@code traceparent} requirement still applies. - * - *

This exists only to exercise end-to-end propagation against a server that does not yet advertise - * the capability. Remove before any production use.

- */ -public final class OtelTracePropagationTestToggle { - public static volatile boolean FORCE_PROPAGATION = false; - - private OtelTracePropagationTestToggle() { - } -} diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java deleted file mode 100644 index 1af0ca854fa..00000000000 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java +++ /dev/null @@ -1,81 +0,0 @@ -/* - * Copyright 2008-present MongoDB, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.mongodb.internal.connection; - -import com.mongodb.ServerAddress; -import com.mongodb.connection.ClusterConnectionMode; -import com.mongodb.connection.ClusterId; -import com.mongodb.connection.ConnectionDescription; -import com.mongodb.connection.ConnectionId; -import com.mongodb.connection.ServerId; -import org.bson.BsonBoolean; -import org.bson.BsonDocument; -import org.junit.jupiter.api.Test; - -import static org.junit.jupiter.api.Assertions.assertEquals; -import static org.junit.jupiter.api.Assertions.assertFalse; -import static org.junit.jupiter.api.Assertions.assertTrue; - -class DescriptionHelperTracingTest { - private static final ConnectionId CONNECTION_ID = - new ConnectionId(new ServerId(new ClusterId(), new ServerAddress())); - - private static BsonDocument hello(final boolean tracing) { - BsonDocument doc = BsonDocument.parse( - "{ ok: 1, ismaster: true, maxWireVersion: 25, minWireVersion: 0," - + " maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, maxWriteBatchSize: 100000 }"); - if (tracing) { - doc.put("tracingSupport", BsonBoolean.TRUE); - } - return doc; - } - - @Test - void parsesTracingSupportTrue() { - ConnectionDescription description = DescriptionHelper.createConnectionDescription( - ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(true)); - assertTrue(description.isTracingSupported()); - } - - @Test - void defaultsTracingSupportFalseWhenAbsent() { - ConnectionDescription description = DescriptionHelper.createConnectionDescription( - ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(false)); - assertFalse(description.isTracingSupported()); - } - - @Test - void parsesTracingSupportFalseWhenExplicitlyFalse() { - BsonDocument helloResult = hello(false); - helloResult.put("tracingSupport", BsonBoolean.FALSE); - ConnectionDescription description = DescriptionHelper.createConnectionDescription( - ClusterConnectionMode.SINGLE, CONNECTION_ID, helloResult); - assertFalse(description.isTracingSupported()); - } - - @Test - void withTracingSupportPreservesOtherFields() { - ConnectionDescription original = DescriptionHelper.createConnectionDescription( - ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(false)); - ConnectionDescription updated = original.withTracingSupport(true); - - assertTrue(updated.isTracingSupported()); - assertEquals(original.getConnectionId(), updated.getConnectionId()); - assertEquals(original.getMaxWireVersion(), updated.getMaxWireVersion()); - assertEquals(original.getServerType(), updated.getServerType()); - } -} From d943d9ec3ee1def94f697d86a722b6faaa860821 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 14 Jul 2026 13:58:01 +0100 Subject: [PATCH 10/48] DRIVERS-3454: prose-test definitions for trace-context propagation --- ...7-13-otel-telemetry-section-prose-tests.md | 39 +++++++++++++++++++ .../CommandMessageOtelTraceContextTest.java | 16 ++++++++ .../micrometer/MicrometerTraceParentTest.java | 4 ++ 3 files changed, 59 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md diff --git a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md new file mode 100644 index 00000000000..862074ffc2d --- /dev/null +++ b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md @@ -0,0 +1,39 @@ +# Trace-Context Propagation Prose Tests (DRIVERS-3454) + +These tests complement the OpenTelemetry spec prose tests. Tests 1–4 are unit-level and +MUST NOT require a server. Test 5 requires a MongoDB 9.0+ server (maxWireVersion >= 29) +started with OTel tracing enabled and the OTLP file exporter +(`--setParameter opentelemetryTraceDirectory=` plus the tracing feature flag and +sampling parameters), running on the same host as the test. + +## 1. Telemetry section is attached when supported and traced + +With tracing enabled and an active operation span, encode a command targeting a +connection whose `maxWireVersion` is >= 29. Assert the resulting OP_MSG contains exactly +one section of kind 3 whose payload is a BSON document of the form +`{otel: {traceparent: }}`, where `traceparent` is 55 characters, +`00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>`, and the trace-id/span-id match +the active span's context. + +## 2. Telemetry section is omitted for older servers + +Same as (1) but with `maxWireVersion` < 29. Assert no section of kind 3 is present. + +## 3. Telemetry section is omitted without an active span + +With tracing disabled (or no active span, e.g. monitoring/auth commands), encode a +command at `maxWireVersion` >= 29. Assert no section of kind 3 is present. + +## 4. Malformed trace context is never sent + +With an active span whose context cannot produce a valid W3C traceparent (zero trace-id, +zero span-id, wrong-length or non-lowercase-hex ids), assert no section of kind 3 is +present (drivers MUST omit the section rather than send an invalid traceparent). + +## 5. End-to-end server span linkage + +With tracing enabled, run a CRUD operation (e.g. `find`) against the configured 9.0+ +server. Capture the driver's finished command span (client side). Then read the server's +exported OTLP JSON from the trace directory and assert a server span exists whose +`traceId` equals the client command span's trace-id and whose `parentSpanId` equals the +client command span's span-id. Allow for the server's batch export interval when polling. diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java index d1baf2f201c..be17456e791 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java @@ -55,6 +55,10 @@ class CommandMessageOtelTraceContextTest { private static final MongoNamespace NAMESPACE = new MongoNamespace("db.test"); private static final BsonDocument COMMAND = new BsonDocument("find", new BsonString(NAMESPACE.getCollectionName())); + /** + * Prose test 1: Telemetry section is attached when supported and traced + * (see {@code docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md}) + */ @Test void shouldWriteTelemetrySectionWhenWireVersionAtLeast29AndSpanActive() { CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); @@ -70,6 +74,10 @@ void shouldWriteTelemetrySectionWhenWireVersionAtLeast29AndSpanActive() { } } + /** + * Prose test 2: Telemetry section is omitted for older servers + * (see {@code docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md}) + */ @Test void shouldNotWriteTelemetrySectionWhenWireVersionBelow29() { CommandMessage message = buildCommandMessage(EIGHT_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); @@ -84,6 +92,10 @@ void shouldNotWriteTelemetrySectionWhenWireVersionBelow29() { } } + /** + * Prose test 3: Telemetry section is omitted without an active span + * (see {@code docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md}) + */ @Test void shouldNotWriteTelemetrySectionWhenNoSpan() { CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); @@ -97,6 +109,10 @@ void shouldNotWriteTelemetrySectionWhenNoSpan() { } } + /** + * Prose test 4: Malformed trace context is never sent + * (see {@code docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md}) + */ @Test void shouldNotWriteTelemetrySectionWhenTraceParentNull() { CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); diff --git a/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java index 72dc47482f7..22485a9f28d 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java @@ -45,6 +45,10 @@ void shouldFormatUnsampledTraceParentWithZeroFlags() { traceParentFor(VALID_TRACE_ID, VALID_SPAN_ID, null)); } + /** + * Prose test 4: Malformed trace context is never sent + * (see {@code docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md}) + */ @Test void shouldReturnNullForInvalidIds() { assertNull(traceParentFor("00000000000000000000000000000000", VALID_SPAN_ID, true)); From 0176242a56d810165c3dccf6f6733d96d0e56332 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 14 Jul 2026 14:03:38 +0100 Subject: [PATCH 11/48] DRIVERS-3454: e2e server-span linkage prose test (OTLP file exporter) --- .../ServerSpanLinkageProseTest.java | 135 ++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100644 driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java diff --git a/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java b/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java new file mode 100644 index 00000000000..6db26354ca7 --- /dev/null +++ b/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java @@ -0,0 +1,135 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.client.observability; + +import com.mongodb.MongoClientSettings; +import com.mongodb.client.MongoClient; +import com.mongodb.client.MongoClients; +import com.mongodb.client.MongoCollection; +import com.mongodb.observability.ObservabilitySettings; +import io.micrometer.observation.ObservationRegistry; +import io.micrometer.tracing.exporter.FinishedSpan; +import io.micrometer.tracing.test.reporter.inmemory.InMemoryOtelSetup; +import org.bson.Document; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import static com.mongodb.ClusterFixture.getDefaultDatabaseName; +import static com.mongodb.client.Fixture.getMongoClientSettingsBuilder; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; + +/** + * Prose test 5 (docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md): + * end-to-end server span linkage via the server's OTLP file exporter. + * + *

Requires a MongoDB 9.0+ (maxWireVersion >= 29) server on the same host, started with + * {@code --setParameter opentelemetryTraceDirectory=

} (plus tracing feature flag and + * sampling parameters), and the test run with + * {@code -Dorg.mongodb.test.otel.trace.dir=}. Skipped otherwise.

+ */ +@EnabledIfSystemProperty(named = "org.mongodb.test.otel.trace.dir", matches = ".+") +public class ServerSpanLinkageProseTest { + + private static final long EXPORT_POLL_TIMEOUT_MS = 30_000; + private static final long EXPORT_POLL_INTERVAL_MS = 1_000; + + private final ObservationRegistry observationRegistry = ObservationRegistry.create(); + private InMemoryOtelSetup memoryOtelSetup; + private InMemoryOtelSetup.Builder.OtelBuildingBlocks inMemoryOtel; + + @BeforeEach + void setUp() { + memoryOtelSetup = InMemoryOtelSetup.builder().register(observationRegistry); + inMemoryOtel = memoryOtelSetup.getBuildingBlocks(); + } + + @AfterEach + void tearDown() { + memoryOtelSetup.close(); + } + + @Test + @DisplayName("Prose test 5: server emits a child span of the driver command span") + void testServerSpanLinkage() throws Exception { + MongoClientSettings clientSettings = getMongoClientSettingsBuilder() + .observabilitySettings(ObservabilitySettings.micrometerBuilder() + .observationRegistry(observationRegistry) + .build()) + .build(); + + try (MongoClient client = MongoClients.create(clientSettings)) { + MongoCollection collection = + client.getDatabase(getDefaultDatabaseName()).getCollection("serverSpanLinkage"); + collection.find().first(); + } + + List clientSpans = inMemoryOtel.getFinishedSpans(); + assertTrue(clientSpans.size() >= 2, "expected operation + command client spans, got: " + clientSpans); + // command span is the innermost (first finished) span; see AbstractMicrometerProseTest ordering + FinishedSpan commandSpan = clientSpans.get(0); + String traceId = commandSpan.getTraceId(); + String commandSpanId = commandSpan.getSpanId(); + + Path traceDir = Paths.get(System.getProperty("org.mongodb.test.otel.trace.dir")); + long deadline = System.currentTimeMillis() + EXPORT_POLL_TIMEOUT_MS; + while (System.currentTimeMillis() < deadline) { + if (serverSpanLinked(traceDir, traceId, commandSpanId)) { + return; + } + Thread.sleep(EXPORT_POLL_INTERVAL_MS); + } + fail("no server span found in " + traceDir + " with traceId=" + traceId + + " and parentSpanId=" + commandSpanId); + } + + /** + * Scans OTLP JSON export files for a span with the given traceId whose parentSpanId is the + * driver command span. OTLP file exports contain resourceSpans[].scopeSpans[].spans[] objects + * with hex-encoded traceId/spanId/parentSpanId fields; a simple containment check on the two + * hex ids in the same file line is sufficient and avoids a protobuf/JSON-schema dependency. + */ + private static boolean serverSpanLinked(final Path traceDir, final String traceId, final String parentSpanId) + throws IOException { + if (!Files.isDirectory(traceDir)) { + return false; + } + try (Stream files = Files.walk(traceDir)) { + List exportFiles = files.filter(Files::isRegularFile).collect(Collectors.toList()); + for (Path file : exportFiles) { + for (String line : Files.readAllLines(file, StandardCharsets.UTF_8)) { + if (line.contains(traceId) && line.contains("\"parentSpanId\":\"" + parentSpanId + "\"")) { + return true; + } + } + } + } + return false; + } +} From 321b7c39215307329d86d83e3ddcd8e36232b4f3 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 14 Jul 2026 15:03:05 +0100 Subject: [PATCH 12/48] DRIVERS-3454: mark POC design superseded --- .../specs/2026-06-01-otel-opmsg-propagation-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md b/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md index 415c31fbf73..537f4cb6fcc 100644 --- a/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md +++ b/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md @@ -1,7 +1,7 @@ # Design: OpenTelemetry Trace-Context Propagation over OP_MSG - **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) — Support trace context propagation to the server -- **Status:** Design (Investigating phase) +- **Status:** SUPERSEDED by `2026-07-13-otel-telemetry-section-reference-impl-design.md` (payload is now a BSON document; gating is by maxWireVersion >= 29, not a hello flag) - **Author:** Nabil Hachicha - **Date:** 2026-06-01 - **Related:** [DRIVERS-719](https://jira.mongodb.org/browse/DRIVERS-719) (client-side OTel tracing), [SERVER-107128](https://jira.mongodb.org/browse/SERVER-107128) (define trace context in OP_MSG), server POC [10gen/mongo#49930](https://github.com/10gen/mongo/pull/49930) From 48d76fcb87df9b08b4e4478a71373a16e2318598 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 14 Jul 2026 16:15:49 +0100 Subject: [PATCH 13/48] DRIVERS-3454: update e2e runbook for 9.0 server and OTLP file exporter Executed end-to-end against mongodb-mongo-master ee3a67f8 (9.0.0-alpha0): ServerSpanLinkageProseTest passes. Confirmed real server parameters (featureFlagOtelTraceSampling, openTelemetryTracingSampling BSON doc, opentelemetryTraceDirectory) and NDJSON export format. Fixed the prose test to assert operation-span parentage: the driver injects the traceparent from the OperationContext's operation span (per design 3.1), so the server span is a sibling of the client command span. --- docs/superpowers/runbooks/otel-opmsg-e2e.md | 254 +++++++++++++++--- ...7-13-otel-telemetry-section-prose-tests.md | 9 +- .../ServerSpanLinkageProseTest.java | 29 +- 3 files changed, 248 insertions(+), 44 deletions(-) diff --git a/docs/superpowers/runbooks/otel-opmsg-e2e.md b/docs/superpowers/runbooks/otel-opmsg-e2e.md index ac41c43ba79..da690fdabaa 100644 --- a/docs/superpowers/runbooks/otel-opmsg-e2e.md +++ b/docs/superpowers/runbooks/otel-opmsg-e2e.md @@ -1,37 +1,229 @@ # Runbook: OTel OP_MSG propagation end-to-end (manual) -Validates DRIVERS-3454 end to end: a sync-driver client span linked to a server child span. +Validates DRIVERS-3454 end to end against a real MongoDB 9.0/master server: a +sync-driver client trace linked to a server-side span created from the OP_MSG +telemetry section (section kind 3, `{otel: {traceparent: }}`), via the +server's OTLP file exporter. + +This supersedes the old POC-branch flow (10gen/mongo PR #49930). The server +feature landed as [10gen/mongo#56646](https://github.com/10gen/mongo/pull/56646) +(SERVER-129959, merged 2026-07-06) — no more `OtelTracePropagationTestToggle` / +`tracingSupport` hello-capability hacks are needed; the driver sends the +telemetry section whenever `maxWireVersion >= 29` (9.0+) and an active tracing +span yields a valid traceparent. + +Last executed successfully: 2026-07-14, against +`gitVersion ee3a67f8f8735b5e4aedd2b66ccb700e92d55205` +(`9.0.0-alpha0-ee3a67f8`, mongodb-mongo-master mainline of 2026-07-14 — +after the #56646 merge), linux arm64 dist-test binary run in Docker on a macOS +arm64 host. `ServerSpanLinkageProseTest` PASSED. ## Prerequisites -- A local build of the server POC branch (10gen/mongo PR #49930), which accepts OP_MSG - section kind 3 and starts a server span from it. NOTE: the POC does NOT yet advertise - `tracingSupport` in hello. Until SERVER-107128 adds it, temporarily force the driver - capability on for this manual run (see step 3). -- An OpenTelemetry collector / exporter the server POC is configured to export traces to - (e.g. Jaeger via OTLP), plus a Micrometer Tracing OTel bridge on the client. - -## Steps -1. Build & run the server POC `mongod` with tracing enabled and sampling at 100%. -2. Start a Jaeger all-in-one (or OTLP collector) and point both server and client exporters at it. -3. In a scratch sync-driver program, configure a `MongoClient` with an `ObservationRegistry` - that has an OTel-backed `DefaultTracingObservationHandler` (so `traceParent()` is non-null - and sampled). Run a `find`. - - Temporary capability override for the run (POC server lacks the hello flag): start an - OTel root span yourself, then run the command. Because the server POC accepts kind 3 - unconditionally, set the driver `tracingSupported` to true by connecting to a server - whose hello you patch to include `tracingSupport: true`, OR temporarily hardcode - `isTracingSupport()`/`tracingSupported` to `true` on the local branch for the manual run - only (revert before staging). -4. In Jaeger, confirm a single trace contains BOTH the client `find` span and a server span, - with the server span's parent = the client span id sent in the traceparent. - -## Pass criteria -- One trace, two+ spans, correct parent/child linkage across the client→server boundary. -- With the driver capability off (default), no server span is created (negative check). + +- A `mongod` built from `10gen/mongo` `master` at or after the 2026-07-06 merge + of #56646. Verify with `mongod --version` → `gitVersion` must be a master + commit at/after that date; pre-merge binaries silently lack the telemetry + section support. +- The automated test this runbook exercises: + `driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java` + (prose test 5 in + `docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md`). + It is gated by `@EnabledIfSystemProperty(named = "org.mongodb.test.otel.trace.dir", ...)` + and skipped unless that property is set. + +## Step 1: Get a server binary (what actually worked) + +Do NOT submit new Evergreen patch builds for this — a no-op +`enterprise-macos-arm64` patch (and the same task on mainline) failed in +`bazel compile`, i.e. macOS `archive_dist_test` was broken on master at the +time of writing, and patch builds burn CI resources. Use an already-built +mainline artifact instead: + +1. Authenticate REST calls with the CLI's OAuth token against the **corp** + host (the public host rejects both static API keys and this token): + + ```bash + TOKEN=$(evergreen client get-oauth-token) + API=https://evergreen.corp.mongodb.com/api/rest/v2 + ``` + + (If the CLI complains it is too old, `evergreen get-update` and use the + downloaded binary.) + +2. List recent mainline versions and pick a revision after 2026-07-06: + + ```bash + curl -s -H "Authorization: Bearer $TOKEN" \ + "$API/projects/mongodb-mongo-master/versions?limit=40" + ``` + +3. macOS arm64 variants are rarely activated on mainline; the reliably-green + arm64 compile variant is `amazon-linux2023-arm64-static-compile`. Fetch its + `archive_dist_test` task (task id pattern + `mongodb_mongo_master_amazon_linux2023_arm64_static_compile_archive_dist_test__`, + timestamp = the version's `create_time`) and take the presigned URL of the + "Binaries" artifact (`mongo-.tgz`, ~1 GB, unpacks to `dist-test/`): + + ```bash + curl -s -H "Authorization: Bearer $TOKEN" "$API/tasks/" # → artifacts[].url + curl -L -o mongodb-binaries.tgz "" + tar -xzf mongodb-binaries.tgz # → dist-test/bin/mongod (aarch64 linux ELF) + ``` + + The binary used for the recorded run is kept at + `~/MongoDB/otel-e2e-server/dist-test/bin/mongod` + (from `mongodb-binaries-ee3a67f8.tgz` in the same directory). + +4. Verify: + + ```bash + docker run --rm -v ~/MongoDB/otel-e2e-server/dist-test:/opt/dist-test \ + otel-poc-mongod:latest /opt/dist-test/bin/mongod --version + # db version v9.0.0-alpha0-ee3a67f8, gitVersion ee3a67f8... + ``` + + `otel-poc-mongod:latest` is a local `amazonlinux:2023` image with the + runtime shared libs the dynamically-linked dist-test mongod needs + (`openldap-compat` for `libldap_r`, cyrus-sasl, krb5, net-snmp, …); its + Dockerfile is at `~/MongoDB/otel-poc-server/Dockerfile`. + +## Step 2: Start mongod with tracing + the OTLP file exporter + +Parameter names confirmed both from source +(`src/mongo/otel/traces/tracing_feature_flags.idl`, `trace_settings.idl`, +`trace_sampling_parameters.idl`) and by a live run — the plan's original +guesses (`featureFlagOtelTracing`, `opentelemetrySamplingRates`) were wrong. +Span creation requires **two** feature flags +(`src/mongo/otel/traces/tracing_enablement.cpp`): `featureFlagTracing` +(default true) AND `featureFlagOtelTraceSampling` (default false — set it). + +Working invocation (Docker; note the host port remap — anything already +listening on host 27017, e.g. a local dev mongod, will otherwise shadow the +container because it binds 127.0.0.1 while Docker binds `*`): + +```bash +mkdir -p /tmp/otel-e2e/db /tmp/otel-e2e/traces + +docker run -d --name otel-e2e-mongod \ + -p 27227:27017 \ + -v ~/MongoDB/otel-e2e-server/dist-test:/opt/dist-test \ + -v /tmp/otel-e2e/db:/data/db \ + -v /tmp/otel-e2e/traces:/data/traces \ + otel-poc-mongod:latest \ + /opt/dist-test/bin/mongod --dbpath /data/db --port 27017 --bind_ip_all \ + --setParameter opentelemetryTraceDirectory=/data/traces \ + --setParameter featureFlagOtelTraceSampling=true \ + --setParameter 'openTelemetryTracingSampling={defaultSampling: {samplingFactor: 1.0}}' +``` + +(For a native binary, drop the Docker wrapper and point +`opentelemetryTraceDirectory` straight at `/tmp/otel-e2e/traces`.) + +Sanity checks: + +```bash +mongosh --quiet --port 27227 --eval ' + print(db.version()); // 9.0.0-alpha0-... + print(db.adminCommand({hello:1}).maxWireVersion); // 29 + printjson(db.adminCommand({getParameter:1, featureFlagOtelTraceSampling:1})); + // -> currentlyEnabled: true +' +``` + +Notes: +- `openTelemetryTracingSampling` takes an `OpenTelemetryTracingSamplingConfig` + BSON document, not a raw double; `samplingFactor: 1.0` samples everything + (the default is 0.000045). +- The client's inbound traceparent is treated as an *externally sampled* + trace, governed by `openTelemetryExternalTracing` (token bucket, defaults + refillRate 1/s, maxTokens 10 — ample for a test run). +- Do NOT delete files under the trace directory while mongod runs: the file + exporter keeps its `.jsonl` open and further exports go to the deleted + inode. Restart the container (or mongod) after cleaning the directory. + +## Step 3: Run the e2e test + +```bash +./gradlew :driver-sync:test --tests 'com.mongodb.client.observability.ServerSpanLinkageProseTest' \ + -Dorg.mongodb.test.uri="mongodb://localhost:27227" \ + -Dorg.mongodb.test.otel.trace.dir=/tmp/otel-e2e/traces +``` + +Expected: PASSED. The test runs an instrumented `find`, then polls the trace +directory for an exported server span whose `traceId` equals the client trace +id and whose `parentSpanId` equals the driver's **operation** span id. + +Important semantics (verified empirically on 2026-07-14): the driver injects +the traceparent from the `OperationContext`'s active tracing span — the +*operation* span, per the reference-impl design section 3.1 — because +`CommandMessage.encode()` runs before the command span is created in +`InternalStreamConnection.sendAndReceiveInternal()`. The server `find` span is +therefore a *sibling* of the client command span, both children of the +operation span. The first version of the prose test asserted parentage by the +command span and failed against the real server; it was fixed to assert the +operation-span linkage. + +Debugging order if it fails: +1. `maxWireVersion >= 29`? If lower, the driver never attaches the section + (`NINE_DOT_ZERO_WIRE_VERSION` gate in `CommandMessage`). Beware of another + local mongod shadowing the port (see Step 2). +2. `featureFlagOtelTraceSampling` currentlyEnabled? +3. Traceparent sampled flag (`-01` suffix) set? Unsampled parents are dropped. +4. mongod log: `BadValue` / `UnknownOpMsgSectionKind` on the telemetry section + would indicate a wire-format mismatch. +5. Inspect the export files directly (Step 4). + +## Step 4: OTLP file export format (verified) + +The exporter writes `mongod---trace.jsonl` in the trace +directory: NDJSON, one single-line `{"resourceSpans":[...]}` blob per batch +export. A span's `traceId`, `spanId`, `parentSpanId` (lowercase hex, no +dashes) all appear on the same line, so the test's line-containment matcher +(`serverSpanLinked`) is valid as written. If a future server switches to +pretty-printed JSON, the matcher must parse whole documents instead. + +Example exported span: + +```json +{"resourceSpans":[{"resource":{...service.name="mongod"...},"scopeSpans":[{"scope":{"name":"mongodb"}, + "spans":[{"name":"find","traceId":"c3778d7ccab7feb73a1e82af44140371", + "spanId":"100d03d5a6795500","parentSpanId":"461e3f3509da712b","kind":1,"flags":1,...}]}]}]} +``` + +## Step 5: Clean up + +```bash +docker rm -f otel-e2e-mongod +rm -rf /tmp/otel-e2e +``` + +Keep the downloaded binary (`~/MongoDB/otel-e2e-server/`) for future runs. + +--- + +## Appendix: optional visual verification via Jaeger + +For interactive confirmation, point the server at an OTLP HTTP collector +instead of the file exporter: + +```bash +docker run --rm -d --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:latest + + ... \ + --setParameter opentelemetryHttpEndpoint=http://:4318/v1/traces \ + --setParameter featureFlagOtelTraceSampling=true \ + --setParameter 'openTelemetryTracingSampling={defaultSampling: {samplingFactor: 1.0}}' +``` + +Run the same instrumented `find` and check http://localhost:16686 for a single +trace containing the client operation span, the client command span, and the +server span (the latter two as siblings under the operation span). ## Notes -- This proves the wire format end to end; the automated Phase 1 test - (`CommandMessageOtelTraceContextTest`) is the regression guard. In particular, - `getCommandDocumentIgnoresOtelSection` guards against the trailing kind-3 section - corrupting command-document reconstruction on the send path. -- Findings (any format mismatch, tracestate handling, flags) feed back into the spec. + +- `CommandMessageOtelTraceContextTest` is the regression guard for the + send-path BSON encoding; `ServerSpanLinkageProseTest` (this runbook) is the + regression guard for full linkage against a real server. +- Findings (parent-span semantics, tracestate handling, flags) feed back into + the spec — in particular the operation-vs-command parent-span semantics + above. diff --git a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md index 862074ffc2d..042bd0afc57 100644 --- a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md +++ b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md @@ -33,7 +33,10 @@ present (drivers MUST omit the section rather than send an invalid traceparent). ## 5. End-to-end server span linkage With tracing enabled, run a CRUD operation (e.g. `find`) against the configured 9.0+ -server. Capture the driver's finished command span (client side). Then read the server's +server. Capture the driver's finished spans (client side). Then read the server's exported OTLP JSON from the trace directory and assert a server span exists whose -`traceId` equals the client command span's trace-id and whose `parentSpanId` equals the -client command span's span-id. Allow for the server's batch export interval when polling. +`traceId` equals the client trace-id and whose `parentSpanId` equals the driver's +*operation* span-id (the traceparent is injected from the `OperationContext`'s active +tracing span per the reference-impl design section 3.1, so the server span is a sibling +of the client command span, both parented by the operation span). Allow for the server's +batch export interval when polling. diff --git a/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java b/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java index 6db26354ca7..6290eec10fe 100644 --- a/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java +++ b/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java @@ -76,7 +76,7 @@ void tearDown() { } @Test - @DisplayName("Prose test 5: server emits a child span of the driver command span") + @DisplayName("Prose test 5: server emits a span parented by the driver operation span") void testServerSpanLinkage() throws Exception { MongoClientSettings clientSettings = getMongoClientSettingsBuilder() .observabilitySettings(ObservabilitySettings.micrometerBuilder() @@ -92,26 +92,35 @@ void testServerSpanLinkage() throws Exception { List clientSpans = inMemoryOtel.getFinishedSpans(); assertTrue(clientSpans.size() >= 2, "expected operation + command client spans, got: " + clientSpans); - // command span is the innermost (first finished) span; see AbstractMicrometerProseTest ordering - FinishedSpan commandSpan = clientSpans.get(0); - String traceId = commandSpan.getTraceId(); - String commandSpanId = commandSpan.getSpanId(); + // Per the reference-impl design (section 3.1), the driver injects the traceparent from the + // OperationContext's active tracing span, i.e. the OPERATION span. The command span is a + // client-side child of the operation span, so the server span is a sibling of the command + // span, both parented by the operation span. Finished-span ordering is not guaranteed, so + // find the command span by name ("find") and take its parent as the operation span id. + List commandSpans = clientSpans.stream() + .filter(span -> "find".equals(span.getName())) + .collect(Collectors.toList()); + assertTrue(!commandSpans.isEmpty(), "no client command span named 'find' in: " + clientSpans); Path traceDir = Paths.get(System.getProperty("org.mongodb.test.otel.trace.dir")); long deadline = System.currentTimeMillis() + EXPORT_POLL_TIMEOUT_MS; while (System.currentTimeMillis() < deadline) { - if (serverSpanLinked(traceDir, traceId, commandSpanId)) { - return; + for (FinishedSpan commandSpan : commandSpans) { + String operationSpanId = commandSpan.getParentId(); + if (operationSpanId != null + && serverSpanLinked(traceDir, commandSpan.getTraceId(), operationSpanId)) { + return; + } } Thread.sleep(EXPORT_POLL_INTERVAL_MS); } - fail("no server span found in " + traceDir + " with traceId=" + traceId - + " and parentSpanId=" + commandSpanId); + fail("no server span found in " + traceDir + + " with traceId/parentSpanId matching the operation span of any of " + commandSpans); } /** * Scans OTLP JSON export files for a span with the given traceId whose parentSpanId is the - * driver command span. OTLP file exports contain resourceSpans[].scopeSpans[].spans[] objects + * given driver span. OTLP file exports contain resourceSpans[].scopeSpans[].spans[] objects * with hex-encoded traceId/spanId/parentSpanId fields; a simple containment check on the two * hex ids in the same file line is sufficient and avoids a protobuf/JSON-schema dependency. */ From c36118c1686c3884e07d435c98ea219e9f8203b7 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Wed, 15 Jul 2026 00:00:08 +0100 Subject: [PATCH 14/48] DRIVERS-3454: guard traceparent production against absent optional micrometer-tracing --- .../micrometer/MicrometerTracer.java | 66 +++++++++---- .../micrometer/MicrometerTraceParentTest.java | 97 +++++++++++++++++++ 2 files changed, 146 insertions(+), 17 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java index 78caf6ae6ef..179fd501324 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java @@ -24,7 +24,6 @@ import io.micrometer.observation.Observation; import io.micrometer.observation.ObservationConvention; import io.micrometer.observation.ObservationRegistry; -import io.micrometer.tracing.handler.TracingObservationHandler; import org.bson.BsonDocument; import org.bson.BsonReader; import org.bson.json.JsonMode; @@ -48,11 +47,28 @@ * @since 5.7 */ public class MicrometerTracer implements Tracer { + /** + * {@code micrometer-tracing} is an optional dependency: consumers may have only {@code micrometer-observation} + * on the classpath (e.g. metrics/logging-only observability setups). Guard any use of {@code io.micrometer.tracing.*} + * types with this flag so that such setups never trigger a {@link NoClassDefFoundError} or {@link LinkageError}. + */ + static final boolean MICROMETER_TRACING_ON_CLASSPATH = isMicrometerTracingOnClasspath(); + private final ObservationRegistry observationRegistry; private final boolean allowCommandPayload; private final int textMaxLength; private final ObservationConvention convention; + private static boolean isMicrometerTracingOnClasspath() { + try { + Class.forName("io.micrometer.tracing.handler.TracingObservationHandler", + false, MicrometerTracer.class.getClassLoader()); + return true; + } catch (ClassNotFoundException | LinkageError e) { + return false; + } + } + /** * Constructs a new {@link MicrometerTracer} instance. * @@ -124,30 +140,17 @@ public String traceParent() { if (observation == null) { return null; } - TracingObservationHandler.TracingContext tracingContext = - observation.getContextView().get(TracingObservationHandler.TracingContext.class); - if (tracingContext == null || tracingContext.getSpan() == null) { - return null; - } - // Fully qualified to avoid a name clash with this package's own TraceContext interface. - io.micrometer.tracing.TraceContext ctx = tracingContext.getSpan().context(); - if (ctx == null) { + if (!MICROMETER_TRACING_ON_CLASSPATH) { return null; } - String traceId = ctx.traceId(); - String spanId = ctx.spanId(); - if (!isValidNonZeroLowercaseHex(traceId, 32) || !isValidNonZeroLowercaseHex(spanId, 16)) { - return null; - } - Boolean sampled = ctx.sampled(); - return "00-" + traceId + "-" + spanId + (sampled != null && sampled ? "-01" : "-00"); + return MicrometerTracingSupport.traceParent(observation); } /** * The server ({@code validateW3CTraceparent}) rejects ids that are not exactly the expected * length of lowercase hex, or that are all zeroes. Never emit a traceparent it would reject. */ - private static boolean isValidNonZeroLowercaseHex(@Nullable final String value, final int expectedLength) { + static boolean isValidNonZeroLowercaseHex(@Nullable final String value, final int expectedLength) { if (value == null || value.length() != expectedLength) { return false; } @@ -165,6 +168,35 @@ private static boolean isValidNonZeroLowercaseHex(@Nullable final String value, } } + /** + * Isolates all hard references to {@code io.micrometer.tracing.*} types. This class must only be loaded/invoked + * after {@link #MICROMETER_TRACING_ON_CLASSPATH} has been confirmed {@code true}, so that JVMs which resolve + * classes referenced by bytecode eagerly (e.g. during verification) do not trigger a {@link NoClassDefFoundError} + * when {@code micrometer-tracing} is absent from the classpath. + */ + private static final class MicrometerTracingSupport { + @Nullable + static String traceParent(final Observation observation) { + io.micrometer.tracing.handler.TracingObservationHandler.TracingContext tracingContext = + observation.getContextView().get(io.micrometer.tracing.handler.TracingObservationHandler.TracingContext.class); + if (tracingContext == null || tracingContext.getSpan() == null) { + return null; + } + io.micrometer.tracing.TraceContext ctx = tracingContext.getSpan().context(); + if (ctx == null) { + return null; + } + String traceId = ctx.traceId(); + String spanId = ctx.spanId(); + if (!MicrometerTraceContext.isValidNonZeroLowercaseHex(traceId, 32) + || !MicrometerTraceContext.isValidNonZeroLowercaseHex(spanId, 16)) { + return null; + } + Boolean sampled = ctx.sampled(); + return "00-" + traceId + "-" + spanId + (sampled != null && sampled ? "-01" : "-00"); + } + } + /** * Represents a Micrometer-based span. */ diff --git a/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java index 22485a9f28d..7c6acb7de44 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java @@ -23,7 +23,17 @@ import com.mongodb.observability.micrometer.MongodbObservation; import org.junit.jupiter.api.Test; +import java.lang.reflect.Constructor; +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.List; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNull; class MicrometerTraceParentTest { @@ -60,6 +70,93 @@ void shouldReturnNullForInvalidIds() { assertNull(traceParentFor(VALID_TRACE_ID, null, true)); } + /** + * {@code micrometer-tracing} is an optional dependency. Consumers who only have {@code micrometer-observation} + * on the classpath (metrics/logging-only observability setups) must not see a {@link NoClassDefFoundError} or + * any other {@link LinkageError} the first time {@code traceParent()} is called. + */ + @Test + void shouldExposeMicrometerTracingClasspathGuard() { + // Sanity check on the current test classpath (micrometer-tracing IS present here). + assertEquals(true, MicrometerTracer.MICROMETER_TRACING_ON_CLASSPATH); + } + + /** + * Simulates a consumer that has only {@code micrometer-observation} on the classpath (no + * {@code micrometer-tracing}), by loading {@link MicrometerTracer} (and its dependency classes) through an + * isolated classloader that refuses to resolve {@code io.micrometer.tracing.*}. Asserts that {@code + * traceParent()} returns {@code null} rather than throwing {@link NoClassDefFoundError}. + */ + @Test + void shouldReturnNullWhenMicrometerTracingIsAbsentFromClasspath() throws Exception { + List urls = new ArrayList<>(); + String[] classpathEntries = System.getProperty("java.class.path").split(System.getProperty("path.separator")); + for (String entry : classpathEntries) { + if (entry.contains("micrometer-tracing")) { + continue; + } + urls.add(new java.io.File(entry).toURI().toURL()); + } + + try (URLClassLoader isolatedLoader = new URLClassLoader(urls.toArray(new URL[0]), null) { + @Override + protected Class loadClass(final String name, final boolean resolve) throws ClassNotFoundException { + if (name.startsWith("io.micrometer.tracing.")) { + throw new ClassNotFoundException(name); + } + if (name.startsWith("java.") || name.startsWith("javax.") || name.startsWith("jdk.")) { + return Class.forName(name, resolve, null); + } + return super.loadClass(name, resolve); + } + }) { + Class tracerClass = Class.forName( + "com.mongodb.internal.observability.micrometer.MicrometerTracer", true, isolatedLoader); + + Field guard = tracerClass.getDeclaredField("MICROMETER_TRACING_ON_CLASSPATH"); + guard.setAccessible(true); + assertFalse((Boolean) guard.get(null), + "guard should detect that io.micrometer.tracing is not resolvable in the isolated loader"); + + Class registryClass = Class.forName("io.micrometer.observation.ObservationRegistry", true, isolatedLoader); + Object registry = registryClass.getMethod("create").invoke(null); + + Constructor tracerCtor = tracerClass.getDeclaredConstructor( + registryClass, boolean.class, int.class, + Class.forName("io.micrometer.observation.ObservationConvention", true, isolatedLoader)); + Object micrometerTracer = tracerCtor.newInstance(registry, false, 1000, null); + + Class observationTypeClass = Class.forName( + "com.mongodb.observability.micrometer.MongodbObservation", true, isolatedLoader); + Object observationType = observationTypeClass.getField("MONGODB_COMMAND").get(null); + + Class tracerInterface = Class.forName( + "com.mongodb.internal.observability.micrometer.Tracer", true, isolatedLoader); + Method nextSpan = tracerInterface.getMethod("nextSpan", observationTypeClass, String.class, + Class.forName("com.mongodb.internal.observability.micrometer.TraceContext", true, isolatedLoader), + Class.forName("com.mongodb.MongoNamespace", true, isolatedLoader)); + Object span = nextSpan.invoke(micrometerTracer, observationType, "find", null, null); + + Class spanInterface = Class.forName( + "com.mongodb.internal.observability.micrometer.Span", true, isolatedLoader); + spanInterface.getMethod("openScope").invoke(span); + + Object traceContext = assertDoesNotThrow(() -> spanInterface.getMethod("context").invoke(span), + "obtaining the trace context must not throw when micrometer-tracing is absent"); + + Class traceContextInterface = Class.forName( + "com.mongodb.internal.observability.micrometer.TraceContext", true, isolatedLoader); + Object traceParent = assertDoesNotThrow( + () -> traceContextInterface.getMethod("traceParent").invoke(traceContext), + "traceParent() must not throw NoClassDefFoundError when micrometer-tracing is absent"); + + assertEquals(null, traceParent); + + spanInterface.getMethod("closeScope").invoke(span); + spanInterface.getMethod("end").invoke(span); + } + } + private static String traceParentFor(final String traceId, final String spanId, final Boolean sampled) { ObservationRegistry registry = ObservationRegistry.create(); SimpleTracer tracer = new SimpleTracer(); From 9bf631ae612a8d6fc8a5e7c776c3ae2ba2a9f31f Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Sat, 18 Jul 2026 01:38:58 +0100 Subject: [PATCH 15/48] DRIVERS-3454: drop dead null-check on Span.context() (non-null by @NullMarked contract) --- .../internal/observability/micrometer/MicrometerTracer.java | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java index 179fd501324..5ac287e777a 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java @@ -182,10 +182,9 @@ static String traceParent(final Observation observation) { if (tracingContext == null || tracingContext.getSpan() == null) { return null; } + // Span.context() is non-null by contract (the package is @NullMarked); a no-op span yields a + // NOOP context whose empty ids fail the hex validation below. io.micrometer.tracing.TraceContext ctx = tracingContext.getSpan().context(); - if (ctx == null) { - return null; - } String traceId = ctx.traceId(); String spanId = ctx.spanId(); if (!MicrometerTraceContext.isValidNonZeroLowercaseHex(traceId, 32) From f067ee12e573a353d1c515021bba7bd210a521ba Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Sat, 18 Jul 2026 16:17:26 +0100 Subject: [PATCH 16/48] DRIVERS-3454: IDE polish (suppress always-inverted hint, EMPTY as lambda) --- .../observability/micrometer/MicrometerTracer.java | 1 + .../internal/observability/micrometer/TraceContext.java | 7 +------ 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java index 5ac287e777a..c792e6771ed 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java @@ -150,6 +150,7 @@ public String traceParent() { * The server ({@code validateW3CTraceparent}) rejects ids that are not exactly the expected * length of lowercase hex, or that are all zeroes. Never emit a traceparent it would reject. */ + @SuppressWarnings("BooleanMethodIsAlwaysInverted") static boolean isValidNonZeroLowercaseHex(@Nullable final String value, final int expectedLength) { if (value == null || value.length() != expectedLength) { return false; diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java index 66199bf13ea..c76d86e4d2e 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java @@ -20,12 +20,7 @@ @SuppressWarnings("InterfaceIsType") public interface TraceContext { - TraceContext EMPTY = new TraceContext() { - @Override - public String traceParent() { - return null; - } - }; + TraceContext EMPTY = () -> null; /** * The 55-char W3C {@code traceparent} string for this context From 9d7cfd688c41b3274a496d2f7befe9e47214e5e8 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Sat, 18 Jul 2026 17:53:17 +0100 Subject: [PATCH 17/48] DRIVERS-3454: design for minimal spec change (OP_MSG + OTel specs) --- ...6-07-18-drivers-3454-spec-change-design.md | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md diff --git a/docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md b/docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md new file mode 100644 index 00000000000..35f435993fb --- /dev/null +++ b/docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md @@ -0,0 +1,78 @@ +# Design: DRIVERS-3454 Minimal Spec Change (mongodb/specifications) + +- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) +- **Status:** Approved design +- **Author:** Nabil Hachicha +- **Date:** 2026-07-18 +- **Based on:** the reference implementation on branch `nabil_otel_context` (see + `2026-07-13-otel-telemetry-section-reference-impl-design.md`) and the shipped server contract + ([10gen/mongo#56646](https://github.com/10gen/mongo/pull/56646)). + +## Goal + +Draft the minimal normative spec change reflecting DRIVERS-3454 (OTel trace-context propagation to the server), +split across the two specs that own each layer, in the `mongodb/specifications` repo (worked on locally in the +`testing/resources/specifications` submodule). + +## Where the work happens + +- Branch **`DRIVERS-3454`** in `testing/resources/specifications`, based on **`origin/master`** (the repo's default + branch; there is no `main`). +- Local only; nothing pushed. The parent repo's submodule pointer is NOT updated (the parent repo treats the + submodule as do-not-modify; this work is on an explicit user instruction and stays on a local branch). +- Conventions per the repo's `AGENTS.md`: GitHub Flavored Markdown, 120-character line width, MongoDB docs style, + dated changelog entries prepended inside each spec's `## Changelog`. + +## Change 1 — `source/message/OP_MSG.md` (wire layer) + +1. Extend the `Section` struct's union: + + ```c + document telemetry; // payloadType == 3 + ``` + +2. Add normative rules next to the existing payload-type rules: + - An `OP_MSG` MAY contain **at most one** section with Payload Type 3. + - Payload Type 3 is **request-only**: drivers MAY send it; it MUST NOT appear in server replies. + - Its payload is a single BSON document whose contents are defined by the + [OpenTelemetry specification](../open-telemetry/open-telemetry.md) (cross-link). + - Receivers that do not recognize a section kind fail the message (existing behavior), therefore senders MUST + gate emission on server support as defined by the OpenTelemetry specification. +3. Include a brief explanatory line on numbering: **Payload Type 2 is reserved for server-internal use** (a + security-token section populated by the server/infrastructure components, never by drivers), so the next + available payload type for driver-emitted content is 3. Kind 2 itself remains undocumented here (status quo, + out of scope). +4. Changelog entry dated 2026-07-18. + +## Change 2 — `source/open-telemetry/open-telemetry.md` (driver behavior) + +1. New subsection under Implementation Requirements: **"Propagating Trace Context to the Server"**: + - Payload schema: `{ otel: { traceparent: } }`, where `traceparent` is the W3C traceparent value of the + **operation span** (stated explicitly: drivers attach the context at message-encode time, before the command + span exists; server spans therefore join the trace as children of the operation span). + - Drivers attach the section **iff**: tracing is enabled, the connection's `maxWireVersion >= 29` (MongoDB 9.0), + and a valid traceparent exists. + - Validity mirrors server-side enforcement: exactly 55 characters, + `00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>`, non-zero trace-id and parent-id. Drivers MUST omit + the section rather than send an invalid value; MUST NOT append `tracestate`; unsampled contexts are propagated + with trace-flags `00`. + - At most one telemetry section per message. Connections without an active span (monitoring, authentication) + naturally send no section (informative note). +2. **Design Rationale** additions (brief): + - Wire-version gate instead of a `hello` capability flag: the wire protocol version acts as the schema contract + for the payload; future fields require a wire-version bump (decision of 2026-06-16). + - BSON document instead of a bare traceparent string: extensibility without redesigning the payload format. +3. Changelog entry dated 2026-07-18. +4. **No Future Work addition** (explicitly out of scope per review). + +## Out of scope + +- Test-plan / prose-test additions (`tests/README.md`) — normative text only; tests follow in a later PR. +- Unified test format schema changes; YAML/JSON test files (so no `make -C source` regeneration needed). +- Documenting Payload Type 2 beyond the one-line reservation note. +- Pushing the branch or updating the parent repo's submodule pointer. + +## Validation + +- Run available repo checks locally (`pre-commit run --files ` and/or `mkdocs build --strict`) if the + tooling is installed; otherwise verify 120-char width and markdown lint manually and note it. From 4c3715f935b9c3be52f8f33de7a6eb4203fbab26 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Sat, 18 Jul 2026 18:01:22 +0100 Subject: [PATCH 18/48] DRIVERS-3454: plan for minimal spec change --- .../2026-07-18-drivers-3454-spec-change.md | 235 ++++++++++++++++++ 1 file changed, 235 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-18-drivers-3454-spec-change.md diff --git a/docs/superpowers/plans/2026-07-18-drivers-3454-spec-change.md b/docs/superpowers/plans/2026-07-18-drivers-3454-spec-change.md new file mode 100644 index 00000000000..8228a6855d6 --- /dev/null +++ b/docs/superpowers/plans/2026-07-18-drivers-3454-spec-change.md @@ -0,0 +1,235 @@ +# DRIVERS-3454 Minimal Spec Change Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Draft the DRIVERS-3454 normative spec change (OP_MSG Payload Type 3 + OTel trace-context propagation) on a local branch of the `mongodb/specifications` submodule. + +**Architecture:** Two markdown edits in the spec submodule at `testing/resources/specifications`: the wire-format layer lands in `source/message/OP_MSG.md`, the driver-behavior layer in `source/open-telemetry/open-telemetry.md`. No test files, no tooling regeneration. + +**Tech Stack:** GitHub Flavored Markdown (120-char line width), git. + +**Spec:** `docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md` (in the parent repo) — read it first. + +## Global Constraints + +- **LOCAL ONLY: never `git push`**, never update the parent repo's submodule pointer. +- All work happens inside `/Users/nabil.hachicha/MongoDB/mongo-java-driver/nabil_otel_context/testing/resources/specifications` on branch `DRIVERS-3454` based on `origin/master` (repo default; there is no `main`). +- Markdown at **120-character line width**; match each file's existing formatting exactly (note: `OP_MSG.md` uses `### Changelog`, `open-telemetry.md` uses `## Changelog`). +- Changelog entries dated **2026-07-18**, prepended as the first bullet. +- Normative text only: no changes under `tests/`, no Future Work section additions. +- Normative content must match the shipped server contract: payload `{ otel: { traceparent: } }`; gate `maxWireVersion >= 29` (MongoDB 9.0); traceparent exactly 55 chars `00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>`, non-zero ids; omit-on-invalid; no tracestate; unsampled propagated with flags `00`; at most one section per message; request-only; **operation span** is what's propagated. + +--- + +### Task 1: Branch setup + +**Files:** none (git only), all commands run from `testing/resources/specifications`. + +**Interfaces:** +- Produces: local branch `DRIVERS-3454` at `origin/master`, on which Tasks 2–3 commit. + +- [ ] **Step 1: Fetch and branch** + +```bash +cd /Users/nabil.hachicha/MongoDB/mongo-java-driver/nabil_otel_context/testing/resources/specifications +git fetch origin master +git checkout -b DRIVERS-3454 origin/master +git log --oneline -1 # note the base commit in your report +``` + +Expected: new branch `DRIVERS-3454` checked out, clean status. (The submodule was previously in detached-HEAD state; that is normal. Do NOT touch the parent repo.) + +--- + +### Task 2: `source/message/OP_MSG.md` — Payload Type 3 + +**Files:** +- Modify: `testing/resources/specifications/source/message/OP_MSG.md` (Section struct ~line 46-55; payload-type rules ~line 74-75; payload-type detail list ~line 168-186; `### Changelog` ~line 406) + +**Interfaces:** +- Produces: the OP_MSG-side definition that Task 3's cross-link relies on (anchor: the Payload Type 3 paragraphs added here). + +- [ ] **Step 1: Extend the `Section` struct** + +In the `struct Section` code block, add one union member after the `sequence` struct: + +```c + document telemetry; // payloadType == 3 +``` + +- [ ] **Step 2: Add the payload-type rules** + +Immediately after the paragraph ending "…MUST do so when the batch contains more than one entry." (~line 75), insert: + +```markdown + +Each `OP_MSG` request MAY additionally contain **at most one** section with `Payload Type 3`, carrying telemetry +context. `Payload Type 3` is request-only: drivers MAY send it, and it MUST NOT appear in server replies. Its payload +is a single BSON document whose contents are defined by the +[OpenTelemetry specification](../open-telemetry/open-telemetry.md). Servers and intermediaries that do not recognize a +section kind fail the message (see below), so senders MUST gate emission of this section on server support as defined +by the OpenTelemetry specification. + +> [!NOTE] +> `Payload Type 2` is reserved for server-internal use (a security-token section populated by the server and +> infrastructure components, never by drivers), so the next payload type available for driver-emitted content is 3. +``` + +(If the file does not use `> [!NOTE]` callouts elsewhere, render the note as a plain paragraph starting with +"Note:" instead — check with `grep -n '\[!NOTE\]' source/message/OP_MSG.md` and match house style.) + +- [ ] **Step 3: Add the payload detail entry** + +After the "When the Payload Type is 1, the content of the payload is:" block (~line 174-180) and before the "Any +unknown Payload Types MUST result in an error…" paragraph (~line 182), insert: + +```markdown +When the Payload Type is 3, the content of the payload is: + +- document — a single BSON document containing telemetry context, as defined by the + [OpenTelemetry specification](../open-telemetry/open-telemetry.md). +``` + +Also update the sentence at ~line 185 "A fully constructed `OP_MSG` MUST contain exactly one `Payload Type 0`, and +optionally any number of `Payload Type 1`" so it reads: + +```markdown +A fully constructed `OP_MSG` MUST contain exactly one `Payload Type 0`, optionally any number of `Payload Type 1`, +and optionally at most one `Payload Type 3` +``` + +(keeping the remainder of that sentence/paragraph intact). + +- [ ] **Step 4: Changelog entry** + +Prepend under `### Changelog` (~line 406): + +```markdown +- 2026-07-18: Add `Payload Type 3` (telemetry context BSON document; DRIVERS-3454). +``` + +- [ ] **Step 5: Verify formatting and commit** + +```bash +awk 'length > 120 {print FILENAME": "NR" ("length")"}' source/message/OP_MSG.md # expect no output +git add source/message/OP_MSG.md +git commit -m "DRIVERS-3454: define OP_MSG Payload Type 3 (telemetry context)" +``` + +--- + +### Task 3: `source/open-telemetry/open-telemetry.md` — propagation behavior + +**Files:** +- Modify: `testing/resources/specifications/source/open-telemetry/open-telemetry.md` (new `####` subsection at the end of Implementation Requirements, i.e. after the `##### Exceptions` block that precedes `## Motivation for Change` ~line 325; Design Rationale ~line 415; `## Changelog` ~line 426) + +**Interfaces:** +- Consumes: Task 2's OP_MSG Payload Type 3 definition (cross-link `../message/OP_MSG.md`). + +- [ ] **Step 1: Add the propagation subsection** + +Insert immediately before `## Motivation for Change` (so it is the last `####` inside Implementation Requirements): + +```markdown +#### Propagating Trace Context to the Server + +Drivers MUST propagate the active trace context to servers that support it, so that server-generated spans join the +same distributed trace as the driver's spans. + +The trace context is carried in an `OP_MSG` section with +[`Payload Type 3`](../message/OP_MSG.md), whose payload is a single BSON document with the following schema: + +```json +{ "otel": { "traceparent": "" } } +``` + +`traceparent` is the [W3C traceparent](https://www.w3.org/TR/trace-context/#traceparent-header) value of the +**operation span**. Drivers attach the trace context when encoding the wire message, before the command span is +created; server spans therefore join the trace as children of the operation span. + +Drivers MUST attach the section to a command if and only if all of the following hold: + +1. Tracing is enabled on the `MongoClient`. +2. The connection's `maxWireVersion` is greater than or equal to 29 (MongoDB 9.0). +3. A valid `traceparent` value is available from the active operation span. + +A `traceparent` value is valid if and only if it is exactly 55 characters of the form +`00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>` and neither the trace-id nor the parent-id is all zeroes. +This mirrors the server-side validation. If no valid value is available, drivers MUST omit the section entirely +rather than send an invalid or truncated value. Drivers MUST NOT append a `tracestate` suffix. Unsampled trace +contexts are propagated with trace-flags `00`. + +A message MUST NOT contain more than one telemetry section. Connections that carry no operation span (for example +server monitoring and authentication) naturally send no section, since condition 3 cannot hold. + +No tracing data is returned in server responses as part of this feature. +``` + +Note on the nested code fence: the JSON block inside this section uses a standard triple-backtick fence; when +inserting, make sure the surrounding document structure stays valid (the snippet above shows the intended rendered +content — reproduce it with correct fencing in the file). + +- [ ] **Step 2: Add Design Rationale entries** + +After the `### No URI options` block (before `## Changelog`), insert: + +```markdown +### Wire protocol version gate instead of a hello capability + +An alternative was for servers to advertise support via a `hello` response field. Gating on `maxWireVersion` was +chosen instead: the wire protocol version acts as the schema contract for the telemetry payload, so drivers know +exactly which fields a server accepts, and any future payload additions require a wire version bump rather than a +second, parallel negotiation mechanism. + +### BSON document instead of a bare traceparent string + +Carrying the traceparent inside a BSON document allows future propagation fields to be added to the same section +without redesigning the payload format. +``` + +- [ ] **Step 3: Changelog entry** + +Prepend under `## Changelog`: + +```markdown +- 2026-07-18: Add trace context propagation to the server via the `OP_MSG` telemetry section (DRIVERS-3454). +``` + +- [ ] **Step 4: Verify formatting and commit** + +```bash +awk 'length > 120 {print FILENAME": "NR" ("length")"}' source/open-telemetry/open-telemetry.md # expect no output +git add source/open-telemetry/open-telemetry.md +git commit -m "DRIVERS-3454: specify trace context propagation to the server" +``` + +--- + +### Task 4: Repo checks + +**Files:** none. + +- [ ] **Step 1: Run available repo tooling** + +```bash +cd /Users/nabil.hachicha/MongoDB/mongo-java-driver/nabil_otel_context/testing/resources/specifications +command -v pre-commit && pre-commit run --files source/message/OP_MSG.md source/open-telemetry/open-telemetry.md || echo "pre-commit unavailable" +command -v mkdocs && mkdocs build --strict 2>&1 | tail -5 || echo "mkdocs unavailable" +``` + +If a hook rewrites formatting, review the diff, re-verify content is intact, and amend the relevant commit. If +tooling is unavailable, note that in the report; the `awk` width checks from Tasks 2-3 plus a markdown render +sanity-check (e.g. `grep -c '^#### Propagating'`) suffice. + +- [ ] **Step 2: Final state check** + +```bash +git log --oneline origin/master..HEAD # expect exactly the 2 commits from Tasks 2-3 +git status --porcelain # expect clean +cd /Users/nabil.hachicha/MongoDB/mongo-java-driver/nabil_otel_context && git status --porcelain -- testing/resources/specifications +``` + +The parent repo will show the submodule as modified (new HEAD); do NOT commit that pointer change — leave it, and +say so in the report. + +**Do not push anything. Everything stays local.** From fe04efdce2e385747569a7b19d386c612482d44a Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Mon, 20 Jul 2026 18:52:42 +0100 Subject: [PATCH 19/48] DRIVERS-3454: design for command-span trace propagation (Option A) --- ...6-07-18-command-span-propagation-design.md | 85 +++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-18-command-span-propagation-design.md diff --git a/docs/superpowers/specs/2026-07-18-command-span-propagation-design.md b/docs/superpowers/specs/2026-07-18-command-span-propagation-design.md new file mode 100644 index 00000000000..d2349260535 --- /dev/null +++ b/docs/superpowers/specs/2026-07-18-command-span-propagation-design.md @@ -0,0 +1,85 @@ +# Design: Propagate the Command Span's Trace Context (Option A) + +- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) +- **Status:** Approved design (chosen in discussion 2026-07-18; "split creation from decoration" variant) +- **Author:** Nabil Hachicha +- **Date:** 2026-07-18 +- **Amends:** `2026-07-13-otel-telemetry-section-reference-impl-design.md` (which recorded operation-span + propagation as a limitation) and the spec branch `DRIVERS-3454` in `testing/resources/specifications`. + +## Problem + +The telemetry section currently propagates the **operation** span's traceparent because +`InternalStreamConnection.sendAndReceiveInternal` encodes the message (`:445`) before creating the command span +(`:448`), and `TracingManager.createTracingSpan` eagerly re-parses the **encoded** bytes +(`commandDocumentSupplier.get()` → `message.getCommandDocument(bsonOutput)`) just to obtain the command name, +sensitive-command verdict, and `getMore` cursor id. Server spans therefore parent to the operation span — siblings +of the command span instead of children — losing per-command / per-retry attribution. + +## Key insight + +All *eager* inputs of `createTracingSpan` are available from the **raw, un-encoded** command document that +`CommandMessage` already stores (`CommandMessage.java:96`): first key = command name; sensitive check needs only +the name; `getMore` cursor id is a field of the raw document. The only encode-dependent input, the folded command +document for `db.query.text`, is already consumed lazily *after* span creation (`setQueryText`). So span creation +can move before `encode()` with no loss. + +In the Micrometer Observation model, span-id allocation and observation start are the same event, so +"pre-allocating ids" and "starting the span earlier" are the same change; decoration (attributes, scope, query +text) stays where it is today. + +## Changes (driver-core, internal only) + +1. **`TracingManager.createTracingSpan`** — replace the `Supplier commandDocumentSupplier` + parameter with the raw command document (or have the method read it from `message`). Body otherwise unchanged + (name, sensitive check, namespace resolution, observation-context population, `getMore` cursor id, session + fields). +2. **`CommandMessage`** — expose the raw command document (package-private accessor) for (1); add an `encode` + overload `encode(ByteBufferBsonOutput, OperationContext, @Nullable Span commandSpan)`. The base + `RequestMessage.encode(bsonOutput, operationContext)` signature is untouched (`CompressedMessage` and other + message types keep it). +3. **`writeTelemetryContextSection`** — read the traceparent from the passed `commandSpan` instead of + `operationContext.getTracingSpan()`. Null command span (tracing disabled, sensitive command, monitoring/auth) + ⇒ section omitted. The operation span is still used for *parenting* the command span in `TracingManager`. +4. **`InternalStreamConnection`** — in both the sync (`:445`) and async (`:620`) paths: create the command span + *before* `encode()`, pass it to the new overload, and wrap `encode` so that an encode failure ends the span + with error status (today encode failures happen span-less). The one-way `send` path (`:508`) has no command + span today; it passes `null` (no section — matches current behavior since w:0 fire-and-forget commands carry + no command span). + +## Behavior changes (all intentional) + +- Server spans become children of the **command** span. Each retry attempt / each `getMore` gets its own command + span, so server spans attribute to the exact wire command that caused them. +- The command span's duration now includes BSON serialization (unavoidable: ids ⇒ started; and defensible — + serialization is work done for that command). Microseconds against a network round trip. +- Security-sensitive commands can never carry a telemetry section (null command span), even where an operation + span exists — strictly safer than today. + +## Spec/doc updates (same effort) + +- **Spec branch `DRIVERS-3454`** (`testing/resources/specifications`, `source/open-telemetry/open-telemetry.md`): + flip the normative text from operation span to **command span** ("the propagated context MUST be that of the + command span; server spans join the trace as children of the command span"), and drop the encode-ordering + parenthetical. Changelog wording stays under the same 2026-07-18 entry (branch not yet merged). +- **Prose-test definitions** (`docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md`): test 1 + and test 5 assert the *command* span's ids. +- **E2E test** (`ServerSpanLinkageProseTest`): assert `parentSpanId == command span id` + (`getFinishedSpans().get(0)`, per `AbstractMicrometerProseTest` ordering). +- **Unit tests**: `CommandMessageOtelTraceContextTest` passes the span via the new `encode` overload; add a case + that a null command span omits the section even when an operation span exists. + +## Testing + +- Rework unit tests as above; run `:driver-core:test` affected suites. +- Micrometer prose tests (`AbstractMicrometerProseTest`) must stay green (span counts/names unchanged; only start + timing moves). +- Re-run the e2e server-span linkage test against the local 9.0 Docker server (runbook) and confirm the server + span's `parentSpanId` equals the **command** span id. + +## Out of scope + +- `Tracer.nextSpan()`-based true id pre-allocation (requires abandoning the Observation pipeline / new public + API). +- Option B (append-section-after-encode with header backpatch). +- Any public API change. From 346a959a46a28ab97b7187185687c298dfa299f1 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Mon, 20 Jul 2026 18:55:18 +0100 Subject: [PATCH 20/48] DRIVERS-3454: plan for command-span trace propagation --- .../2026-07-18-command-span-propagation.md | 353 ++++++++++++++++++ 1 file changed, 353 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-18-command-span-propagation.md diff --git a/docs/superpowers/plans/2026-07-18-command-span-propagation.md b/docs/superpowers/plans/2026-07-18-command-span-propagation.md new file mode 100644 index 00000000000..942f3e0189b --- /dev/null +++ b/docs/superpowers/plans/2026-07-18-command-span-propagation.md @@ -0,0 +1,353 @@ +# Command-Span Trace Propagation Implementation Plan (Option A) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Propagate the traceparent of the **command** span (not the operation span) in the OP_MSG telemetry section, by creating the command span before message encoding. + +**Architecture:** `TracingManager.createTracingSpan` stops depending on encoded bytes (it reads the raw command document that `CommandMessage` already holds); `InternalStreamConnection` hoists span creation above `encode()` in both sync and async paths; `CommandMessage` gains an `encode` overload carrying the command span, consumed by `writeTelemetryContextSection`. Spec branch, prose docs, and e2e assertions flip from operation-span to command-span parentage. + +**Tech Stack:** Java 8 (driver-core), JUnit 5, GFM specs. + +**Spec:** `docs/superpowers/specs/2026-07-18-command-span-propagation-design.md` — read it first. + +## Global Constraints + +- **LOCAL ONLY: never `git push`** (parent repo branch `nabil_otel_context`; spec submodule branch `DRIVERS-3454`). +- Java 8 source; no public API changes; everything under `com.mongodb.internal.*`. +- `docs/superpowers/` is gitignored in the parent repo — use `git add -f` for docs. +- Do not change `RequestMessage.encode(ByteBufferBsonOutput, OperationContext)`'s signature (other message types use it). +- The operation span REMAINS the command span's parent (`TracingManager.createTracingSpan` line ~191-192) — only the *propagated* context changes. +- Behavior contract after this change: telemetry section carries the command span's traceparent; null command span (tracing disabled / sensitive command / no-span paths) ⇒ **no section**, even if an operation span exists. + +--- + +### Task 1: `TracingManager.createTracingSpan` reads the raw command document + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/TracingManager.java:174-186` +- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` (add package-private raw-command accessor) +- Modify: `driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java:448-455, 624-631` (call sites — argument change only in this task; reorder happens in Task 3) + +**Interfaces:** +- Produces: `public Span createTracingSpan(CommandMessage message, OperationContext operationContext, Predicate isSensitiveCommand, Supplier serverAddressSupplier, Supplier connectionIdSupplier)` — the `Supplier commandDocumentSupplier` parameter is REMOVED; the method reads `message.getRawCommandDocument()`. +- Produces: `CommandMessage`: `BsonDocument getRawCommandDocument()` (package-private is impossible across packages — TracingManager is in a different package, so make it `public` on the internal class, javadoc'd as internal) returning the `command` field (`CommandMessage.java:96`). + +- [ ] **Step 1: Add the accessor to `CommandMessage`** + +Next to the other getters: + +```java + /** + * The raw (pre-encoding) command document this message was constructed with. Unlike + * {@link #getCommandDocument(ByteBufferBsonOutput)}, this does not include fields added during encoding + * ({@code $db}, {@code $readPreference}) nor document-sequence fields, but its first key is always the + * command name, and it is available before {@code encode()} runs. + */ + public BsonDocument getRawCommandDocument() { + return command; + } +``` + +- [ ] **Step 2: Change `createTracingSpan`** + +In `TracingManager.java`, remove the `commandDocumentSupplier` parameter and replace its use: + +```java + public Span createTracingSpan(final CommandMessage message, + final OperationContext operationContext, + final Predicate isSensitiveCommand, + final Supplier serverAddressSupplier, + final Supplier connectionIdSupplier + ) { + + if (!isEnabled()) { + return null; + } + BsonDocument command = message.getRawCommandDocument(); + String commandName = command.getFirstKey(); + ... +``` + +The rest of the body is unchanged — `command.containsKey("getMore")` / `command.getInt64("getMore")` now read the raw document, where the `getMore` field is identically present. + +- [ ] **Step 3: Update the two call sites in `InternalStreamConnection` (drop the supplier argument only)** + +At `:448` and `:624`, delete the `() -> message.getCommandDocument(bsonOutput),` argument line. Do NOT move the calls yet (Task 3). + +- [ ] **Step 4: Compile and run tracing tests** + +Run: `./gradlew :driver-core:compileJava :driver-core:test --tests 'com.mongodb.internal.observability.micrometer.*' -q` +Expected: BUILD SUCCESSFUL, tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add driver-core/src/main +git commit -m "DRIVERS-3454: createTracingSpan reads the raw command document (no encoded-bytes dependency)" +``` + +--- + +### Task 2: `CommandMessage.encode` overload carrying the command span + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java:250, 295, 302-315` +- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` + +**Interfaces:** +- Consumes: existing `RequestMessage.encode(ByteBufferBsonOutput, OperationContext)` (untouched). +- Produces: `public void encode(ByteBufferBsonOutput bsonOutput, OperationContext operationContext, @Nullable Span commandSpan)` on `CommandMessage`. The 2-arg `encode` inherited from `RequestMessage` now results in NO telemetry section (span defaults to null). + +- [ ] **Step 1: Rework the unit test to the new contract (TDD)** + +In `CommandMessageOtelTraceContextTest`, change every `message.encode(output, operationContext)` call in the positive-path tests to `message.encode(output, operationContext, span)`, and repurpose `buildOperationContext(span)` → `buildOperationContext()` (the operation context no longer needs to carry the span for these tests; keep the method if other setup depends on it, passing null). Update the test matrix: + +```java + // Prose test 1 — section present: encode(output, operationContext, span) with wire version 29 => kind-3 section + // whose traceparent matches the span passed to encode (the COMMAND span). + // Prose test 2 — wire version 25 + span passed => no section. + // Prose test 3 — encode(output, operationContext, null) at wire version 29 => no section + // (rename shouldNotWriteTelemetrySectionWhenNoSpan; note: this now also covers "operation span exists but + // command span is null", since the operation context's span is irrelevant to the section). + // Prose test 4 — span whose context().traceParent() returns null => no section. + // getCommandDocumentIgnoresTelemetrySection — unchanged semantics, 3-arg encode with span. + // shouldWriteTelemetrySectionAfterDocumentSequences — 3-arg encode with span. + // NEW: shouldNotWriteTelemetrySectionViaTwoArgEncode — message.encode(output, operationContext) (2-arg) + // never writes a section even at wire version 29 (guards monitoring/compression/other callers). +``` + +- [ ] **Step 2: Run to verify failures** + +Run: `./gradlew :driver-core:test --tests 'com.mongodb.internal.connection.CommandMessageOtelTraceContextTest' -q` +Expected: FAIL (no 3-arg encode exists). + +- [ ] **Step 3: Implement in `CommandMessage`** + +Add a transient field + overload (a `CommandMessage` is encoded at most once per send attempt, single-threaded, so a field set for the duration of encode is safe; the overload documents that): + +```java + @Nullable + private Span commandSpanForEncoding; + + /** + * Encodes the message, attaching the OP_MSG telemetry section with {@code commandSpan}'s trace context when + * the gating conditions hold (see writeTelemetryContextSection). The 2-arg {@code encode} never attaches the + * section. Not thread-safe: a message must be encoded by one thread at a time (already the case — a + * CommandMessage is built and encoded per send attempt). + */ + public void encode(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext, + @Nullable final Span commandSpan) { + this.commandSpanForEncoding = commandSpan; + try { + encode(bsonOutput, operationContext); + } finally { + this.commandSpanForEncoding = null; + } + } +``` + +Change `writeTelemetryContextSection` (drop the `operationContext` parameter — no longer used; update its call at `:295`): + +```java + private void writeTelemetryContextSection(final ByteBufferBsonOutput bsonOutput) { + if (getSettings().getMaxWireVersion() < NINE_DOT_ZERO_WIRE_VERSION) { + return; + } + Span commandSpan = this.commandSpanForEncoding; + if (commandSpan == null) { + return; + } + String traceParent = commandSpan.context().traceParent(); + if (traceParent == null) { + return; + } + bsonOutput.writeByte(PAYLOAD_TYPE_3_TELEMETRY); + BsonDocument telemetry = new BsonDocument("otel", + new BsonDocument("traceparent", new BsonString(traceParent))); + new BsonDocumentCodec().encode(new BsonBinaryWriter(bsonOutput), telemetry, + EncoderContext.builder().build()); + } +``` + +Remove the now-unused `operationContext.getTracingSpan()` read here (do NOT remove `OperationContext.getTracingSpan()` itself — `TracingManager` uses it for parenting). + +- [ ] **Step 4: Run tests, verify pass** + +Run: `./gradlew :driver-core:test --tests 'com.mongodb.internal.connection.CommandMessageOtelTraceContextTest' -q` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +git commit -m "DRIVERS-3454: encode overload carries the command span for the telemetry section" +``` + +--- + +### Task 3: Hoist span creation above encode in `InternalStreamConnection` + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java` sync `sendAndReceiveInternal` (~`:441-480`) and async equivalent (~`:615-650`). The one-way `send` path (`:508`) and compression paths (`:540`, `:681`) keep the 2-arg encode (no command span there today). + +**Interfaces:** +- Consumes: Task 1's `createTracingSpan` (no supplier param); Task 2's `encode(bsonOutput, operationContext, commandSpan)`. + +- [ ] **Step 1: Reorder the sync path** + +Current shape (`:443-455`): `encode(...)` then `tracingSpan = createTracingSpan(...)`. New shape: + +```java + CommandEventSender commandEventSender; + Span tracingSpan; + try (ByteBufferBsonOutput bsonOutput = new ByteBufferBsonOutput(this)) { + tracingSpan = operationContext + .getTracingManager() + .createTracingSpan(message, + operationContext, + cmdName -> SECURITY_SENSITIVE_COMMANDS.contains(cmdName) + || SECURITY_SENSITIVE_HELLO_COMMANDS.contains(cmdName), + () -> getDescription().getServerAddress(), + () -> getDescription().getConnectionId() + ); + try { + message.encode(bsonOutput, operationContext, tracingSpan); + } catch (RuntimeException | Error e) { + if (tracingSpan != null) { + tracingSpan.error(e); + tracingSpan.end(); + } + throw e; + } + ...rest unchanged (isLoggingCommandNeeded, getCommandDocument hydration, setQueryText, openScope)... +``` + +The existing failure handling after this point (`:484-486` `error/closeScope/end`) is unchanged — the new catch only covers the window where the span exists but the established handlers don't yet. + +- [ ] **Step 2: Reorder the async path identically** + +Same transformation at ~`:618-631` — `tracingSpan = createTracingSpan(...)` first, then `message.encode(bsonOutput, operationContext, tracingSpan)` inside a try/catch that calls `tracingSpan.error(e); tracingSpan.end();` before rethrowing/propagating through the existing async error handling (match how that block reports errors — it already has a `try` whose catch handles cleanup; ensure the span ends exactly once: if the surrounding catch already ends the span when non-null, rely on it instead of a nested catch — inspect and pick the single-end structure that fits the existing block). + +- [ ] **Step 3: Run driver-core connection + tracing tests** + +Run: `./gradlew :driver-core:test --tests 'com.mongodb.internal.connection.*' --tests 'com.mongodb.internal.observability.micrometer.*' -q` +Expected: PASS. + +- [ ] **Step 4: Prose tests against local server (functional)** + +Requires a running MongoDB at localhost:27017. Run: +`./gradlew :driver-sync:test --tests 'com.mongodb.client.observability.MicrometerProseTest' -Dorg.mongodb.test.uri="mongodb://localhost:27017" -q` +Expected: PASS (span counts/names unchanged — only creation timing moved). + +- [ ] **Step 5: Commit** + +```bash +git add driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java +git commit -m "DRIVERS-3454: create command span before encode; propagate its context" +``` + +--- + +### Task 4: Flip e2e + prose docs to command-span parentage + +**Files:** +- Modify: `driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java` (assertion comments/javadoc — the span it selects, `getFinishedSpans().get(0)` named e.g. "find", IS the command span per `AbstractMicrometerProseTest` ordering; verify and update the javadoc/comments that currently say "operation span", and any variable names like `operationSpanId`) +- Modify: `docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md` (tests 1 and 5: trace-id/span-id of the **command span**) +- Modify: `docs/superpowers/runbooks/otel-opmsg-e2e.md` (any operation-span wording) + +**Interfaces:** +- Consumes: Tasks 1-3 (behavior now propagates the command span). + +- [ ] **Step 1: Update the e2e test's span selection and wording** + +Read the test first. If it currently picks the span by name "find" via `get(0)` — that IS the command span (`AbstractMicrometerProseTest` asserts `get(0).getName() == "find"` = command span, `get(1)` = operation span "find db.test"); then the previous operation-span behavior was asserted via `getParentId()` or similar adaptation made in the earlier fix — align the selection so the asserted parent is the command span's own span-id, and fix all comments/javadoc that explain operation-span semantics. + +- [ ] **Step 2: Update prose-test definitions and runbook wording** + +In `2026-07-13-otel-telemetry-section-prose-tests.md`: test 1's "trace-id/span-id match the active span's context" → "match the **command span**'s context"; test 5's "parentSpanId equals the client command span's span-id" (already command-oriented from the original draft — make sure any 2026-07-14-era rewording to "operation span" is reverted to command span). Same sweep in the runbook. + +- [ ] **Step 3: Run the e2e test against the local 9.0 server (the real proof)** + +Start the wire-29 server (from the runbook; binary at `~/MongoDB/otel-e2e-server/dist-test`, Docker image `otel-poc-mongod:latest`) with the OTLP file exporter + `featureFlagOtelTraceSampling=true` + `openTelemetryTracingSampling={defaultSampling: {samplingFactor: 1.0}}` and a trace directory volume-mounted to the host. Then: + +```bash +./gradlew :driver-sync:test --tests 'com.mongodb.client.observability.ServerSpanLinkageProseTest' \ + -Dorg.mongodb.test.uri="mongodb://localhost:27017" \ + -Dorg.mongodb.test.otel.trace.dir= --rerun +``` + +Expected: PASS with the server span's `parentSpanId` == the client **command** span id. Stop and remove the container afterwards. + +- [ ] **Step 4: Commit** + +```bash +git add -f driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md docs/superpowers/runbooks/otel-opmsg-e2e.md +git commit -m "DRIVERS-3454: e2e + prose docs assert command-span parentage" +``` + +--- + +### Task 5: Flip the spec branch to command span + +**Files:** +- Modify: `testing/resources/specifications/source/open-telemetry/open-telemetry.md` (branch `DRIVERS-3454` in the submodule) + +**Interfaces:** +- Consumes: nothing from other tasks (text-only), but merge AFTER Task 3 proves the behavior. + +- [ ] **Step 1: Amend the propagation subsection** + +On submodule branch `DRIVERS-3454` (verify `git branch --show-current`), in the "Propagating Trace Context to the Server" section, replace the operation-span paragraph and condition 3: + +Current text (post-review): "`traceparent` is the ... value of the **operation span**: the propagated context MUST be that of the operation span, not the command span; server spans therefore join the trace as children of the operation span. (In practice, drivers attach the trace context when encoding the wire message, before the command span is created.)" and "3. A valid `traceparent` value is available from the active operation span." + +New text: + +```markdown +`traceparent` is the [W3C traceparent](https://www.w3.org/TR/trace-context/#traceparent-header) value of the **command +span**: the propagated context MUST be that of the command span for the command being sent, so server spans join the +trace as children of the exact command (and retry attempt) that produced them. +``` + +and condition 3: "3. A valid `traceparent` value is available from the command span for the command being sent." Also update the monitoring/auth sentence if it references "operation span" ("Connections that carry no operation span" → "Commands that carry no command span (for example server monitoring, authentication, and security-sensitive commands) naturally send no section, since condition 3 cannot hold."). + +- [ ] **Step 2: Amend the OTel-spec commit and verify hooks** + +```bash +cd testing/resources/specifications +git add source/open-telemetry/open-telemetry.md +git commit --amend --no-edit # amends "DRIVERS-3454: specify trace context propagation to the server" +pre-commit run --files source/open-telemetry/open-telemetry.md +``` + +Expected: hooks pass (if mdformat rewrites, re-add and re-amend until clean). If the OTel commit is not HEAD, use the reset-soft rebuild pattern from the ledger instead of amend. NOTE: the user's fork branch `nabil/nh/otel/DRIVERS-3454` will need another force push (their call — do not push). + +- [ ] **Step 3: Update the parent-repo design docs** + +In the parent repo: `docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md` — add one line under §3.1 noting the 2026-07-18 amendment (command span propagated; see `2026-07-18-command-span-propagation-design.md`). Commit with `git add -f`. + +```bash +git add -f docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md +git commit -m "DRIVERS-3454: note command-span amendment in reference-impl design" +``` + +--- + +### Task 6: Full validation + +**Files:** none. + +- [ ] **Step 1: Full driver-core check (no concurrent Gradle builds!)** + +```bash +./gradlew spotlessApply :driver-core:check -q +``` + +Expected: BUILD SUCCESSFUL, except the known pre-existing environmental failure `TypeMqlValuesFunctionalTest.asStringTestNested` (fails identically on origin/main vs the local server — ignore it; anything else failing must be investigated). + +- [ ] **Step 2: Commit any spotless fixups** + +```bash +git status --porcelain # if formatting changed files: +git add -A driver-core && git commit -m "DRIVERS-3454: formatting fixups" +``` From e295fa074bb18cc6ec23141bd0123f6a7a1b8a6c Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 21 Jul 2026 11:09:55 +0100 Subject: [PATCH 21/48] DRIVERS-3454: createTracingSpan reads the raw command document (no encoded-bytes dependency) --- .../mongodb/internal/connection/CommandMessage.java | 10 ++++++++++ .../internal/connection/InternalStreamConnection.java | 2 -- .../observability/micrometer/TracingManager.java | 4 +--- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java index 81ac10536aa..09ea96be2c6 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +++ b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java @@ -209,6 +209,16 @@ BsonDocument getCommandDocument(final ByteBufferBsonOutput bsonOutput) { } } + /** + * The raw (pre-encoding) command document this message was constructed with. Unlike + * {@link #getCommandDocument(ByteBufferBsonOutput)}, this does not include fields added during encoding + * ({@code $db}, {@code $readPreference}) nor document-sequence fields, but its first key is always the + * command name, and it is available before {@code encode()} runs. + */ + public BsonDocument getRawCommandDocument() { + return command; + } + /** * Get the field name from a buffer positioned at the start of the document sequence identifier of an OP_MSG Section of type * `PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE`. diff --git a/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java b/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java index 0cad654a73a..878b240ae2e 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java +++ b/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java @@ -447,7 +447,6 @@ private T sendAndReceiveInternal(final CommandMessage message, final Decoder .getTracingManager() .createTracingSpan(message, operationContext, - () -> message.getCommandDocument(bsonOutput), cmdName -> SECURITY_SENSITIVE_COMMANDS.contains(cmdName) || SECURITY_SENSITIVE_HELLO_COMMANDS.contains(cmdName), () -> getDescription().getServerAddress(), @@ -623,7 +622,6 @@ private void sendAndReceiveAsyncInternal(final CommandMessage message, final .getTracingManager() .createTracingSpan(message, operationContext, - () -> message.getCommandDocument(bsonOutput), cmdName -> SECURITY_SENSITIVE_COMMANDS.contains(cmdName) || SECURITY_SENSITIVE_HELLO_COMMANDS.contains(cmdName), () -> getDescription().getServerAddress(), diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TracingManager.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TracingManager.java index d177f02fae7..78f3c814eee 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TracingManager.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TracingManager.java @@ -164,7 +164,6 @@ public boolean isCommandPayloadEnabled() { * * @param message the command message to trace * @param operationContext the operation context containing tracing and session information - * @param commandDocumentSupplier a supplier that provides the command document when needed * @param isSensitiveCommand a predicate that determines if a command is security-sensitive based on its name * @param serverAddressSupplier a supplier that provides the server address when needed * @param connectionIdSupplier a supplier that provides the connection ID when needed @@ -173,7 +172,6 @@ public boolean isCommandPayloadEnabled() { @Nullable public Span createTracingSpan(final CommandMessage message, final OperationContext operationContext, - final Supplier commandDocumentSupplier, final Predicate isSensitiveCommand, final Supplier serverAddressSupplier, final Supplier connectionIdSupplier @@ -182,7 +180,7 @@ public Span createTracingSpan(final CommandMessage message, if (!isEnabled()) { return null; } - BsonDocument command = commandDocumentSupplier.get(); + BsonDocument command = message.getRawCommandDocument(); String commandName = command.getFirstKey(); if (isSensitiveCommand.test(commandName)) { return null; From 6db604c0b85434f04466bccb549a65192210c1c8 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 21 Jul 2026 11:42:21 +0100 Subject: [PATCH 22/48] DRIVERS-3454: encode overload carries the command span for the telemetry section --- .../internal/connection/CommandMessage.java | 37 ++++++++++++-- .../CommandMessageOtelTraceContextTest.java | 49 +++++++++++++------ 2 files changed, 66 insertions(+), 20 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java index 09ea96be2c6..39d078017ff 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +++ b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java @@ -111,6 +111,16 @@ public final class CommandMessage extends RequestMessage { private final ClusterConnectionMode clusterConnectionMode; private final ServerApi serverApi; + /** + * The command span to attach to the OP_MSG telemetry section during {@linkplain + * #encode(ByteBufferBsonOutput, OperationContext, Span) encoding}. Set for the duration of that overload's + * call and cleared afterward; {@code null} otherwise (including for the plain 2-arg {@code encode}, which + * never attaches a telemetry section). Not thread-safe: a {@code CommandMessage} is built and encoded by a + * single thread per send attempt, so a transient field is safe. + */ + @Nullable + private Span commandSpanForEncoding; + CommandMessage(final String database, final BsonDocument command, final FieldNameValidator commandFieldNameValidator, final ReadPreference readPreference, final MessageSettings settings, final ClusterConnectionMode clusterConnectionMode, @Nullable final ServerApi serverApi) { @@ -260,6 +270,23 @@ protected void encodeMessageBody(final ByteBufferBsonOutput bsonOutput, final Op this.firstDocumentPosition = useOpMsg() ? writeOpMsg(bsonOutput, operationContext) : writeOpQuery(bsonOutput); } + /** + * Encodes the message, attaching the OP_MSG telemetry section with {@code commandSpan}'s trace context when + * the gating conditions hold (see {@link #writeTelemetryContextSection}). The 2-arg {@link + * #encode(ByteBufferBsonOutput, OperationContext)} never attaches the section. Not thread-safe: a message + * must be encoded by one thread at a time (already the case — a {@code CommandMessage} is built and encoded + * per send attempt). + */ + public void encode(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext, + @Nullable final Span commandSpan) { + this.commandSpanForEncoding = commandSpan; + try { + encode(bsonOutput, operationContext); + } finally { + this.commandSpanForEncoding = null; + } + } + @SuppressWarnings("try") private int writeOpMsg(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext) { int messageStartPosition = bsonOutput.getPosition() - MESSAGE_PROLOGUE_LENGTH; @@ -302,22 +329,22 @@ private int writeOpMsg(final ByteBufferBsonOutput bsonOutput, final OperationCon fail(sequences.toString()); } - writeTelemetryContextSection(bsonOutput, operationContext); + writeTelemetryContextSection(bsonOutput); // Write the flag bits bsonOutput.writeInt32(flagPosition, getOpMsgFlagBits()); return commandStartPosition; } - private void writeTelemetryContextSection(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext) { + private void writeTelemetryContextSection(final ByteBufferBsonOutput bsonOutput) { if (getSettings().getMaxWireVersion() < NINE_DOT_ZERO_WIRE_VERSION) { return; } - Span tracingSpan = operationContext.getTracingSpan(); - if (tracingSpan == null) { + Span commandSpan = this.commandSpanForEncoding; + if (commandSpan == null) { return; } - String traceParent = tracingSpan.context().traceParent(); + String traceParent = commandSpan.context().traceParent(); if (traceParent == null) { return; } diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java index be17456e791..0e98b081eab 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java @@ -63,10 +63,10 @@ class CommandMessageOtelTraceContextTest { void shouldWriteTelemetrySectionWhenWireVersionAtLeast29AndSpanActive() { CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); Span span = spanWithTraceParent(TRACEPARENT); - OperationContext operationContext = buildOperationContext(span); + OperationContext operationContext = buildOperationContext(); try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { - message.encode(output, operationContext); + message.encode(output, operationContext, span); byte[] buffer = output.toByteArray(); BsonDocument telemetry = readTelemetrySectionDocument(buffer); @@ -82,10 +82,10 @@ void shouldWriteTelemetrySectionWhenWireVersionAtLeast29AndSpanActive() { void shouldNotWriteTelemetrySectionWhenWireVersionBelow29() { CommandMessage message = buildCommandMessage(EIGHT_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); Span span = spanWithTraceParent(TRACEPARENT); - OperationContext operationContext = buildOperationContext(span); + OperationContext operationContext = buildOperationContext(); try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { - message.encode(output, operationContext); + message.encode(output, operationContext, span); byte[] buffer = output.toByteArray(); assertNull(findTelemetrySectionDocument(buffer)); @@ -93,16 +93,18 @@ void shouldNotWriteTelemetrySectionWhenWireVersionBelow29() { } /** - * Prose test 3: Telemetry section is omitted without an active span + * Prose test 3: Telemetry section is omitted without an active span. This also covers the case where the + * operation context carries a span for parenting but no command span is passed to encode, since the + * operation context's span is irrelevant to the telemetry section. * (see {@code docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md}) */ @Test void shouldNotWriteTelemetrySectionWhenNoSpan() { CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); - OperationContext operationContext = buildOperationContext(null); + OperationContext operationContext = buildOperationContext(); try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { - message.encode(output, operationContext); + message.encode(output, operationContext, null); byte[] buffer = output.toByteArray(); assertNull(findTelemetrySectionDocument(buffer)); @@ -117,10 +119,10 @@ void shouldNotWriteTelemetrySectionWhenNoSpan() { void shouldNotWriteTelemetrySectionWhenTraceParentNull() { CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); Span span = spanWithTraceParent(null); - OperationContext operationContext = buildOperationContext(span); + OperationContext operationContext = buildOperationContext(); try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { - message.encode(output, operationContext); + message.encode(output, operationContext, span); byte[] buffer = output.toByteArray(); assertNull(findTelemetrySectionDocument(buffer)); @@ -133,10 +135,10 @@ void getCommandDocumentIgnoresTelemetrySection() { // compression). The trailing kind-3 section must not corrupt command-document reconstruction. CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); Span span = spanWithTraceParent(TRACEPARENT); - OperationContext operationContext = buildOperationContext(span); + OperationContext operationContext = buildOperationContext(); try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { - message.encode(output, operationContext); + message.encode(output, operationContext, span); BsonDocument commandDocument = message.getCommandDocument(output); assertEquals("test", commandDocument.getString("find").getValue()); assertEquals("db", commandDocument.getString("$db").getValue()); @@ -154,10 +156,10 @@ void shouldWriteTelemetrySectionAfterDocumentSequences() { true, NoOpFieldNameValidator.INSTANCE); CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, payload); Span span = spanWithTraceParent(TRACEPARENT); - OperationContext operationContext = buildOperationContext(span); + OperationContext operationContext = buildOperationContext(); try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { - message.encode(output, operationContext); + message.encode(output, operationContext, span); byte[] buffer = output.toByteArray(); // Sequence section(s) precede the kind-3 telemetry section; scanning left-to-right and taking @@ -170,6 +172,24 @@ void shouldWriteTelemetrySectionAfterDocumentSequences() { } } + /** + * NEW: The 2-arg {@code encode} inherited from {@code RequestMessage} never attaches the telemetry section, + * even when the wire version supports it. This guards monitoring/compression/other callers that still use + * the 2-arg overload. + */ + @Test + void shouldNotWriteTelemetrySectionViaTwoArgEncode() { + CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); + OperationContext operationContext = buildOperationContext(); + + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + byte[] buffer = output.toByteArray(); + + assertNull(findTelemetrySectionDocument(buffer)); + } + } + // --- helpers --- private static Span spanWithTraceParent(final String traceParent) { @@ -194,7 +214,7 @@ private static CommandMessage buildCommandMessage(final int maxWireVersion, fina null); } - private static OperationContext buildOperationContext(final Span span) { + private static OperationContext buildOperationContext() { SessionContext sessionContext = mock(SessionContext.class, mock -> { when(mock.getClusterTime()).thenReturn(null); when(mock.hasSession()).thenReturn(false); @@ -207,7 +227,6 @@ private static OperationContext buildOperationContext(final Span span) { return mock(OperationContext.class, mock -> { when(mock.getSessionContext()).thenReturn(sessionContext); when(mock.getTimeoutContext()).thenReturn(timeoutContext); - when(mock.getTracingSpan()).thenReturn(span); }); } From 7352282266a5b38ede0ccbf944fc35675cf24314 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 21 Jul 2026 12:06:27 +0100 Subject: [PATCH 23/48] DRIVERS-3454: create command span before encode; propagate its context --- .../connection/InternalStreamConnection.java | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java b/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java index 878b240ae2e..a323f1ba4e2 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java +++ b/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java @@ -442,7 +442,6 @@ private T sendAndReceiveInternal(final CommandMessage message, final Decoder CommandEventSender commandEventSender; Span tracingSpan; try (ByteBufferBsonOutput bsonOutput = new ByteBufferBsonOutput(this)) { - message.encode(bsonOutput, operationContext); tracingSpan = operationContext .getTracingManager() .createTracingSpan(message, @@ -452,6 +451,15 @@ private T sendAndReceiveInternal(final CommandMessage message, final Decoder () -> getDescription().getServerAddress(), () -> getDescription().getConnectionId() ); + try { + message.encode(bsonOutput, operationContext, tracingSpan); + } catch (RuntimeException | Error e) { + if (tracingSpan != null) { + tracingSpan.error(e); + tracingSpan.end(); + } + throw e; + } boolean isLoggingCommandNeeded = isLoggingCommandNeeded(); boolean isTracingCommandPayloadNeeded = tracingSpan != null && operationContext.getTracingManager().isCommandPayloadEnabled(); @@ -616,8 +624,6 @@ private void sendAndReceiveAsyncInternal(final CommandMessage message, final Span tracingSpan = null; try { - message.encode(bsonOutput, operationContext); - tracingSpan = operationContext .getTracingManager() .createTracingSpan(message, @@ -628,6 +634,8 @@ private void sendAndReceiveAsyncInternal(final CommandMessage message, final () -> getDescription().getConnectionId() ); + message.encode(bsonOutput, operationContext, tracingSpan); + CommandEventSender commandEventSender; boolean isLoggingCommandNeeded = isLoggingCommandNeeded(); boolean isTracingCommandPayloadNeeded = tracingSpan != null && operationContext.getTracingManager().isCommandPayloadEnabled(); From b3e5358a9c64066dae009b32d60883ccde421f6b Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 21 Jul 2026 12:30:30 +0100 Subject: [PATCH 24/48] DRIVERS-3454: e2e + prose docs assert command-span parentage --- docs/superpowers/runbooks/otel-opmsg-e2e.md | 33 +++++++++++-------- ...7-13-otel-telemetry-section-prose-tests.md | 12 +++---- .../ServerSpanLinkageProseTest.java | 18 +++++----- 3 files changed, 35 insertions(+), 28 deletions(-) diff --git a/docs/superpowers/runbooks/otel-opmsg-e2e.md b/docs/superpowers/runbooks/otel-opmsg-e2e.md index da690fdabaa..f274f3b3bc1 100644 --- a/docs/superpowers/runbooks/otel-opmsg-e2e.md +++ b/docs/superpowers/runbooks/otel-opmsg-e2e.md @@ -18,6 +18,12 @@ Last executed successfully: 2026-07-14, against after the #56646 merge), linux arm64 dist-test binary run in Docker on a macOS arm64 host. `ServerSpanLinkageProseTest` PASSED. +Re-verified 2026-07-21 against a `9.0.0-alpha0` server in Docker container +`otel-opta` after the driver was changed to inject the traceparent from the +command span (rather than the operation span): `ServerSpanLinkageProseTest` +PASSED, with the server's `find` span `parentSpanId` equal to the client +command span's own span id. + ## Prerequisites - A `mongod` built from `10gen/mongo` `master` at or after the 2026-07-06 merge @@ -151,17 +157,18 @@ Notes: Expected: PASSED. The test runs an instrumented `find`, then polls the trace directory for an exported server span whose `traceId` equals the client trace -id and whose `parentSpanId` equals the driver's **operation** span id. - -Important semantics (verified empirically on 2026-07-14): the driver injects -the traceparent from the `OperationContext`'s active tracing span — the -*operation* span, per the reference-impl design section 3.1 — because -`CommandMessage.encode()` runs before the command span is created in -`InternalStreamConnection.sendAndReceiveInternal()`. The server `find` span is -therefore a *sibling* of the client command span, both children of the -operation span. The first version of the prose test asserted parentage by the -command span and failed against the real server; it was fixed to assert the -operation-span linkage. +id and whose `parentSpanId` equals the driver's **command** span id. + +Important semantics: the driver injects the traceparent from the +`OperationContext`'s active tracing span — the *command* span, per the +reference-impl design section 3.1 — because command-span creation was hoisted +above `CommandMessage.encode()` in `InternalStreamConnection.sendAndReceiveInternal()`. +The server `find` span is therefore a *direct child* of the client command +span. (An earlier iteration of the driver created the command span after +encoding, so the traceparent carried the operation span instead and the +server span was a sibling of the command span; the prose test was adjusted +accordingly at the time but has since been reverted to assert command-span +linkage now that the command span is created before encoding.) Debugging order if it fails: 1. `maxWireVersion >= 29`? If lower, the driver never attaches the section @@ -216,8 +223,8 @@ docker run --rm -d --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/all-i ``` Run the same instrumented `find` and check http://localhost:16686 for a single -trace containing the client operation span, the client command span, and the -server span (the latter two as siblings under the operation span). +trace containing the client operation span, the client command span (a child +of the operation span), and the server span (a child of the command span). ## Notes diff --git a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md index 042bd0afc57..fe71fc49532 100644 --- a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md +++ b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md @@ -13,7 +13,7 @@ connection whose `maxWireVersion` is >= 29. Assert the resulting OP_MSG contains one section of kind 3 whose payload is a BSON document of the form `{otel: {traceparent: }}`, where `traceparent` is 55 characters, `00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>`, and the trace-id/span-id match -the active span's context. +the **command span**'s context. ## 2. Telemetry section is omitted for older servers @@ -35,8 +35,8 @@ present (drivers MUST omit the section rather than send an invalid traceparent). With tracing enabled, run a CRUD operation (e.g. `find`) against the configured 9.0+ server. Capture the driver's finished spans (client side). Then read the server's exported OTLP JSON from the trace directory and assert a server span exists whose -`traceId` equals the client trace-id and whose `parentSpanId` equals the driver's -*operation* span-id (the traceparent is injected from the `OperationContext`'s active -tracing span per the reference-impl design section 3.1, so the server span is a sibling -of the client command span, both parented by the operation span). Allow for the server's -batch export interval when polling. +`traceId` equals the client trace-id and whose `parentSpanId` equals the client +**command** span's span-id (the traceparent is injected from the `OperationContext`'s +active tracing span per the reference-impl design section 3.1, which is now the command +span, so the server span is a direct child of the client command span). Allow for the +server's batch export interval when polling. diff --git a/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java b/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java index 6290eec10fe..9e51f1b80e3 100644 --- a/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java +++ b/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java @@ -76,7 +76,7 @@ void tearDown() { } @Test - @DisplayName("Prose test 5: server emits a span parented by the driver operation span") + @DisplayName("Prose test 5: server emits a span parented by the driver command span") void testServerSpanLinkage() throws Exception { MongoClientSettings clientSettings = getMongoClientSettingsBuilder() .observabilitySettings(ObservabilitySettings.micrometerBuilder() @@ -93,10 +93,10 @@ void testServerSpanLinkage() throws Exception { List clientSpans = inMemoryOtel.getFinishedSpans(); assertTrue(clientSpans.size() >= 2, "expected operation + command client spans, got: " + clientSpans); // Per the reference-impl design (section 3.1), the driver injects the traceparent from the - // OperationContext's active tracing span, i.e. the OPERATION span. The command span is a - // client-side child of the operation span, so the server span is a sibling of the command - // span, both parented by the operation span. Finished-span ordering is not guaranteed, so - // find the command span by name ("find") and take its parent as the operation span id. + // OperationContext's active tracing span, i.e. the COMMAND span. The server span is a + // child of the command span, so the server span's parentSpanId equals the command span's + // own span id. Finished-span ordering is not guaranteed, so find the command span by name + // ("find") and use its own span id as the expected parent of the server span. List commandSpans = clientSpans.stream() .filter(span -> "find".equals(span.getName())) .collect(Collectors.toList()); @@ -106,16 +106,16 @@ void testServerSpanLinkage() throws Exception { long deadline = System.currentTimeMillis() + EXPORT_POLL_TIMEOUT_MS; while (System.currentTimeMillis() < deadline) { for (FinishedSpan commandSpan : commandSpans) { - String operationSpanId = commandSpan.getParentId(); - if (operationSpanId != null - && serverSpanLinked(traceDir, commandSpan.getTraceId(), operationSpanId)) { + String commandSpanId = commandSpan.getSpanId(); + if (commandSpanId != null + && serverSpanLinked(traceDir, commandSpan.getTraceId(), commandSpanId)) { return; } } Thread.sleep(EXPORT_POLL_INTERVAL_MS); } fail("no server span found in " + traceDir - + " with traceId/parentSpanId matching the operation span of any of " + commandSpans); + + " with traceId/parentSpanId matching the command span of any of " + commandSpans); } /** From dc9fa676809d4d6d5e67128d20ca39f5a8a577df Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 21 Jul 2026 12:37:04 +0100 Subject: [PATCH 25/48] DRIVERS-3454: note command-span amendment in reference-impl design --- ...2026-07-13-otel-telemetry-section-reference-impl-design.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md index a3b2be7d103..dacaf0358d8 100644 --- a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md +++ b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md @@ -74,6 +74,10 @@ span (per the DRIVERS-719 implementation), so those exclusions fall out of condi without extra plumbing. `MessageSettings` already carries `maxWireVersion`, so no new capability plumbing is needed. +> **2026-07-18 amendment:** condition 2 now requires the **command span** rather than the +> operation span — see `2026-07-18-command-span-propagation-design.md` for the rationale +> and the updated spec text. + ### 3.2 Wire encoding (`CommandMessage`) In `writeOpMsg()`, after the body and any document-sequence sections, when the gating rule From 80b0c2ff7456ebe8a75c0b28aee9e44206ea349f Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 21 Jul 2026 12:42:47 +0100 Subject: [PATCH 26/48] DRIVERS-3454: fix stale mechanism comment in e2e test --- .../client/observability/ServerSpanLinkageProseTest.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java b/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java index 9e51f1b80e3..725c249f63b 100644 --- a/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java +++ b/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java @@ -92,8 +92,8 @@ void testServerSpanLinkage() throws Exception { List clientSpans = inMemoryOtel.getFinishedSpans(); assertTrue(clientSpans.size() >= 2, "expected operation + command client spans, got: " + clientSpans); - // Per the reference-impl design (section 3.1), the driver injects the traceparent from the - // OperationContext's active tracing span, i.e. the COMMAND span. The server span is a + // Per the command-span propagation design (2026-07-18), the driver injects the traceparent + // of the COMMAND span, passed to CommandMessage.encode(...). The server span is a // child of the command span, so the server span's parentSpanId equals the command span's // own span id. Finished-span ordering is not guaranteed, so find the command span by name // ("find") and use its own span id as the expected parent of the server span. From 4c5e5dc7795b6e88793c8226204e2ff0a6df99f5 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 21 Jul 2026 12:46:45 +0100 Subject: [PATCH 27/48] DRIVERS-3454: correct traceparent-mechanism wording in prose doc and runbook --- docs/superpowers/runbooks/otel-opmsg-e2e.md | 9 +++++---- .../2026-07-13-otel-telemetry-section-prose-tests.md | 9 +++++---- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/runbooks/otel-opmsg-e2e.md b/docs/superpowers/runbooks/otel-opmsg-e2e.md index f274f3b3bc1..f19151a1a6b 100644 --- a/docs/superpowers/runbooks/otel-opmsg-e2e.md +++ b/docs/superpowers/runbooks/otel-opmsg-e2e.md @@ -159,10 +159,11 @@ Expected: PASSED. The test runs an instrumented `find`, then polls the trace directory for an exported server span whose `traceId` equals the client trace id and whose `parentSpanId` equals the driver's **command** span id. -Important semantics: the driver injects the traceparent from the -`OperationContext`'s active tracing span — the *command* span, per the -reference-impl design section 3.1 — because command-span creation was hoisted -above `CommandMessage.encode()` in `InternalStreamConnection.sendAndReceiveInternal()`. +Important semantics: the driver injects the traceparent of the *command* span, +which is created before encoding and passed explicitly to +`CommandMessage.encode(bsonOutput, operationContext, commandSpan)` in +`InternalStreamConnection` (`OperationContext.getTracingSpan()` still returns the +*operation* span and is used only to parent the command span). The server `find` span is therefore a *direct child* of the client command span. (An earlier iteration of the driver created the command span after encoding, so the traceparent carried the operation span instead and the diff --git a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md index fe71fc49532..34ef2c488a5 100644 --- a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md +++ b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md @@ -36,7 +36,8 @@ With tracing enabled, run a CRUD operation (e.g. `find`) against the configured server. Capture the driver's finished spans (client side). Then read the server's exported OTLP JSON from the trace directory and assert a server span exists whose `traceId` equals the client trace-id and whose `parentSpanId` equals the client -**command** span's span-id (the traceparent is injected from the `OperationContext`'s -active tracing span per the reference-impl design section 3.1, which is now the command -span, so the server span is a direct child of the client command span). Allow for the -server's batch export interval when polling. +**command** span's span-id (the traceparent is that of the command span, passed to +`CommandMessage.encode(...)` per the 2026-07-18 command-span propagation design; +`OperationContext.getTracingSpan()` still returns the operation span, used only for +parenting the command span — so the server span is a direct child of the client command +span). Allow for the server's batch export interval when polling. From 8cade287fa2e02926c20d1db477eb23551c79fe9 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 21 Jul 2026 13:33:27 +0100 Subject: [PATCH 28/48] DRIVERS-3454: document pre-existing sync-path span-leak window (future JAVA ticket) --- .../2026-07-21-sync-span-leak-window.md | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 docs/superpowers/followups/2026-07-21-sync-span-leak-window.md diff --git a/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md b/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md new file mode 100644 index 00000000000..98f67baad7e --- /dev/null +++ b/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md @@ -0,0 +1,71 @@ +# Follow-up: sync send-path command-span leak window (pre-existing) + +- **For:** future JAVA ticket (not yet filed) +- **Found by:** final code review of the DRIVERS-3454 command-span propagation change (2026-07-21); confirmed + **pre-existing** — the window is the same size before and after that change, it was not introduced by it. +- **Component:** `driver-core`, `com.mongodb.internal.connection.InternalStreamConnection`, sync path + `sendAndReceiveInternal` (as of commit `4c5e5dc779`, lines ~463–485). + +## The bug + +In the sync send path, the command tracing span (a started Micrometer `Observation`) is created, and encode +failures are handled (`InternalStreamConnection.java:456-462` ends the span and rethrows). The *next* failure +handler is the catch around `sendCommandMessage` (`:488-497`), which also ends the span. But the statements +**between** those two protected regions run with a live span and no handler: + +1. `message.getCommandDocument(bsonOutput)` hydration (`:469`) — re-parses the encoded message; can throw on a + BSON invariant violation; +2. `LoggingCommandEventSender` construction + `commandEventSender.sendStartedEvent()` (`:472-476`) — command + listeners are user code and can throw anything; +3. `tracingSpan.setQueryText(commandDocument)` (`:481`) — serializes the command to (truncated) JSON; +4. `tracingSpan.openScope()` (`:483-485`). + +If any of these throws, the exception propagates out of `sendAndReceiveInternal` while the command span is never +`end()`ed (and for #4's window, potentially with an opened-but-never-closed scope). + +## Impact + +- The command `Observation` never completes → the span never reaches the exporter (or hangs "in flight" in some + backends), and any registered non-tracing observation handlers (metrics/logging) never see `onStop` — so a + command that failed in this window is invisible or counted as forever-running in observability data. +- With `TracingObservationHandler`, an unclosed scope from a failed `openScope()` sequence could leak the span + into the thread's context, corrupting parentage of subsequent unrelated spans on that pooled thread. +- Most likely real-world trigger: a throwing `CommandListener.commandStarted` (user code) with tracing enabled. + +Severity is low-ish (requires a throw in a narrow window, and the command fails anyway), but the trace/metrics +corruption mode (scope leak on a pooled thread) justifies a fix. + +## Why the async path does not have it + +The async equivalent (`:615-704`) wraps the *entire* block — span creation through send initiation — in a single +outer `catch (Throwable)` that ends the span exactly once; after `sendCommandMessageAsync` is invoked, ending is +owned solely by the tracing callback. The sync path evolved differently (try-with-resources + two narrow +catches) and never got whole-block coverage. + +## Suggested fix + +Restructure the sync path so the span has exactly one owner from creation to hand-off, e.g. wrap everything from +just after `createTracingSpan` up to (and including) the `sendCommandMessage` try in one +`catch (RuntimeException | Error e)` that does `error(e)`, `closeScope()` if the scope was opened, `end()`, and +`commandEventSender.sendFailedEvent(e)` when the sender exists — replacing the current two separate catches +(mirroring the async path's single-owner structure). Care points: + +- `closeScope()` must only run if `openScope()` succeeded (track a boolean, or make `closeScope()` idempotent / + safe before open — check `MicrometerSpan`'s scope implementation). +- Do not double-send `sendFailedEvent` (currently only the `sendCommandMessage` catch sends it; the started event + fires at `:476`, so a failure before that must not emit a failed event for a never-started command). +- Keep the encode-failure semantics added by the command-span work: encode failures end the span before any scope + is opened (no `closeScope` on that path). + +## Test idea + +Unit test with a mock/failing `CommandListener` whose `commandStarted` throws, tracing enabled with +`InMemoryOtelSetup` (or a recording `ObservationRegistry`): assert the command observation is stopped with an +error (finished-span count includes the errored command span) and that no observation scope remains open on the +thread afterwards. A second case: `setQueryText` throwing (e.g. command payload capture enabled with a document +that fails JSON serialization via a hostile codec). + +## References + +- Final review notes: `.superpowers/sdd/progress.md` (Option A phase) — finding 1. +- Related design: `docs/superpowers/specs/2026-07-18-command-span-propagation-design.md`. From 7369f1ff683fe030c75b5aa37333e2b0d22d16cb Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Tue, 21 Jul 2026 14:53:00 +0100 Subject: [PATCH 29/48] DRIVERS-3454: record prose-test decision (deferred; interop test chosen, limitation noted) --- ...6-07-18-drivers-3454-spec-change-design.md | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md b/docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md index 35f435993fb..81bda87c548 100644 --- a/docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md +++ b/docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md @@ -76,3 +76,34 @@ split across the two specs that own each layer, in the `mongodb/specifications` - Run available repo checks locally (`pre-commit run --files ` and/or `mkdocs build --strict`) if the tooling is installed; otherwise verify 120-char width and markdown lint manually and note it. + +--- + +## Decision record (2026-07-21): spec prose test — chosen candidate, deferred + +**Decision:** the spec PR remains normative-text-only for now; no prose test was added to +`source/open-telemetry/tests/README.md`. When the DRIVERS review asks for a test plan (the PR template expects +one), the agreed candidate is a single appended prose test (*Test 3*, numbering is load-bearing — existing tests +1–2 must not be renumbered): + +> **Test 3: Trace Context Propagation Does Not Affect Command Execution.** With tracing enabled, run CRUD +> operations across the standard server-version/topology matrix, on both sides of the wire-version gate +> (`maxWireVersion` < 29 and >= 29), and assert the operations succeed and spans are emitted. + +**Why this candidate (and not the alternatives):** cross-driver prose tests can only rely on observables every +driver has. The telemetry section is invisible to command monitoring/APM (transport-layer), and server spans are +observable only with special mongod startup parameters (OTLP exporter + feature flag) that no driver CI matrix +provisions. That rules out prescribing (a) wire-level gating-matrix assertions (driver-internal byte inspection) +and (b) the full server-span-linkage e2e (infra burden other driver teams would reject). The chosen test instead +uses the **server's strict validation as the oracle**: a 9.0+ server `uasserts` on any malformed traceparent and +a pre-9.0 server rejects unknown OP_MSG section kinds, so plain command success across the matrix simultaneously +verifies the wire-version gate and the well-formedness of anything sent — and directly tests the spec's key +operational property that telemetry must never become an availability failure. It runs on existing CI with zero +new infrastructure. + +**Known limitation (accepted):** the test cannot prove the section is ever *sent* — a driver that never attaches +the section passes trivially (false-negative tolerant). Positive proof of emission/omission stays in +driver-internal unit tests (in the Java reference implementation: the `CommandMessageOtelTraceContextTest` +OP_MSG round-trip matrix and the `ServerSpanLinkageProseTest` e2e, see +`docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md`); the spec test text should recommend — +not mandate — equivalent driver-level verification. From e34ba1fd96ff010b85a1a55d2ca43083c8d850d6 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Wed, 22 Jul 2026 10:41:41 +0100 Subject: [PATCH 30/48] DRIVERS-3454: clarify getRawCommandDocument javadoc --- .../com/mongodb/internal/connection/CommandMessage.java | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java index 39d078017ff..e48576faabf 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +++ b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java @@ -220,10 +220,11 @@ BsonDocument getCommandDocument(final ByteBufferBsonOutput bsonOutput) { } /** - * The raw (pre-encoding) command document this message was constructed with. Unlike - * {@link #getCommandDocument(ByteBufferBsonOutput)}, this does not include fields added during encoding - * ({@code $db}, {@code $readPreference}) nor document-sequence fields, but its first key is always the - * command name, and it is available before {@code encode()} runs. + * The raw command document this message was constructed with, usable before {@code encode()} runs + * (unlike {@link #getCommandDocument(ByteBufferBsonOutput)}, which re-parses the encoded buffer). + * It lacks the fields added during encoding ({@code $db}, {@code $readPreference}, ...) and any + * document-sequence fields; since encoding only appends fields, its first key is the command name, + * same as in the encoded document. */ public BsonDocument getRawCommandDocument() { return command; From fc850bf89891f36b29ff6c5267a63901db94e662 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Wed, 22 Jul 2026 11:56:50 +0100 Subject: [PATCH 31/48] DRIVERS-3454: narrow raw-command access to getCommandName/getGetMoreCursorId; use for compression checks --- .../internal/connection/CommandMessage.java | 24 +++++++++++++------ .../connection/InternalStreamConnection.java | 4 ++-- .../micrometer/TracingManager.java | 11 ++++----- 3 files changed, 24 insertions(+), 15 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java index e48576faabf..eea6d4c94c2 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +++ b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java @@ -35,6 +35,7 @@ import org.bson.BsonElement; import org.bson.BsonInt64; import org.bson.BsonString; +import org.bson.BsonValue; import org.bson.ByteBuf; import org.bson.FieldNameValidator; import org.bson.codecs.BsonDocumentCodec; @@ -220,14 +221,23 @@ BsonDocument getCommandDocument(final ByteBufferBsonOutput bsonOutput) { } /** - * The raw command document this message was constructed with, usable before {@code encode()} runs - * (unlike {@link #getCommandDocument(ByteBufferBsonOutput)}, which re-parses the encoded buffer). - * It lacks the fields added during encoding ({@code $db}, {@code $readPreference}, ...) and any - * document-sequence fields; since encoding only appends fields, its first key is the command name, - * same as in the encoded document. + * The command name (first key of the command document — for MongoDB commands the first element + * names the command). Available before {@code encode()} runs, unlike + * {@link #getCommandDocument(ByteBufferBsonOutput)}; identical to the encoded document's first + * key, since encoding only appends fields. */ - public BsonDocument getRawCommandDocument() { - return command; + public String getCommandName() { + return command.getFirstKey(); + } + + /** + * The cursor id if this is a {@code getMore} command, or {@code null} otherwise. + * Available before {@code encode()} runs. + */ + @Nullable + public BsonInt64 getGetMoreCursorId() { + BsonValue value = command.get("getMore"); + return value instanceof BsonInt64 ? (BsonInt64) value : null; } /** diff --git a/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java b/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java index a323f1ba4e2..a85b85ad618 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java +++ b/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java @@ -535,7 +535,7 @@ private void sendCommandMessage(final CommandMessage message, final ByteBufferBs final OperationContext operationContext) { Compressor localSendCompressor = sendCompressor; - if (localSendCompressor == null || SECURITY_SENSITIVE_COMMANDS.contains(message.getCommandDocument(bsonOutput).getFirstKey())) { + if (localSendCompressor == null || SECURITY_SENSITIVE_COMMANDS.contains(message.getCommandName())) { trySendMessage(message, bsonOutput, operationContext); } else { ByteBufferBsonOutput compressedBsonOutput; @@ -676,7 +676,7 @@ private void sendAndReceiveAsyncInternal(final CommandMessage message, final commandEventSender.sendStartedEvent(); Compressor localSendCompressor = sendCompressor; - if (localSendCompressor == null || SECURITY_SENSITIVE_COMMANDS.contains(message.getCommandDocument(bsonOutput).getFirstKey())) { + if (localSendCompressor == null || SECURITY_SENSITIVE_COMMANDS.contains(message.getCommandName())) { sendCommandMessageAsync(message.getId(), decoder, operationContext, tracingCallback, bsonOutput, commandEventSender, message.isResponseExpected()); } else { diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TracingManager.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TracingManager.java index 78f3c814eee..6e25ad2779e 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TracingManager.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TracingManager.java @@ -30,7 +30,7 @@ import com.mongodb.observability.micrometer.MongodbObservation; import com.mongodb.observability.micrometer.MongodbObservationContext; import io.micrometer.observation.ObservationRegistry; -import org.bson.BsonDocument; +import org.bson.BsonInt64; import java.util.function.Predicate; import java.util.function.Supplier; @@ -180,8 +180,7 @@ public Span createTracingSpan(final CommandMessage message, if (!isEnabled()) { return null; } - BsonDocument command = message.getRawCommandDocument(); - String commandName = command.getFirstKey(); + String commandName = message.getCommandName(); if (isSensitiveCommand.test(commandName)) { return null; } @@ -222,9 +221,9 @@ public Span createTracingSpan(final CommandMessage message, ConnectionId connectionId = connectionIdSupplier.get(); mongodbContext.setConnectionId(connectionId); - if (command.containsKey("getMore")) { - long cursorId = command.getInt64("getMore").longValue(); - mongodbContext.setCursorId(cursorId); + BsonInt64 getMoreCursorId = message.getGetMoreCursorId(); + if (getMoreCursorId != null) { + mongodbContext.setCursorId(getMoreCursorId.longValue()); } SessionContext sessionContext = operationContext.getSessionContext(); From 1bd5edf808226ce4a1cc330b33a138d19962aaca Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Wed, 22 Jul 2026 13:26:14 +0100 Subject: [PATCH 32/48] DRIVERS-3454: refine span-leak follow-up (unconditional closeScope, event pairing, exception-type unification) --- .../2026-07-21-sync-span-leak-window.md | 35 +++++++++++++------ 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md b/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md index 98f67baad7e..f982ab8af20 100644 --- a/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md +++ b/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md @@ -44,18 +44,31 @@ catches) and never got whole-block coverage. ## Suggested fix -Restructure the sync path so the span has exactly one owner from creation to hand-off, e.g. wrap everything from -just after `createTracingSpan` up to (and including) the `sendCommandMessage` try in one -`catch (RuntimeException | Error e)` that does `error(e)`, `closeScope()` if the scope was opened, `end()`, and -`commandEventSender.sendFailedEvent(e)` when the sender exists — replacing the current two separate catches -(mirroring the async path's single-owner structure). Care points: - -- `closeScope()` must only run if `openScope()` succeeded (track a boolean, or make `closeScope()` idempotent / - safe before open — check `MicrometerSpan`'s scope implementation). +Restructure the sync path so the span has exactly one owner from creation to hand-off: wrap everything from just +after `createTracingSpan` up to (and including) `sendCommandMessage` in one `catch (RuntimeException | Error e)` +that does `error(e)`, `closeScope()`, `end()`, and `commandEventSender.sendFailedEvent(e)` — replacing the +current two separate catches (mirroring the async path's single-owner structure). This also removes the +duplicated error/end snippets between the encode catch and the send catch. + +The unified catch can call `closeScope()` unconditionally — no "was the scope opened" flag is needed (reviewed +2026-07-22): `MicrometerSpan.closeScope()` null-guards its `scope` field and nulls it after closing, so calling +it before `openScope()` or twice is a no-op, and `Span.EMPTY` no-ops trivially. Care points: + +- **Promote that no-op behavior from implementation detail to interface contract**: add to + `Span.closeScope()`'s javadoc — "Calling this without a prior `openScope()`, or more than once, is a no-op." — + so the unified error path (and any future `Span` implementation or test double) can rely on it, and so a later + refactor doesn't "simplify away" the null-guard. Note `openScope()`'s current javadoc ("Must be paired with + {@link #closeScope()} in a try-finally block") reads stricter than this usage; adjust wording accordingly. - Do not double-send `sendFailedEvent` (currently only the `sendCommandMessage` catch sends it; the started event - fires at `:476`, so a failure before that must not emit a failed event for a never-started command). -- Keep the encode-failure semantics added by the command-span work: encode failures end the span before any scope - is opened (no `closeScope` on that path). + fires at `:476`). Initialize `commandEventSender` to `NoOpCommandEventSender` before the block so failures + prior to sender creation no-op. New edge to decide consciously: if `sendStartedEvent` itself throws, the + logging sender is already assigned and a failed event would be emitted for a command whose started event never + completed — arguably more correct than today's silence, but it is a command-monitoring behavior change. +- Unify the exception types: the current send catch is `catch (Exception)`, which misses `Error` (the encode + catch already handles `RuntimeException | Error`); pick `RuntimeException | Error` (or `Throwable`, matching + the async path) for the unified catch. +- Keep the encode-failure semantics added by the command-span work: an encode failure ends the span before any + scope was opened — with unconditional `closeScope()` this holds automatically per the no-op contract above. ## Test idea From 746de73d1024c03ed0f65d46cd8358b6f79d5c4b Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Wed, 22 Jul 2026 13:37:58 +0100 Subject: [PATCH 33/48] =?UTF-8?q?DRIVERS-3454:=20follow-up=20doc=20?= =?UTF-8?q?=E2=80=94=20settle=20exception=20unification=20on=20catch(Throw?= =?UTF-8?q?able)=20with=20precise=20rethrow?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../followups/2026-07-21-sync-span-leak-window.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md b/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md index f982ab8af20..c3338e1af5d 100644 --- a/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md +++ b/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md @@ -64,9 +64,12 @@ it before `openScope()` or twice is a no-op, and `Span.EMPTY` no-ops trivially. prior to sender creation no-op. New edge to decide consciously: if `sendStartedEvent` itself throws, the logging sender is already assigned and a failed event would be emitted for a command whose started event never completed — arguably more correct than today's silence, but it is a command-monitoring behavior change. -- Unify the exception types: the current send catch is `catch (Exception)`, which misses `Error` (the encode - catch already handles `RuntimeException | Error`); pick `RuntimeException | Error` (or `Throwable`, matching - the async path) for the unified catch. +- Unify the exception types on `catch (Throwable t)` + `throw t;`, matching the async path (decided 2026-07-22). + This compiles without signature changes via Java 7 precise rethrow (the catch parameter is effectively final + and the try body declares no checked exceptions — the same rule the current `catch (Exception) { throw e; }` + already relies on), fixes the current send catch's `Error` blind spot, and forces a compile error if a future + edit introduces a checked-exception throw site. Trade-off (accepted, same as async): fatal errors like OOM run + best-effort cleanup before propagating. - Keep the encode-failure semantics added by the command-span work: an encode failure ends the span before any scope was opened — with unconditional `closeScope()` this holds automatically per the no-op contract above. From b4c8e8caf31c4b84eecc15bb5488986ac7780e40 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Wed, 22 Jul 2026 14:08:57 +0100 Subject: [PATCH 34/48] JAVA-XXXX DRIVERS-3454: fix sync-path span-leak window with single-owner error handling - InternalStreamConnection sync path: one catch(Throwable) owns span cleanup and failed-event emission from span creation through sendCommandMessage, covering the previously unprotected hydration/sendStartedEvent/setQueryText/ openScope window (span leak). Uses precise rethrow; also fixes the old send catch's Error blind spot. - LoggingCommandEventSender: terminal events/logs are suppressed until the started event completes, so error paths may emit unconditionally without pairing knowledge. - Span.closeScope(): no-op-without-open promoted to interface contract. --- .../connection/InternalStreamConnection.java | 39 +++++++------------ .../connection/LoggingCommandEventSender.java | 14 +++++++ .../observability/micrometer/Span.java | 4 +- ...gingCommandEventSenderSpecification.groovy | 29 ++++++++++++++ 4 files changed, 61 insertions(+), 25 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java b/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java index a85b85ad618..34accfc332d 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java +++ b/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java @@ -439,8 +439,12 @@ public boolean reauthenticationIsTriggered(@Nullable final Throwable t) { @Nullable private T sendAndReceiveInternal(final CommandMessage message, final Decoder decoder, final OperationContext operationContext) { - CommandEventSender commandEventSender; - Span tracingSpan; + CommandEventSender commandEventSender = new NoOpCommandEventSender(); + Span tracingSpan = null; + // Single owner of the span and event sender until the message is handed off to the receive phase: + // any failure from span creation through sendCommandMessage ends the span exactly once and emits at + // most one failed event (LoggingCommandEventSender suppresses terminal events until the started event + // has completed, and Span.closeScope() is a no-op if no scope was opened). try (ByteBufferBsonOutput bsonOutput = new ByteBufferBsonOutput(this)) { tracingSpan = operationContext .getTracingManager() @@ -451,15 +455,7 @@ private T sendAndReceiveInternal(final CommandMessage message, final Decoder () -> getDescription().getServerAddress(), () -> getDescription().getConnectionId() ); - try { - message.encode(bsonOutput, operationContext, tracingSpan); - } catch (RuntimeException | Error e) { - if (tracingSpan != null) { - tracingSpan.error(e); - tracingSpan.end(); - } - throw e; - } + message.encode(bsonOutput, operationContext, tracingSpan); boolean isLoggingCommandNeeded = isLoggingCommandNeeded(); boolean isTracingCommandPayloadNeeded = tracingSpan != null && operationContext.getTracingManager().isCommandPayloadEnabled(); @@ -474,8 +470,6 @@ private T sendAndReceiveInternal(final CommandMessage message, final Decoder operationContext, message, commandDocument, COMMAND_PROTOCOL_LOGGER, loggerSettings); commandEventSender.sendStartedEvent(); - } else { - commandEventSender = new NoOpCommandEventSender(); } if (isTracingCommandPayloadNeeded) { tracingSpan.setQueryText(commandDocument); @@ -483,18 +477,15 @@ private T sendAndReceiveInternal(final CommandMessage message, final Decoder if (tracingSpan != null) { tracingSpan.openScope(); } - - try { - sendCommandMessage(message, bsonOutput, operationContext); - } catch (Exception e) { - if (tracingSpan != null) { - tracingSpan.error(e); - tracingSpan.closeScope(); - tracingSpan.end(); - } - commandEventSender.sendFailedEvent(e); - throw e; + sendCommandMessage(message, bsonOutput, operationContext); + } catch (Throwable t) { + if (tracingSpan != null) { + tracingSpan.error(t); + tracingSpan.closeScope(); + tracingSpan.end(); } + commandEventSender.sendFailedEvent(t); + throw t; } if (message.isResponseExpected()) { diff --git a/driver-core/src/main/com/mongodb/internal/connection/LoggingCommandEventSender.java b/driver-core/src/main/com/mongodb/internal/connection/LoggingCommandEventSender.java index 1972e2caaab..6da708d0518 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/LoggingCommandEventSender.java +++ b/driver-core/src/main/com/mongodb/internal/connection/LoggingCommandEventSender.java @@ -72,6 +72,13 @@ class LoggingCommandEventSender implements CommandEventSender { private final String commandName; private volatile BsonDocument commandDocument; private final boolean redactionRequired; + /** + * Guards event pairing: terminal (failed/succeeded) events and logs are emitted only after the started + * event has completed, so callers may invoke {@link #sendFailedEvent} unconditionally from any error path + * without pairing knowledge. Volatile because the started and terminal events may fire on different + * threads in the async path. + */ + private volatile boolean startedEventCompleted; LoggingCommandEventSender(final Set securitySensitiveCommands, final Set securitySensitiveHelloCommands, final ConnectionDescription description, @@ -117,11 +124,15 @@ public void sendStartedEvent() { // the buffer underlying the command document may be released after the started event, so set to null to ensure it's not used // when sending the failed or succeeded event commandDocument = null; + startedEventCompleted = true; } @Override public void sendFailedEvent(final Throwable t) { + if (!startedEventCompleted) { + return; + } Throwable commandEventException = t; if (t instanceof MongoCommandException && redactionRequired) { commandEventException = MongoCommandExceptionUtils.redacted((MongoCommandException) t); @@ -157,6 +168,9 @@ public void sendSucceededEventForOneWayCommand() { } private void sendSucceededEvent(final BsonDocument reply) { + if (!startedEventCompleted) { + return; + } long elapsedTimeNanos = System.nanoTime() - startTimeNanos; if (loggingRequired()) { diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/Span.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/Span.java index d060c82b0d4..91902057db4 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/Span.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/Span.java @@ -90,12 +90,14 @@ public MongodbObservationContext getMongodbObservationContext() { /** * Opens a scope for this span, making it the current observation on the thread. - * Must be paired with {@link #closeScope()} in a try-finally block. + * A successful {@code openScope()} must eventually be followed by {@link #closeScope()}. */ void openScope(); /** * Closes the scope previously opened by {@link #openScope()}, restoring the previous observation. + * Calling this without a prior {@code openScope()}, or more than once, is a no-op — error-handling + * paths may therefore call it unconditionally. */ void closeScope(); diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/LoggingCommandEventSenderSpecification.groovy b/driver-core/src/test/unit/com/mongodb/internal/connection/LoggingCommandEventSenderSpecification.groovy index e6f6afb02e0..00faee01f0a 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/LoggingCommandEventSenderSpecification.groovy +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/LoggingCommandEventSenderSpecification.groovy @@ -91,6 +91,35 @@ class LoggingCommandEventSenderSpecification extends Specification { debugLoggingEnabled << [true, false] } + def 'should not send failed or succeeded events if the started event was not completed'() { + given: + def connectionDescription = new ConnectionDescription(new ServerId(new ClusterId(), new ServerAddress())) + def database = 'test' + def messageSettings = MessageSettings.builder().maxWireVersion(LATEST_WIRE_VERSION).build() + def commandListener = new TestCommandListener() + def commandDocument = new BsonDocument('ping', new BsonInt32(1)) + def replyDocument = new BsonDocument('ok', new BsonInt32(1)) + def message = new CommandMessage(database, commandDocument, + NoOpFieldNameValidator.INSTANCE, ReadPreference.primary(), messageSettings, MULTIPLE, null) + def bsonOutput = new ByteBufferBsonOutput(new SimpleBufferProvider()) + message.encode(bsonOutput, new OperationContext(IgnorableRequestContext.INSTANCE, NoOpSessionContext.INSTANCE, + Stub(TimeoutContext), null)) + def logger = Stub(Logger) { + isDebugEnabled() >> false + } + def sender = new LoggingCommandEventSender([] as Set, [] as Set, connectionDescription, commandListener, + OPERATION_CONTEXT, message, message.getCommandDocument(bsonOutput), + new StructuredLogger(logger), LoggerSettings.builder().build()) + + when: 'terminal events are sent without a completed started event' + sender.sendFailedEvent(new MongoInternalException('failure!')) + sender.sendSucceededEvent(MessageHelper.buildSuccessfulReply(message.getId(), replyDocument.toJson())) + sender.sendSucceededEventForOneWayCommand() + + then: 'no events are delivered' + commandListener.getEvents().isEmpty() + } + def 'should log events'() { given: def serverId = new ServerId(new ClusterId(), new ServerAddress()) From 6c709f73431cc94102a304a75091d38369d18524 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Wed, 22 Jul 2026 14:09:15 +0100 Subject: [PATCH 35/48] DRIVERS-3454: mark span-leak follow-up implemented --- .../followups/2026-07-21-sync-span-leak-window.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md b/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md index c3338e1af5d..ac728aac014 100644 --- a/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md +++ b/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md @@ -1,5 +1,11 @@ # Follow-up: sync send-path command-span leak window (pre-existing) +- **Status: IMPLEMENTED 2026-07-22** in commit `b4c8e8caf3` (single-owner `catch (Throwable)` in the sync path, + `LoggingCommandEventSender` started-event pairing guard [option C], `Span.closeScope()` no-op contract javadoc). + Verified: `LoggingCommandEventSenderSpecification` (incl. new pairing test), `InternalStreamConnection*`, + `CommandMessageOtelTraceContextTest`, micrometer suites (124 tests green), and `MicrometerProseTest` against a + live wire-29 server. A JAVA ticket should still be filed retroactively for release notes; the commit message + carries a `JAVA-XXXX` placeholder to amend once the ticket exists. - **For:** future JAVA ticket (not yet filed) - **Found by:** final code review of the DRIVERS-3454 command-span propagation change (2026-07-21); confirmed **pre-existing** — the window is the same size before and after that change, it was not introduced by it. @@ -75,6 +81,11 @@ it before `openScope()` or twice is a no-op, and `Span.EMPTY` no-ops trivially. ## Test idea +> Note (2026-07-22): the throwing-`CommandListener` variant below is NOT viable — user listener exceptions are +> already swallowed in `ProtocolHelper.sendCommandStartedEvent` (`catch (Exception)` + warn log), so they cannot +> reach this window. The implemented test instead unit-tests the sender's pairing guard directly +> (`LoggingCommandEventSenderSpecification`: terminal events without a completed started event are suppressed). + Unit test with a mock/failing `CommandListener` whose `commandStarted` throws, tracing enabled with `InMemoryOtelSetup` (or a recording `ObservationRegistry`): assert the command observation is stopped with an error (finished-span count includes the errored command span) and that no observation scope remains open on the From 8f3940ae8f1ec30d5cdc875bfa137e0bb7eed8cc Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Wed, 22 Jul 2026 18:30:39 +0100 Subject: [PATCH 36/48] DRIVERS-3454: extract expected telemetry document constant in test --- .../CommandMessageOtelTraceContextTest.java | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java index 0e98b081eab..11f4fc6447b 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java @@ -52,12 +52,13 @@ class CommandMessageOtelTraceContextTest { private static final String TRACEPARENT = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"; + private static final BsonDocument EXPECTED_TELEMETRY_DOCUMENT = + new BsonDocument("otel", new BsonDocument("traceparent", new BsonString(TRACEPARENT))); private static final MongoNamespace NAMESPACE = new MongoNamespace("db.test"); private static final BsonDocument COMMAND = new BsonDocument("find", new BsonString(NAMESPACE.getCollectionName())); /** - * Prose test 1: Telemetry section is attached when supported and traced - * (see {@code docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md}) + * Telemetry section is attached when supported and traced. */ @Test void shouldWriteTelemetrySectionWhenWireVersionAtLeast29AndSpanActive() { @@ -70,7 +71,7 @@ void shouldWriteTelemetrySectionWhenWireVersionAtLeast29AndSpanActive() { byte[] buffer = output.toByteArray(); BsonDocument telemetry = readTelemetrySectionDocument(buffer); - assertEquals(new BsonDocument("otel", new BsonDocument("traceparent", new BsonString(TRACEPARENT))), telemetry); + assertEquals(EXPECTED_TELEMETRY_DOCUMENT, telemetry); } } @@ -165,7 +166,7 @@ void shouldWriteTelemetrySectionAfterDocumentSequences() { // Sequence section(s) precede the kind-3 telemetry section; scanning left-to-right and taking // the first kind-3 occurrence therefore validates ordering as well as content. BsonDocument telemetry = readTelemetrySectionDocument(buffer); - assertEquals(new BsonDocument("otel", new BsonDocument("traceparent", new BsonString(TRACEPARENT))), telemetry); + assertEquals(EXPECTED_TELEMETRY_DOCUMENT, telemetry); BsonDocument commandDocument = message.getCommandDocument(output); assertEquals("test", commandDocument.getString("find").getValue()); @@ -231,9 +232,8 @@ private static OperationContext buildOperationContext() { } /** - * Walks the OP_MSG sections exactly like the production parser (see {@code CommandMessage#getCommandDocument}): - * skip the type-0 body document, then for each subsequent section read the kind byte; for kind 1 - * (document sequence) skip past the {@code int32} section size, and for kind 3 (telemetry) decode and + * Walks the OP_MSG sections skip the type-0 body document, then for each subsequent section read the kind byte; + * for kind 1 (document sequence) skip past the {@code int32} section size, and for kind 3 (telemetry) decode and * return the BSON document payload. Returns {@code null} if no kind-3 section is found. */ private static BsonDocument findTelemetrySectionDocument(final byte[] buffer) { From 7ecaf5740ad207e11d60ca2751111a85bf23b34e Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Wed, 22 Jul 2026 18:43:46 +0100 Subject: [PATCH 37/48] DRIVERS-3454: assert reconstructed command has no otel key --- .../CommandMessageOtelTraceContextTest.java | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java index 11f4fc6447b..d8786f9d7ea 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java @@ -46,6 +46,7 @@ import static com.mongodb.internal.operation.ServerVersionHelper.EIGHT_DOT_ZERO_WIRE_VERSION; import static com.mongodb.internal.operation.ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNull; import static org.mockito.Mockito.when; @@ -76,8 +77,7 @@ void shouldWriteTelemetrySectionWhenWireVersionAtLeast29AndSpanActive() { } /** - * Prose test 2: Telemetry section is omitted for older servers - * (see {@code docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md}) + * Telemetry section is omitted for older servers */ @Test void shouldNotWriteTelemetrySectionWhenWireVersionBelow29() { @@ -94,10 +94,7 @@ void shouldNotWriteTelemetrySectionWhenWireVersionBelow29() { } /** - * Prose test 3: Telemetry section is omitted without an active span. This also covers the case where the - * operation context carries a span for parenting but no command span is passed to encode, since the - * operation context's span is irrelevant to the telemetry section. - * (see {@code docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md}) + * Telemetry section is omitted without an active span. */ @Test void shouldNotWriteTelemetrySectionWhenNoSpan() { @@ -113,8 +110,7 @@ void shouldNotWriteTelemetrySectionWhenNoSpan() { } /** - * Prose test 4: Malformed trace context is never sent - * (see {@code docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md}) + * Malformed trace context is never sent. */ @Test void shouldNotWriteTelemetrySectionWhenTraceParentNull() { @@ -144,7 +140,8 @@ void getCommandDocumentIgnoresTelemetrySection() { assertEquals("test", commandDocument.getString("find").getValue()); assertEquals("db", commandDocument.getString("$db").getValue()); // The reconstructed command is exactly the body section (find + $db); the trailing - // kind-3 section must not leak any extra fields into it. + // kind-3 section must not leak into it. + assertFalse(commandDocument.containsKey("otel")); assertEquals(2, commandDocument.size()); } } From b9f56be2b4a31132bda5566de5a0d8e3f8795d53 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Wed, 22 Jul 2026 18:48:20 +0100 Subject: [PATCH 38/48] DRIVERS-3454: cover DualMessageSequences (clientBulkWrite path) in telemetry round-trip tests --- .../CommandMessageOtelTraceContextTest.java | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java index d8786f9d7ea..19c4075ae24 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java @@ -32,11 +32,13 @@ import com.mongodb.internal.validator.NoOpFieldNameValidator; import org.bson.BsonBinaryReader; import org.bson.BsonDocument; +import org.bson.BsonInt32; import org.bson.BsonString; import org.bson.ByteBuf; import org.bson.ByteBufNIO; import org.bson.codecs.BsonDocumentCodec; import org.bson.codecs.DecoderContext; +import org.bson.codecs.EncoderContext; import org.junit.jupiter.api.Test; import java.nio.ByteBuffer; @@ -170,6 +172,47 @@ void shouldWriteTelemetrySectionAfterDocumentSequences() { } } + /** + * Same guarantee as {@link #shouldWriteTelemetrySectionAfterDocumentSequences()}, but for the + * {@link DualMessageSequences} branch (the {@code clientBulkWrite} path). It is the only branch that + * assembles its two kind-1 sections via {@link ByteBufferBsonOutput#branch()} (out-of-order buffer + * segments merged on close), so the trailing kind-3 section's placement is worth pinning separately. + */ + @Test + void shouldWriteTelemetrySectionAfterDualMessageSequences() { + DualMessageSequences sequences = new DualMessageSequences( + "ops", NoOpFieldNameValidator.INSTANCE, "nsInfo", NoOpFieldNameValidator.INSTANCE) { + @Override + public EncodeDocumentsResult encodeDocuments(final WritersProviderAndLimitsChecker writersProviderAndLimitsChecker) { + writersProviderAndLimitsChecker.tryWrite((firstWriter, secondWriter) -> { + new BsonDocumentCodec().encode(firstWriter, + new BsonDocument("insert", new BsonInt32(0)), + EncoderContext.builder().build()); + new BsonDocumentCodec().encode(secondWriter, + new BsonDocument("ns", new BsonString(NAMESPACE.getFullName())), + EncoderContext.builder().build()); + return 1; + }); + return new EncodeDocumentsResult(true, Collections.emptyList()); + } + }; + CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, sequences); + Span span = spanWithTraceParent(TRACEPARENT); + OperationContext operationContext = buildOperationContext(); + + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext, span); + byte[] buffer = output.toByteArray(); + + BsonDocument telemetry = readTelemetrySectionDocument(buffer); + assertEquals(EXPECTED_TELEMETRY_DOCUMENT, telemetry); + + BsonDocument commandDocument = message.getCommandDocument(output); + assertEquals("test", commandDocument.getString("find").getValue()); + assertFalse(commandDocument.containsKey("otel")); + } + } + /** * NEW: The 2-arg {@code encode} inherited from {@code RequestMessage} never attaches the telemetry section, * even when the wire version supports it. This guards monitoring/compression/other callers that still use From b96eef98805216f4ab77c85b976763fbe2217a30 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Wed, 22 Jul 2026 18:55:18 +0100 Subject: [PATCH 39/48] DRIVERS-3454: parametrize telemetry-section omission cases --- .../CommandMessageOtelTraceContextTest.java | 57 +++++++------------ 1 file changed, 19 insertions(+), 38 deletions(-) diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java index 19c4075ae24..3a5d8014cba 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java @@ -30,6 +30,7 @@ import com.mongodb.internal.observability.micrometer.TraceContext; import com.mongodb.internal.session.SessionContext; import com.mongodb.internal.validator.NoOpFieldNameValidator; +import com.mongodb.lang.Nullable; import org.bson.BsonBinaryReader; import org.bson.BsonDocument; import org.bson.BsonInt32; @@ -40,9 +41,13 @@ import org.bson.codecs.DecoderContext; import org.bson.codecs.EncoderContext; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; import java.nio.ByteBuffer; import java.util.Collections; +import java.util.stream.Stream; import static com.mongodb.internal.mockito.MongoMockito.mock; import static com.mongodb.internal.operation.ServerVersionHelper.EIGHT_DOT_ZERO_WIRE_VERSION; @@ -50,6 +55,8 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Named.named; +import static org.junit.jupiter.params.provider.Arguments.arguments; import static org.mockito.Mockito.when; class CommandMessageOtelTraceContextTest { @@ -78,46 +85,20 @@ void shouldWriteTelemetrySectionWhenWireVersionAtLeast29AndSpanActive() { } } - /** - * Telemetry section is omitted for older servers - */ - @Test - void shouldNotWriteTelemetrySectionWhenWireVersionBelow29() { - CommandMessage message = buildCommandMessage(EIGHT_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); - Span span = spanWithTraceParent(TRACEPARENT); - OperationContext operationContext = buildOperationContext(); - - try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { - message.encode(output, operationContext, span); - byte[] buffer = output.toByteArray(); - - assertNull(findTelemetrySectionDocument(buffer)); - } + private static Stream telemetrySectionOmittedCases() { + return Stream.of( + arguments(named("wire version below 29", EIGHT_DOT_ZERO_WIRE_VERSION), + named("valid span", spanWithTraceParent(TRACEPARENT))), + arguments(named("wire version 29", NINE_DOT_ZERO_WIRE_VERSION), + named("no command span", null)), + arguments(named("wire version 29", NINE_DOT_ZERO_WIRE_VERSION), + named("span without traceparent", spanWithTraceParent(null)))); } - /** - * Telemetry section is omitted without an active span. - */ - @Test - void shouldNotWriteTelemetrySectionWhenNoSpan() { - CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); - OperationContext operationContext = buildOperationContext(); - - try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { - message.encode(output, operationContext, null); - byte[] buffer = output.toByteArray(); - - assertNull(findTelemetrySectionDocument(buffer)); - } - } - - /** - * Malformed trace context is never sent. - */ - @Test - void shouldNotWriteTelemetrySectionWhenTraceParentNull() { - CommandMessage message = buildCommandMessage(NINE_DOT_ZERO_WIRE_VERSION, EmptyMessageSequences.INSTANCE); - Span span = spanWithTraceParent(null); + @ParameterizedTest(name = "{0}, {1}") + @MethodSource("telemetrySectionOmittedCases") + void shouldNotWriteTelemetrySection(final int maxWireVersion, @Nullable final Span span) { + CommandMessage message = buildCommandMessage(maxWireVersion, EmptyMessageSequences.INSTANCE); OperationContext operationContext = buildOperationContext(); try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { From baaa522f8bde384ef4288107f63ced9b860f1b46 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Thu, 23 Jul 2026 12:35:35 +0100 Subject: [PATCH 40/48] DRIVERS-3454: untrack ServerSpanLinkageProseTest (kept as local-only e2e harness) --- .../ServerSpanLinkageProseTest.java | 144 ------------------ 1 file changed, 144 deletions(-) delete mode 100644 driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java diff --git a/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java b/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java deleted file mode 100644 index 725c249f63b..00000000000 --- a/driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java +++ /dev/null @@ -1,144 +0,0 @@ -/* - * Copyright 2008-present MongoDB, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.mongodb.client.observability; - -import com.mongodb.MongoClientSettings; -import com.mongodb.client.MongoClient; -import com.mongodb.client.MongoClients; -import com.mongodb.client.MongoCollection; -import com.mongodb.observability.ObservabilitySettings; -import io.micrometer.observation.ObservationRegistry; -import io.micrometer.tracing.exporter.FinishedSpan; -import io.micrometer.tracing.test.reporter.inmemory.InMemoryOtelSetup; -import org.bson.Document; -import org.junit.jupiter.api.AfterEach; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.DisplayName; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.condition.EnabledIfSystemProperty; - -import java.io.IOException; -import java.nio.charset.StandardCharsets; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.util.List; -import java.util.stream.Collectors; -import java.util.stream.Stream; - -import static com.mongodb.ClusterFixture.getDefaultDatabaseName; -import static com.mongodb.client.Fixture.getMongoClientSettingsBuilder; -import static org.junit.jupiter.api.Assertions.assertTrue; -import static org.junit.jupiter.api.Assertions.fail; - -/** - * Prose test 5 (docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md): - * end-to-end server span linkage via the server's OTLP file exporter. - * - *

Requires a MongoDB 9.0+ (maxWireVersion >= 29) server on the same host, started with - * {@code --setParameter opentelemetryTraceDirectory=

} (plus tracing feature flag and - * sampling parameters), and the test run with - * {@code -Dorg.mongodb.test.otel.trace.dir=}. Skipped otherwise.

- */ -@EnabledIfSystemProperty(named = "org.mongodb.test.otel.trace.dir", matches = ".+") -public class ServerSpanLinkageProseTest { - - private static final long EXPORT_POLL_TIMEOUT_MS = 30_000; - private static final long EXPORT_POLL_INTERVAL_MS = 1_000; - - private final ObservationRegistry observationRegistry = ObservationRegistry.create(); - private InMemoryOtelSetup memoryOtelSetup; - private InMemoryOtelSetup.Builder.OtelBuildingBlocks inMemoryOtel; - - @BeforeEach - void setUp() { - memoryOtelSetup = InMemoryOtelSetup.builder().register(observationRegistry); - inMemoryOtel = memoryOtelSetup.getBuildingBlocks(); - } - - @AfterEach - void tearDown() { - memoryOtelSetup.close(); - } - - @Test - @DisplayName("Prose test 5: server emits a span parented by the driver command span") - void testServerSpanLinkage() throws Exception { - MongoClientSettings clientSettings = getMongoClientSettingsBuilder() - .observabilitySettings(ObservabilitySettings.micrometerBuilder() - .observationRegistry(observationRegistry) - .build()) - .build(); - - try (MongoClient client = MongoClients.create(clientSettings)) { - MongoCollection collection = - client.getDatabase(getDefaultDatabaseName()).getCollection("serverSpanLinkage"); - collection.find().first(); - } - - List clientSpans = inMemoryOtel.getFinishedSpans(); - assertTrue(clientSpans.size() >= 2, "expected operation + command client spans, got: " + clientSpans); - // Per the command-span propagation design (2026-07-18), the driver injects the traceparent - // of the COMMAND span, passed to CommandMessage.encode(...). The server span is a - // child of the command span, so the server span's parentSpanId equals the command span's - // own span id. Finished-span ordering is not guaranteed, so find the command span by name - // ("find") and use its own span id as the expected parent of the server span. - List commandSpans = clientSpans.stream() - .filter(span -> "find".equals(span.getName())) - .collect(Collectors.toList()); - assertTrue(!commandSpans.isEmpty(), "no client command span named 'find' in: " + clientSpans); - - Path traceDir = Paths.get(System.getProperty("org.mongodb.test.otel.trace.dir")); - long deadline = System.currentTimeMillis() + EXPORT_POLL_TIMEOUT_MS; - while (System.currentTimeMillis() < deadline) { - for (FinishedSpan commandSpan : commandSpans) { - String commandSpanId = commandSpan.getSpanId(); - if (commandSpanId != null - && serverSpanLinked(traceDir, commandSpan.getTraceId(), commandSpanId)) { - return; - } - } - Thread.sleep(EXPORT_POLL_INTERVAL_MS); - } - fail("no server span found in " + traceDir - + " with traceId/parentSpanId matching the command span of any of " + commandSpans); - } - - /** - * Scans OTLP JSON export files for a span with the given traceId whose parentSpanId is the - * given driver span. OTLP file exports contain resourceSpans[].scopeSpans[].spans[] objects - * with hex-encoded traceId/spanId/parentSpanId fields; a simple containment check on the two - * hex ids in the same file line is sufficient and avoids a protobuf/JSON-schema dependency. - */ - private static boolean serverSpanLinked(final Path traceDir, final String traceId, final String parentSpanId) - throws IOException { - if (!Files.isDirectory(traceDir)) { - return false; - } - try (Stream files = Files.walk(traceDir)) { - List exportFiles = files.filter(Files::isRegularFile).collect(Collectors.toList()); - for (Path file : exportFiles) { - for (String line : Files.readAllLines(file, StandardCharsets.UTF_8)) { - if (line.contains(traceId) && line.contains("\"parentSpanId\":\"" + parentSpanId + "\"")) { - return true; - } - } - } - } - return false; - } -} From 019f53d6e52f69feda7cadfcdfd4ab1d232bd925 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Thu, 23 Jul 2026 12:36:53 +0100 Subject: [PATCH 41/48] Cleanup --- .../internal/connection/CommandMessage.java | 15 ++++----------- .../CommandMessageOtelTraceContextTest.java | 6 ++---- .../micrometer/MicrometerTraceParentTest.java | 9 +++------ 3 files changed, 9 insertions(+), 21 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java index eea6d4c94c2..08160ff2bb8 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +++ b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java @@ -87,8 +87,7 @@ public final class CommandMessage extends RequestMessage { private static final byte PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE = 1; /** * Specifies that the `OP_MSG` section payload is a BSON telemetry document - * ({@code {otel: {traceparent: }}}). Mirrors the server's {@code kTelemetry = 3} - * section kind (DRIVERS-3454, 10gen/mongo#56646). + * ({@code {otel: {traceparent: }}}). */ private static final byte PAYLOAD_TYPE_3_TELEMETRY = 3; @@ -115,9 +114,7 @@ public final class CommandMessage extends RequestMessage { /** * The command span to attach to the OP_MSG telemetry section during {@linkplain * #encode(ByteBufferBsonOutput, OperationContext, Span) encoding}. Set for the duration of that overload's - * call and cleared afterward; {@code null} otherwise (including for the plain 2-arg {@code encode}, which - * never attaches a telemetry section). Not thread-safe: a {@code CommandMessage} is built and encoded by a - * single thread per send attempt, so a transient field is safe. + * call and cleared afterward; {@code null} otherwise. */ @Nullable private Span commandSpanForEncoding; @@ -221,8 +218,7 @@ BsonDocument getCommandDocument(final ByteBufferBsonOutput bsonOutput) { } /** - * The command name (first key of the command document — for MongoDB commands the first element - * names the command). Available before {@code encode()} runs, unlike + * The command name (first key of the command document. Available before {@code encode()} runs, unlike * {@link #getCommandDocument(ByteBufferBsonOutput)}; identical to the encoded document's first * key, since encoding only appends fields. */ @@ -232,7 +228,6 @@ public String getCommandName() { /** * The cursor id if this is a {@code getMore} command, or {@code null} otherwise. - * Available before {@code encode()} runs. */ @Nullable public BsonInt64 getGetMoreCursorId() { @@ -284,9 +279,7 @@ protected void encodeMessageBody(final ByteBufferBsonOutput bsonOutput, final Op /** * Encodes the message, attaching the OP_MSG telemetry section with {@code commandSpan}'s trace context when * the gating conditions hold (see {@link #writeTelemetryContextSection}). The 2-arg {@link - * #encode(ByteBufferBsonOutput, OperationContext)} never attaches the section. Not thread-safe: a message - * must be encoded by one thread at a time (already the case — a {@code CommandMessage} is built and encoded - * per send attempt). + * #encode(ByteBufferBsonOutput, OperationContext)} never attaches the section. */ public void encode(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext, @Nullable final Span commandSpan) { diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java index 3a5d8014cba..762876bf253 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java @@ -155,9 +155,7 @@ void shouldWriteTelemetrySectionAfterDocumentSequences() { /** * Same guarantee as {@link #shouldWriteTelemetrySectionAfterDocumentSequences()}, but for the - * {@link DualMessageSequences} branch (the {@code clientBulkWrite} path). It is the only branch that - * assembles its two kind-1 sections via {@link ByteBufferBsonOutput#branch()} (out-of-order buffer - * segments merged on close), so the trailing kind-3 section's placement is worth pinning separately. + * {@link DualMessageSequences} branch (the {@code clientBulkWrite} path). */ @Test void shouldWriteTelemetrySectionAfterDualMessageSequences() { @@ -195,7 +193,7 @@ public EncodeDocumentsResult encodeDocuments(final WritersProviderAndLimitsCheck } /** - * NEW: The 2-arg {@code encode} inherited from {@code RequestMessage} never attaches the telemetry section, + * The 2-arg {@code encode} inherited from {@code RequestMessage} never attaches the telemetry section, * even when the wire version supports it. This guards monitoring/compression/other callers that still use * the 2-arg overload. */ diff --git a/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java index 7c6acb7de44..8cb3a670d1e 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java +++ b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java @@ -35,6 +35,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; class MicrometerTraceParentTest { private static final String VALID_TRACE_ID = "0af7651916cd43dd8448eb211c80319c"; @@ -55,10 +56,6 @@ void shouldFormatUnsampledTraceParentWithZeroFlags() { traceParentFor(VALID_TRACE_ID, VALID_SPAN_ID, null)); } - /** - * Prose test 4: Malformed trace context is never sent - * (see {@code docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md}) - */ @Test void shouldReturnNullForInvalidIds() { assertNull(traceParentFor("00000000000000000000000000000000", VALID_SPAN_ID, true)); @@ -78,7 +75,7 @@ void shouldReturnNullForInvalidIds() { @Test void shouldExposeMicrometerTracingClasspathGuard() { // Sanity check on the current test classpath (micrometer-tracing IS present here). - assertEquals(true, MicrometerTracer.MICROMETER_TRACING_ON_CLASSPATH); + assertTrue(MicrometerTracer.MICROMETER_TRACING_ON_CLASSPATH); } /** @@ -150,7 +147,7 @@ protected Class loadClass(final String name, final boolean resolve) throws Cl () -> traceContextInterface.getMethod("traceParent").invoke(traceContext), "traceParent() must not throw NoClassDefFoundError when micrometer-tracing is absent"); - assertEquals(null, traceParent); + assertNull(traceParent); spanInterface.getMethod("closeScope").invoke(span); spanInterface.getMethod("end").invoke(span); From 2a31215b3fe8ec444ce3ca1d4b70b1ae9dcebbe4 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Thu, 23 Jul 2026 12:38:54 +0100 Subject: [PATCH 42/48] DRIVERS-3454: untrack docs/superpowers (kept locally; already gitignored upstream) --- .../2026-06-03-otel-span-linkage-followup.md | 80 -- ...-08-mongos-version-skew-and-negotiation.md | 102 --- .../2026-07-21-sync-span-leak-window.md | 98 --- .../plans/2026-06-02-otel-e2e-jaeger.md | 507 ----------- .../2026-06-02-otel-opmsg-propagation-poc.md | 815 ------------------ ...3-otel-telemetry-section-reference-impl.md | 695 --------------- .../2026-07-18-command-span-propagation.md | 353 -------- .../2026-07-18-drivers-3454-spec-change.md | 235 ----- docs/superpowers/runbooks/otel-opmsg-e2e.md | 237 ----- ...026-06-01-otel-opmsg-propagation-design.md | 221 ----- .../2026-06-02-otel-e2e-jaeger-design.md | 218 ----- ...7-13-otel-telemetry-section-prose-tests.md | 43 - ...telemetry-section-reference-impl-design.md | 171 ---- ...6-07-18-command-span-propagation-design.md | 85 -- ...6-07-18-drivers-3454-spec-change-design.md | 109 --- 15 files changed, 3969 deletions(-) delete mode 100644 docs/superpowers/followups/2026-06-03-otel-span-linkage-followup.md delete mode 100644 docs/superpowers/followups/2026-06-08-mongos-version-skew-and-negotiation.md delete mode 100644 docs/superpowers/followups/2026-07-21-sync-span-leak-window.md delete mode 100644 docs/superpowers/plans/2026-06-02-otel-e2e-jaeger.md delete mode 100644 docs/superpowers/plans/2026-06-02-otel-opmsg-propagation-poc.md delete mode 100644 docs/superpowers/plans/2026-07-13-otel-telemetry-section-reference-impl.md delete mode 100644 docs/superpowers/plans/2026-07-18-command-span-propagation.md delete mode 100644 docs/superpowers/plans/2026-07-18-drivers-3454-spec-change.md delete mode 100644 docs/superpowers/runbooks/otel-opmsg-e2e.md delete mode 100644 docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md delete mode 100644 docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md delete mode 100644 docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md delete mode 100644 docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md delete mode 100644 docs/superpowers/specs/2026-07-18-command-span-propagation-design.md delete mode 100644 docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md diff --git a/docs/superpowers/followups/2026-06-03-otel-span-linkage-followup.md b/docs/superpowers/followups/2026-06-03-otel-span-linkage-followup.md deleted file mode 100644 index a1fd6d426e6..00000000000 --- a/docs/superpowers/followups/2026-06-03-otel-span-linkage-followup.md +++ /dev/null @@ -1,80 +0,0 @@ -# Follow-up: server span links to the operation span, not the command span - -- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) -- **Status:** Known limitation of the POC — fix after the POC is validated. -- **Date:** 2026-06-03 -- **Related:** `docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md`, - `docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md` - -## Symptom (observed in Jaeger) - -For a single client operation, the exported trace nests as: - -``` -insert test.ping (client OPERATION span, id=4d5f339b) - ├─ insert (client COMMAND span, id=d49879c6, parent=4d5f339b) - └─ insert (mongod SERVER span, id=b89795e6, parent=4d5f339b) -``` - -The `mongod` server span is parented to the client **operation** span, making it a **sibling** of -the client **command** span. The expected/ideal hierarchy is -`operation span → command span → server span` (server span as a child of the command span). Sibling -spans render in Jaeger by start time, which is why the server span can appear "before" the command -span. - -## Root cause - -The OP_MSG trace-context section is written during message **encoding**, which happens **before** the -per-command span is created. At encode time the only span available in the `OperationContext` is the -operation span, so that is the context propagated on the wire. - -Code path (current `nabil_otel_context`): - -1. `InternalStreamConnection.sendAndReceiveInternal` (~line 445): `message.encode(bsonOutput, operationContext)` - runs first. `CommandMessage.writeOtelTraceContextSection` executes inside `encode` and reads - `operationContext.getTracingSpan()`. -2. `operationContext.getTracingSpan()` returns the **operation** span, set earlier by - `TracingManager.createOperationSpan` → `operationContext.setTracingSpan(span)` - (`TracingManager.java` ~line 284). -3. `InternalStreamConnection.sendAndReceiveInternal` (~lines 446–455): the **command** span is created - **after** encode via `createTracingSpan(...)`, parented to the operation span - (`TracingManager.java` ~lines 191–192: `addSpan(MONGODB_COMMAND, …, operationSpan.context())`). - -So the `traceparent` carries the operation span's id; the server attaches its span to the operation -span; the command span is a separate sibling under the same operation span. - -## Why the command span is the better parent - -The command span represents an individual wire RPC, and one operation can issue several commands -(retries, `getMore`, split bulk batches). Parenting each server span under the corresponding command -span ties it to the exact RPC that produced it. Under the operation span, multiple server spans pile -up as indistinguishable siblings. - -## Why it is not a trivial reorder (chicken-and-egg) - -`createTracingSpan` currently derives the command name from `message.getCommandDocument(bsonOutput)`, -i.e. it reads the **already-encoded** bytes. So the command span cannot simply be created before -`encode()` without changing how it obtains the command name. - -## Candidate fixes (after the POC) - -1. **Create the command span before `encode()` using the in-memory command name.** The command name - is available from `CommandMessage`'s in-memory command `BsonDocument` (its first key), without - re-parsing the encoded output. Create the command span first, then have - `writeOtelTraceContextSection` propagate the command span's context. Requires refactoring - `createTracingSpan` so the name/parent no longer depend on `getCommandDocument(bsonOutput)`. - *Preferred — keeps the section inside encoding and yields the correct parent.* - -2. **Append the section after the command span is created.** Move section-writing out of - `CommandMessage.writeOpMsg` into a post-`createTracingSpan` step in `InternalStreamConnection`, - appending the kind-3 section and re-backpatching the OP_MSG message length. More invasive to wire - framing; risks interacting with compression and `getCommandDocument` re-parsing. - -3. **Accept operation-span parenting for the POC.** The trace is still correctly linked end to end — - just one level shallower than ideal. No change. - -## Decision - -Defer. The POC's goal (end-to-end propagation visible in Jaeger) is met with operation-span -parenting. Revisit with option 1 when productionizing, alongside removing the test-only -`OtelTracePropagationTestToggle` and adding the real `hello` `tracingSupport` negotiation. diff --git a/docs/superpowers/followups/2026-06-08-mongos-version-skew-and-negotiation.md b/docs/superpowers/followups/2026-06-08-mongos-version-skew-and-negotiation.md deleted file mode 100644 index e2a4cd9bcbd..00000000000 --- a/docs/superpowers/followups/2026-06-08-mongos-version-skew-and-negotiation.md +++ /dev/null @@ -1,102 +0,0 @@ -# Follow-up: mongos/mongod version skew and why mongos-`hello` negotiation is sufficient - -- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) -- **Date:** 2026-06-08 -- **Context:** Confirms that gating OP_MSG trace-context propagation on the **mongos** `hello` - capability is safe across sharded-cluster version skew. Pairs with the stock-server rejection - finding (`2026-06-08-stock-server-rejection.md`) and the propagation spec. - -## Experiment that motivated this - -Forcing the kind-3 section (`OtelTracePropagationTestToggle.FORCE_PROPAGATION=true`) against a stock -**MongoDB 7.0 sharded cluster** (config RS + shard RS + mongos) made the mongos reject **every** -command with `error 40432 (Location40432): "Unknown section kind 3"` — a clean, recoverable command -error (server stayed healthy), but a 100% app outage for those operations. This is the exact failure -the `hello`-capability gate must prevent, so it matters *which* node the driver negotiates with. - -## Are mongos and mongod the same version? - -Yes. `mongos` and `mongod` ship as one MongoDB Server release (same version stream), upgraded -together. The only standard divergence is **during a rolling sharded-cluster upgrade**, and the skew -direction is fixed by the required order: - -``` -disable balancer → config servers → shards → mongos (LAST) → re-enable balancer → bump FCV -``` - -So during the window the **mongos is the OLDEST** component (upgraded last); shards/config are already -newer. - -### Hard constraints that bound the skew (from official docs + server source) - -- *"The mongos binary cannot connect to mongod instances whose **feature compatibility version (FCV) - is greater than that of the mongos**."* (5.0→8.0 sharded-upgrade docs.) ⇒ mongos is **never behind - the cluster FCV**. -- During upgrade/downgrade the **internal-cluster `hello`** clamps `minWireVersion == maxWireVersion - == LATEST` so an out-of-range internal peer fails to connect - (`src/mongo/db/commands/feature_compatibility_version.cpp`, - `jstests/noPassthrough/.../hello_feature_compatibility_version.js`). This clamp is for - `internalClient` only; external app drivers still receive the full `minWireVersion=0..max` range. -- Internal TSE knowledge lists *"mongos version skew"* as a known upgrade failure mode - (`10gen/tse-strategy-backtest-scoreboard` upgrade-paths SKILL). - -**Net:** mongos may be one major behind the *binaries* transiently, but never behind the *FCV*, and -mongos ≤ shard versions during the upgrade. - -## Why mongos-`hello` negotiation is sufficient - -1. The driver connects to the **mongos** and reads its `hello`. Because mongos is upgraded **last**, - it's the **lowest common denominator** the driver talks to — too-old mongos ⇒ no `tracingSupport` - advertised ⇒ driver doesn't send the section. Conservative and safe. -2. The **mongos→shard** hop is covered: shards are upgraded **before** mongos, so once the mongos - advertises support, the shards already support it (no risk of forwarding kind-3 to a shard that - can't — *assuming the server re-propagates the section onward*). -3. **FCV-gating** closes the remaining gap: if advertisement is FCV-gated, the mongos won't advertise - until FCV is bumped, which only happens after every component (incl. mongos) is upgraded ⇒ uniform - support. - -Conclusion: the driver does **not** need to probe each shard; negotiating once at the mongos is -sufficient given the documented upgrade order + FCV rule. - -## Atlas Proxy / mongonet — the third (and most important) parsing surface - -In Atlas the driver connects to the **Atlas Proxy**, not to mongos/mongod directly, so the proxy is -the surface that matters most. - -- The Atlas Proxy uses **mongonet** for all wire-protocol parse/serialize/forward (Atlas Proxy - Resources wiki). It fully decodes each `OP_MSG`, iterates `msg.Sections`, and re-serializes before - forwarding. -- mongonet models sections as an **enumerated set of typed structs** (`BodySection`, - `DocumentSequenceSection`, `SecurityToken`). The security-token section (**kind 2**) had to be - **explicitly added to mongonet** (CLOUDP-167278 era: *"Add UnsignedSecurityToken section to - mongonet OP_MSG"*). mongonet does **not** generically pass through arbitrary section kinds. -- **Consequence:** an un-updated Atlas Proxy will not understand **kind 3** — it will either reject it - (most likely, mirroring mongod's `40432` → hard outage) or fail to forward it (silently broken - propagation, section dropped before the backend). Either way Atlas needs **explicit mongonet + - Atlas Proxy work** to parse-and-forward the section, even if mongos/mongod already support it. -- **Negotiation still works at the endpoint:** the driver's `hello` in Atlas comes *through the - proxy*, so the proxy is the natural capability gate — it won't advertise `tracingSupport` until - mongonet can handle the section. Same lowest-common-denominator logic as mongos-upgraded-last. - -This is already flagged internally: James Kovacs's DRIVERS-3454 sync notes have an action item to -*"investigate impact on drivers and other software that parses OP_MSG… unknown section type, -especially for existing drivers/services."* Noah Stapp's **"OP_MSG Telemetry Scope"** wants the -section processable *without* server changes and notes the server must support it *"present in -potentially any message"* — plus there is an effort to **consolidate OTel context + retry telemetry + -backpressure into one generic OP_MSG telemetry section**, which may reshape this whole approach. - -## Open confirmations for the server / Atlas team - -1. **`hello` capability** — confirm the plan to advertise trace-context support in `hello` (e.g. - `tracingSupport: true`); this is the production gate replacing the test toggle. -2. **Binary-gated vs FCV-gated** — will `tracingSupport` be advertised based on binary version or on - FCV? (Affects whether the gate is uniform during the upgrade window.) -3. **mongos onward propagation** — does the mongos re-propagate the kind-3 section on the - mongos→shard hop (so shard-side spans link), or is server-side propagation handled separately? -4. **mongonet / Atlas Proxy support** — will mongonet parse *and forward* the new section (and the - proxy re-propagate it to the backend), so Atlas-connected drivers work rather than break? -5. **Proxy `hello` advertisement** — will the Atlas Proxy advertise `tracingSupport` (reflecting proxy - + backend support together), so the driver's capability gate covers the Atlas path? - -Items 2–5 determine whether endpoint-only negotiation (mongos or Atlas Proxy) is fully sufficient end -to end. The consolidation into a single generic telemetry section (above) may also change the design. diff --git a/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md b/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md deleted file mode 100644 index ac728aac014..00000000000 --- a/docs/superpowers/followups/2026-07-21-sync-span-leak-window.md +++ /dev/null @@ -1,98 +0,0 @@ -# Follow-up: sync send-path command-span leak window (pre-existing) - -- **Status: IMPLEMENTED 2026-07-22** in commit `b4c8e8caf3` (single-owner `catch (Throwable)` in the sync path, - `LoggingCommandEventSender` started-event pairing guard [option C], `Span.closeScope()` no-op contract javadoc). - Verified: `LoggingCommandEventSenderSpecification` (incl. new pairing test), `InternalStreamConnection*`, - `CommandMessageOtelTraceContextTest`, micrometer suites (124 tests green), and `MicrometerProseTest` against a - live wire-29 server. A JAVA ticket should still be filed retroactively for release notes; the commit message - carries a `JAVA-XXXX` placeholder to amend once the ticket exists. -- **For:** future JAVA ticket (not yet filed) -- **Found by:** final code review of the DRIVERS-3454 command-span propagation change (2026-07-21); confirmed - **pre-existing** — the window is the same size before and after that change, it was not introduced by it. -- **Component:** `driver-core`, `com.mongodb.internal.connection.InternalStreamConnection`, sync path - `sendAndReceiveInternal` (as of commit `4c5e5dc779`, lines ~463–485). - -## The bug - -In the sync send path, the command tracing span (a started Micrometer `Observation`) is created, and encode -failures are handled (`InternalStreamConnection.java:456-462` ends the span and rethrows). The *next* failure -handler is the catch around `sendCommandMessage` (`:488-497`), which also ends the span. But the statements -**between** those two protected regions run with a live span and no handler: - -1. `message.getCommandDocument(bsonOutput)` hydration (`:469`) — re-parses the encoded message; can throw on a - BSON invariant violation; -2. `LoggingCommandEventSender` construction + `commandEventSender.sendStartedEvent()` (`:472-476`) — command - listeners are user code and can throw anything; -3. `tracingSpan.setQueryText(commandDocument)` (`:481`) — serializes the command to (truncated) JSON; -4. `tracingSpan.openScope()` (`:483-485`). - -If any of these throws, the exception propagates out of `sendAndReceiveInternal` while the command span is never -`end()`ed (and for #4's window, potentially with an opened-but-never-closed scope). - -## Impact - -- The command `Observation` never completes → the span never reaches the exporter (or hangs "in flight" in some - backends), and any registered non-tracing observation handlers (metrics/logging) never see `onStop` — so a - command that failed in this window is invisible or counted as forever-running in observability data. -- With `TracingObservationHandler`, an unclosed scope from a failed `openScope()` sequence could leak the span - into the thread's context, corrupting parentage of subsequent unrelated spans on that pooled thread. -- Most likely real-world trigger: a throwing `CommandListener.commandStarted` (user code) with tracing enabled. - -Severity is low-ish (requires a throw in a narrow window, and the command fails anyway), but the trace/metrics -corruption mode (scope leak on a pooled thread) justifies a fix. - -## Why the async path does not have it - -The async equivalent (`:615-704`) wraps the *entire* block — span creation through send initiation — in a single -outer `catch (Throwable)` that ends the span exactly once; after `sendCommandMessageAsync` is invoked, ending is -owned solely by the tracing callback. The sync path evolved differently (try-with-resources + two narrow -catches) and never got whole-block coverage. - -## Suggested fix - -Restructure the sync path so the span has exactly one owner from creation to hand-off: wrap everything from just -after `createTracingSpan` up to (and including) `sendCommandMessage` in one `catch (RuntimeException | Error e)` -that does `error(e)`, `closeScope()`, `end()`, and `commandEventSender.sendFailedEvent(e)` — replacing the -current two separate catches (mirroring the async path's single-owner structure). This also removes the -duplicated error/end snippets between the encode catch and the send catch. - -The unified catch can call `closeScope()` unconditionally — no "was the scope opened" flag is needed (reviewed -2026-07-22): `MicrometerSpan.closeScope()` null-guards its `scope` field and nulls it after closing, so calling -it before `openScope()` or twice is a no-op, and `Span.EMPTY` no-ops trivially. Care points: - -- **Promote that no-op behavior from implementation detail to interface contract**: add to - `Span.closeScope()`'s javadoc — "Calling this without a prior `openScope()`, or more than once, is a no-op." — - so the unified error path (and any future `Span` implementation or test double) can rely on it, and so a later - refactor doesn't "simplify away" the null-guard. Note `openScope()`'s current javadoc ("Must be paired with - {@link #closeScope()} in a try-finally block") reads stricter than this usage; adjust wording accordingly. -- Do not double-send `sendFailedEvent` (currently only the `sendCommandMessage` catch sends it; the started event - fires at `:476`). Initialize `commandEventSender` to `NoOpCommandEventSender` before the block so failures - prior to sender creation no-op. New edge to decide consciously: if `sendStartedEvent` itself throws, the - logging sender is already assigned and a failed event would be emitted for a command whose started event never - completed — arguably more correct than today's silence, but it is a command-monitoring behavior change. -- Unify the exception types on `catch (Throwable t)` + `throw t;`, matching the async path (decided 2026-07-22). - This compiles without signature changes via Java 7 precise rethrow (the catch parameter is effectively final - and the try body declares no checked exceptions — the same rule the current `catch (Exception) { throw e; }` - already relies on), fixes the current send catch's `Error` blind spot, and forces a compile error if a future - edit introduces a checked-exception throw site. Trade-off (accepted, same as async): fatal errors like OOM run - best-effort cleanup before propagating. -- Keep the encode-failure semantics added by the command-span work: an encode failure ends the span before any - scope was opened — with unconditional `closeScope()` this holds automatically per the no-op contract above. - -## Test idea - -> Note (2026-07-22): the throwing-`CommandListener` variant below is NOT viable — user listener exceptions are -> already swallowed in `ProtocolHelper.sendCommandStartedEvent` (`catch (Exception)` + warn log), so they cannot -> reach this window. The implemented test instead unit-tests the sender's pairing guard directly -> (`LoggingCommandEventSenderSpecification`: terminal events without a completed started event are suppressed). - -Unit test with a mock/failing `CommandListener` whose `commandStarted` throws, tracing enabled with -`InMemoryOtelSetup` (or a recording `ObservationRegistry`): assert the command observation is stopped with an -error (finished-span count includes the errored command span) and that no observation scope remains open on the -thread afterwards. A second case: `setQueryText` throwing (e.g. command payload capture enabled with a document -that fails JSON serialization via a hostile codec). - -## References - -- Final review notes: `.superpowers/sdd/progress.md` (Option A phase) — finding 1. -- Related design: `docs/superpowers/specs/2026-07-18-command-span-propagation-design.md`. diff --git a/docs/superpowers/plans/2026-06-02-otel-e2e-jaeger.md b/docs/superpowers/plans/2026-06-02-otel-e2e-jaeger.md deleted file mode 100644 index eb55898a016..00000000000 --- a/docs/superpowers/plans/2026-06-02-otel-e2e-jaeger.md +++ /dev/null @@ -1,507 +0,0 @@ -# End-to-end OTel Trace Visualization (driver toggle + Spring Boot + Jaeger) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** See a single client→server trace in Jaeger: a Spring Boot app's MongoDB operation, the driver's propagated `traceparent` (forced on via a test toggle), and the POC server's child span, all exported to one Jaeger instance. - -**Architecture:** Add a test-only static toggle in the driver to bypass the server-capability gate; publish the driver to Maven Local; run Jaeger in Docker; reconnect the POC mongod to export OTLP to Jaeger; build a Spring Boot app (Spring Data MongoDB + Micrometer→OTel→Jaeger) that wires the driver's internal tracing via `MongoClientSettingsBuilderCustomizer`, flips the toggle, and exposes a REST trigger. - -**Tech Stack:** Java 8 driver-core (toggle), Gradle (publish), Docker (Jaeger + mongod), Spring Boot 3.3.x / Java 17 / Maven (app), Micrometer Tracing + OpenTelemetry OTLP. - -**Spec:** `docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md` - -> **Standing instruction:** the user wants driver changes STAGED, not committed. For driver tasks, replace any `git commit` with `git add` (stage only). The Spring Boot app lives OUTSIDE the driver repo (`~/MongoDB/otel-poc-server/otel-poc-client/`) and is not added to driver git at all. - -> **Environment facts (already true):** -> - POC mongod binaries extracted at `~/MongoDB/otel-poc-server/dist-test/`; Docker image `otel-poc-mongod` built; container `otel-poc-mongod-run` currently running on the default bridge with port 27017. -> - `featureFlagTracing` is enabled in that build; `opentelemetryHttpEndpoint` is a startup setParameter. -> - Driver OP_MSG kind-3 propagation POC is already staged (gate lives in `CommandMessage.writeOtelTraceContextSection`). - ---- - -## File map - -| File | Responsibility | Change | -|---|---|---| -| `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java` | Test-only force switch | Create | -| `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` | OP_MSG encoding gate | Modify (one clause) | -| `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` | Toggle behavior test | Modify (add 1 test) | -| `~/MongoDB/otel-poc-server/otel-poc-client/pom.xml` | App build, driver override, mavenLocal | Create | -| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/OtelPocApplication.java` | Main + toggle activation | Create | -| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/MongoTracingConfig.java` | `MongoClientSettingsBuilderCustomizer` wiring | Create | -| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/PingController.java` | REST trigger | Create | -| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/resources/application.properties` | URI, sampling, OTLP endpoint | Create | - ---- - -## Task 1: Driver test-only force toggle - -**Files:** -- Create: `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java` -- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` -- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` - -- [ ] **Step 1: Write the failing test** - -Add this method to `CommandMessageOtelTraceContextTest` (it reuses the existing `buildCommandMessage`, `buildOperationContext`, `encodeToBytes`, `containsOtelSection`, and `TRACEPARENT` helpers/constants already in the class). Add the import `import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle;` at the top. - -```java - @Test - void writesSectionWhenForcedEvenIfCapabilityAbsent() { - OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; - try { - CommandMessage message = buildCommandMessage(false); // server did NOT advertise tracingSupport - TraceContext traceContext = () -> TRACEPARENT; - Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); - OperationContext operationContext = buildOperationContext(span); - - byte[] encoded = encodeToBytes(message, operationContext); - - assertTrue(containsOtelSection(encoded, TRACEPARENT), - "With FORCE_PROPAGATION the section must be sent even when the server did not advertise support"); - } finally { - OtelTracePropagationTestToggle.FORCE_PROPAGATION = false; - } - } -``` - -- [ ] **Step 2: Run the test to verify it fails to compile** - -Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest.writesSectionWhenForcedEvenIfCapabilityAbsent" -PskipCryptVerify=true` -Expected: FAIL — `OtelTracePropagationTestToggle` does not exist. - -- [ ] **Step 3: Create the toggle class** - -Create `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java`: - -```java -/* - * Copyright 2008-present MongoDB, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.mongodb.internal.observability.micrometer; - -/** - * TEST-ONLY switch (DRIVERS-3454): when {@code true}, the driver writes the OP_MSG OpenTelemetry - * trace-context section even if the server did not advertise {@code tracingSupport} in its - * {@code hello} response. The sampled-{@code traceparent} requirement still applies. - * - *

This exists only to exercise end-to-end propagation against a server that does not yet advertise - * the capability. Remove before any production use.

- */ -public final class OtelTracePropagationTestToggle { - public static volatile boolean FORCE_PROPAGATION = false; - - private OtelTracePropagationTestToggle() { - } -} -``` - -- [ ] **Step 4: Relax the capability gate in CommandMessage** - -In `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java`, add the import (alphabetically within the `com.mongodb.internal.observability.micrometer` group, next to the existing `Span` import): - -```java -import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; -``` - -In `writeOtelTraceContextSection`, change the first guard from: - -```java - if (!getSettings().isTracingSupported()) { - return; - } -``` - -to: - -```java - if (!getSettings().isTracingSupported() && !OtelTracePropagationTestToggle.FORCE_PROPAGATION) { - return; - } -``` - -- [ ] **Step 5: Run the test to verify it passes** - -Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest" -PskipCryptVerify=true` -Expected: PASS (all cases, including the new one and the existing `omitsSectionWhenCapabilityAbsent` which does NOT set the toggle, so it still passes). - -- [ ] **Step 6: Stage (do NOT commit)** - -```bash -git add driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java \ - driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java \ - driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java -``` - ---- - -## Task 2: Publish the driver to Maven Local - -**Files:** none (build action) - -- [ ] **Step 1: Publish all modules as 5.9.0-SNAPSHOT** - -Run from the repo root: -```bash -./gradlew publishToMavenLocal -PskipCryptVerify=true -``` -Expected: BUILD SUCCESSFUL. - -- [ ] **Step 2: Verify the artifacts landed in the local repo** - -Run: -```bash -ls ~/.m2/repository/org/mongodb/mongodb-driver-sync/5.9.0-SNAPSHOT/ \ - ~/.m2/repository/org/mongodb/mongodb-driver-core/5.9.0-SNAPSHOT/ \ - ~/.m2/repository/org/mongodb/bson/5.9.0-SNAPSHOT/ -``` -Expected: each directory contains a `*-5.9.0-SNAPSHOT.jar`. If `bson-record-codec` is referenced transitively, confirm it too: -```bash -ls ~/.m2/repository/org/mongodb/bson-record-codec/5.9.0-SNAPSHOT/ 2>/dev/null || echo "(bson-record-codec not published — fine unless the app fails to resolve it)" -``` - ---- - -## Task 3: Start Jaeger in Docker - -**Files:** none (infra) - -- [ ] **Step 1: Create the shared network and run Jaeger** - -```bash -docker network create otel-poc-net 2>/dev/null || true -docker rm -f jaeger 2>/dev/null || true -docker run -d --name jaeger --network otel-poc-net \ - -e COLLECTOR_OTLP_ENABLED=true \ - -p 16686:16686 -p 4317:4317 -p 4318:4318 \ - jaegertracing/all-in-one:1.62.0 -``` - -- [ ] **Step 2: Verify the UI is up** - -```bash -until curl -sf http://localhost:16686/ >/dev/null; do sleep 1; done; echo "Jaeger UI reachable" -``` -Expected: `Jaeger UI reachable`. - ---- - -## Task 4: Reconnect mongod to export OTLP to Jaeger - -**Files:** none (infra) - -- [ ] **Step 1: Recreate the mongod container on the shared network with the OTLP endpoint** - -```bash -cd ~/MongoDB/otel-poc-server -docker rm -f otel-poc-mongod-run 2>/dev/null || true -docker run -d --name otel-poc-mongod-run --platform linux/arm64 --network otel-poc-net \ - -v "$PWD/dist-test:/opt/mongo:ro" \ - -v "$PWD/data:/data/db" \ - -p 27017:27017 \ - otel-poc-mongod \ - /opt/mongo/bin/mongod --dbpath /data/db --bind_ip_all --port 27017 \ - --setParameter opentelemetryHttpEndpoint=http://jaeger:4318/v1/traces \ - --setParameter openTelemetryExportIntervalMillis=1000 -``` - -- [ ] **Step 2: Wait for readiness and confirm the endpoint was accepted** - -```bash -until docker logs otel-poc-mongod-run 2>&1 | grep -q "Waiting for connections" || ! docker ps -q --filter name=otel-poc-mongod-run | grep -q .; do sleep 1; done -docker ps --filter name=otel-poc-mongod-run --format '{{.Status}}' -docker logs otel-poc-mongod-run 2>&1 | grep -iE "opentelemetry|otel|trace|export|endpoint" | tail -15 -``` -Expected: container `Up`; no fatal OTLP/endpoint parse error in logs. - -- [ ] **Step 3: Fallback if the endpoint form is rejected** - -If mongod refuses to start or logs an OTLP endpoint error, re-run Step 1 replacing the endpoint with the file exporter (already proven working): -``` - --setParameter opentelemetryTraceDirectory=/data/db/otel-traces -``` -and note in the final report that server spans are inspected as JSONL under `~/MongoDB/otel-poc-server/data/otel-traces/` rather than in the Jaeger UI. Then continue. - ---- - -## Task 5: Scaffold the Spring Boot app (pom + properties + main) - -**Files (all under `~/MongoDB/otel-poc-server/otel-poc-client/`, OUTSIDE the driver repo):** -- Create: `pom.xml` -- Create: `src/main/resources/application.properties` -- Create: `src/main/java/com/example/otelpoc/OtelPocApplication.java` - -- [ ] **Step 1: Create the project directory and Maven wrapper** - -```bash -mkdir -p ~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc -mkdir -p ~/MongoDB/otel-poc-server/otel-poc-client/src/main/resources -``` - -- [ ] **Step 2: Create `pom.xml`** - -```xml - - - 4.0.0 - - - org.springframework.boot - spring-boot-starter-parent - 3.3.5 - - - - com.example - otel-poc-client - 0.0.1-SNAPSHOT - - - 17 - - 5.9.0-SNAPSHOT - - - - - org.springframework.boot - spring-boot-starter-web - - - org.springframework.boot - spring-boot-starter-data-mongodb - - - org.springframework.boot - spring-boot-starter-actuator - - - io.micrometer - micrometer-tracing-bridge-otel - - - io.opentelemetry - opentelemetry-exporter-otlp - - - - - - - org.springframework.boot - spring-boot-maven-plugin - - - - -``` - -> Maven consults the local `~/.m2` repository by default, so the `5.9.0-SNAPSHOT` artifacts published in Task 2 resolve without any extra `` entry. - -- [ ] **Step 3: Create `src/main/resources/application.properties`** - -```properties -spring.application.name=otel-poc-client -server.port=8080 -spring.data.mongodb.uri=mongodb://localhost:27017/test -management.tracing.sampling.probability=1.0 -management.otlp.tracing.endpoint=http://localhost:4318/v1/traces -# Suppress Spring's auto Mongo command instrumentation: the driver's internal tracer is the ONLY -# source of Mongo command spans (no duplicate/competing spans). -management.metrics.enable.mongodb=false -spring.autoconfigure.exclude=org.springframework.boot.actuate.autoconfigure.metrics.mongo.MongoMetricsAutoConfiguration -``` - -- [ ] **Step 4: Create the main class with toggle activation** - -`src/main/java/com/example/otelpoc/OtelPocApplication.java`: - -```java -package com.example.otelpoc; - -import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; -import org.springframework.boot.SpringApplication; -import org.springframework.boot.autoconfigure.SpringBootApplication; - -@SpringBootApplication -public class OtelPocApplication { - public static void main(String[] args) { - // TEST-ONLY: force the driver to send the OP_MSG trace-context section even though this POC - // server does not advertise tracingSupport in hello. Set before any MongoClient is used. - OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; - SpringApplication.run(OtelPocApplication.class, args); - } -} -``` - -- [ ] **Step 5: Generate the Maven wrapper** - -```bash -cd ~/MongoDB/otel-poc-server/otel-poc-client -mvn -N wrapper:wrapper -Dmaven=3.9.9 2>&1 | tail -3 || echo "(if 'mvn' is absent, install it or run with a system Maven in Task 7)" -``` -Expected: `mvnw` and `.mvn/` created. If no system `mvn` exists, skip — Task 7 notes the alternative. - ---- - -## Task 6: Tracing wiring bean + REST trigger - -**Files (under `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/`):** -- Create: `MongoTracingConfig.java` -- Create: `PingController.java` - -- [ ] **Step 1: Create the `MongoClientSettingsBuilderCustomizer` wiring** - -`MongoTracingConfig.java`: - -```java -package com.example.otelpoc; - -import com.mongodb.observability.micrometer.MicrometerObservabilitySettings; -import io.micrometer.observation.ObservationRegistry; -import org.springframework.boot.autoconfigure.mongo.MongoClientSettingsBuilderCustomizer; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.Configuration; - -@Configuration -public class MongoTracingConfig { - - /** - * Activates the driver's INTERNAL tracing (the path that sets operationContext.tracingSpan, which - * traceParent() reads for OP_MSG propagation) and routes its spans through Spring's OTel-bridged - * ObservationRegistry so they export to Jaeger. - */ - @Bean - public MongoClientSettingsBuilderCustomizer tracingCustomizer(final ObservationRegistry registry) { - return builder -> builder.observabilitySettings( - MicrometerObservabilitySettings.builder() - .observationRegistry(registry) - .build()); - } -} -``` - -- [ ] **Step 2: Create the REST trigger** - -`PingController.java`: - -```java -package com.example.otelpoc; - -import org.bson.Document; -import org.springframework.data.mongodb.core.MongoTemplate; -import org.springframework.web.bind.annotation.GetMapping; -import org.springframework.web.bind.annotation.RestController; - -import java.util.Date; - -@RestController -public class PingController { - - private final MongoTemplate mongoTemplate; - - public PingController(final MongoTemplate mongoTemplate) { - this.mongoTemplate = mongoTemplate; - } - - @GetMapping("/ping") - public String ping() { - mongoTemplate.getCollection("ping").insertOne(new Document("at", new Date())); - long count = mongoTemplate.getCollection("ping").countDocuments(); - return "ok, count=" + count + "\n"; - } -} -``` - -- [ ] **Step 3: Compile the app to verify wiring resolves** - -```bash -cd ~/MongoDB/otel-poc-server/otel-poc-client -./mvnw -q -DskipTests compile 2>&1 | tail -20 || mvn -q -DskipTests compile 2>&1 | tail -20 -``` -Expected: BUILD SUCCESS. If compilation fails to resolve `MicrometerObservabilitySettings` or `OtelTracePropagationTestToggle`, confirm Task 2 published `mongodb-driver-core` (the toggle and settings live there) and that `mongodb.version` is `5.9.0-SNAPSHOT`. - ---- - -## Task 7: End-to-end run and Jaeger verification - -**Files:** none (run + verify). Prereqs: Tasks 1–6 done; Jaeger (Task 3) and mongod-on-network (Task 4) running. - -- [ ] **Step 1: Start the Spring Boot app** - -```bash -cd ~/MongoDB/otel-poc-server/otel-poc-client -./mvnw spring-boot:run 2>&1 | tee /tmp/otel-poc-client.log & -# wait until it is listening on 8080 -until curl -sf http://localhost:8080/ping >/dev/null 2>&1; do sleep 2; done -echo "app up" -``` -(If there is no `mvnw`, use `mvn spring-boot:run`.) Expected: `app up`. - -- [ ] **Step 2: Generate a traced operation** - -```bash -curl -s http://localhost:8080/ping -``` -Expected: `ok, count=`. - -- [ ] **Step 3: Confirm the client exported a trace to Jaeger** - -```bash -sleep 3 -curl -s "http://localhost:16686/api/services" | tr ',' '\n' | grep -i "otel-poc-client" && echo "client service present in Jaeger" -``` -Expected: `otel-poc-client` appears in the services list. - -- [ ] **Step 4: Confirm a linked server span exists in the same trace** - -```bash -TRACE_JSON=$(curl -s "http://localhost:16686/api/traces?service=otel-poc-client&limit=1") -echo "$TRACE_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); \ -spans=d['data'][0]['spans']; procs=d['data'][0]['processes']; \ -print('services in trace:', sorted({procs[s['processID']]['serviceName'] for s in spans})); \ -print('span names:', [s['operationName'] for s in spans])" -``` -Expected: the services set includes BOTH `otel-poc-client` and the mongod service (e.g. `mongod`), proving the client span and the server span share one trace. - -- [ ] **Step 5: Visual confirmation** - -Open **http://localhost:16686**, select service `otel-poc-client`, open the most recent trace. Confirm a span tree: HTTP `GET /ping` → driver mongo command span → `mongod` server span (same traceId; server span's parent is the client mongo span). - -- [ ] **Step 6: Negative check (capability gate still works)** - -Stop the app, set the toggle off by editing `OtelPocApplication.main` to `FORCE_PROPAGATION = false` (or comment the line), `./mvnw spring-boot:run` again, `curl /ping`, and confirm in Jaeger the new client trace has **no** `mongod` span (server starts its own unrelated trace). Restore the line afterward. - -- [ ] **Step 7: Record the result** - -If server spans appear in Jaeger: success — note it. If Task 4 used the file-exporter fallback, instead show the server span and matching traceId from the latest `~/MongoDB/otel-poc-server/data/otel-traces/*.jsonl` (grep for the client's traceId) and note that server spans are file-based rather than in the Jaeger UI. - ---- - -## Self-Review - -**Spec coverage:** -- §3 toggle → Task 1 (class + gate + test). §4 publish → Task 2. §5 Jaeger → Task 3. §6 mongod OTLP (+ file fallback) → Task 4 (Steps 1–3). §7 app: deps/version override → Task 5 Step 2; properties → Task 5 Step 3; toggle activation → Task 5 Step 4; `MongoClientSettingsBuilderCustomizer` wiring → Task 6 Step 1; REST trigger → Task 6 Step 2. §8 run & verification → Task 7. §10 risks: OTLP endpoint form → Task 4 Step 3 fallback; duplicate Spring command spans → acceptable, see note below; driver-version override → Task 5 Step 2 + Task 6 Step 3 troubleshooting. - -**Duplicate-span suppression (spec §7, firm):** Spring's auto Mongo command instrumentation is suppressed in `application.properties` (Task 5 Step 3) via `spring.autoconfigure.exclude=…MongoMetricsAutoConfiguration` + `management.metrics.enable.mongodb=false`, so the driver's internal tracer is the only source of Mongo command spans. If startup logs show the excluded class still contributing (version drift), the implementer confirms the exact actuator Mongo autoconfig class for Spring Boot 3.3.5 and excludes that instead. - -**Placeholder scan:** no TBD/TODO; every code/file step has full content; the only conditional is the documented OTLP-endpoint fallback (Task 4 Step 3) and the no-`mvnw` alternative (system `mvn`). - -**Type/name consistency:** `OtelTracePropagationTestToggle.FORCE_PROPAGATION` (Tasks 1, 5); `MicrometerObservabilitySettings.builder().observationRegistry(...)` and `.observabilitySettings(...)` (Task 6, matches driver API verified in spec §7); package `com.example.otelpoc` and class names match the file map; `mongodb.version=5.9.0-SNAPSHOT` consistent (Tasks 2, 5, 6). - -**Staging vs commit:** Task 1 stages only (driver repo); Tasks 5–6 create files outside the driver repo (never added to driver git). diff --git a/docs/superpowers/plans/2026-06-02-otel-opmsg-propagation-poc.md b/docs/superpowers/plans/2026-06-02-otel-opmsg-propagation-poc.md deleted file mode 100644 index 4c443b34720..00000000000 --- a/docs/superpowers/plans/2026-06-02-otel-opmsg-propagation-poc.md +++ /dev/null @@ -1,815 +0,0 @@ -# OTel Trace-Context Propagation over OP_MSG (driver-sync POC) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make the synchronous MongoDB Java driver inject the active client span's W3C `traceparent` into outgoing OP_MSG messages as a new section (kind 3), but only when the server advertised support — validating the wire contract from the DRIVERS-3454 spec. - -**Architecture:** The driver already builds OP_MSG sections in `CommandMessage.writeOpMsg()` and creates per-operation Micrometer spans accessible via `OperationContext.getTracingSpan()`. We (1) expose the W3C `traceparent` from the span's `TraceContext`, (2) read a `tracingSupport` capability from the `hello` handshake into `ConnectionDescription` → `MessageSettings`, and (3) write section kind 3 in `writeOpMsg()` gated on both the capability and a non-null sampled traceparent. Validation is phased: automated OP_MSG encode/parse round-trip tests (Phase 1), then an optional manual end-to-end runbook against the server POC (Phase 2). - -**Tech Stack:** Java 8 (driver-core baseline), Micrometer Observation (runtime) + Micrometer Tracing (extraction), JUnit 5 + Mockito, Gradle. - -**Spec:** `docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md` - -> **POC scope guards (do NOT exceed):** sync path only; no reactive/async; no `mongos`/load-balanced propagation; `tracestate` is pass-through only; no sampling rework. All code is internal — no breaking public-API change (only additive withers/getters). - ---- - -## File map - -| File | Responsibility | Change | -|---|---|---| -| `driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java` | Trace context abstraction | Add `@Nullable String traceParent()` | -| `driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java` | Micrometer impl | Implement `traceParent()` from the `Observation`'s tracing context | -| `driver-core/build.gradle.kts` | Module deps | Add `micrometer-tracing` as `optionalImplementation` | -| `gradle/libs.versions.toml` | Dependency catalog | Add `micrometer-tracing` library coordinate (main) | -| `driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java` | hello parsing | Parse `tracingSupport`, set on `ConnectionDescription` | -| `driver-core/src/main/com/mongodb/connection/ConnectionDescription.java` | Connection capabilities | Add `tracingSupport` field, `withTracingSupport()`, `isTracingSupport()` | -| `driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java` | Per-message settings | Add `tracingSupported` field + builder method + getter | -| `driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java` | Builds `MessageSettings` | Propagate `tracingSupport` from `ConnectionDescription` | -| `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` | OP_MSG encoding | Add section kind 3, write it gated on settings + traceparent | -| `docs/superpowers/runbooks/otel-opmsg-e2e.md` | Phase 2 manual validation | New runbook | - ---- - -## Task 1: Expose `traceParent()` on the `TraceContext` abstraction - -**Files:** -- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java` - -- [ ] **Step 1: Add the method to the interface and make `EMPTY` return null** - -Replace the interface body so `EMPTY` implements the new method: - -```java -@SuppressWarnings("InterfaceIsType") -public interface TraceContext { - TraceContext EMPTY = new TraceContext() { - @Override - public String traceParent() { - return null; - } - }; - - /** - * The W3C {@code traceparent} string for this context - * ({@code 00-<32hex traceId>-<16hex spanId>-<2hex flags>}), - * or {@code null} if unavailable or the span is not sampled. - */ - @Nullable - String traceParent(); -} -``` - -Add the import `import com.mongodb.lang.Nullable;` below the package statement. - -- [ ] **Step 2: Compile to verify the interface change is consistent** - -Run: `./gradlew :driver-core:compileJava` -Expected: SUCCESS. (`Span.EMPTY.context()` returns `TraceContext.EMPTY`, which now implements `traceParent()`.) If any other anonymous `TraceContext` implementers exist they will fail to compile — fix each by adding `traceParent()` returning `null`. - -- [ ] **Step 3: Commit** - -```bash -git add driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java -git commit -m "DRIVERS-3454: add traceParent() to TraceContext" -``` - ---- - -## Task 2: Implement `traceParent()` in the Micrometer tracer - -The driver only has the Micrometer **Observation** at runtime; the W3C trace/span IDs live in Micrometer **Tracing**, which attaches a `TracingObservationHandler.TracingContext` to the observation's context when a tracing bridge is configured. We read that. - -**Files:** -- Modify: `gradle/libs.versions.toml` -- Modify: `driver-core/build.gradle.kts` -- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java` -- Test: `driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java` - -- [ ] **Step 1: Add the main `micrometer-tracing` library coordinate** - -In `gradle/libs.versions.toml`, under `[libraries]` add (the `micrometer-tracing` version `1.6.0-M3` already exists under `[versions]`): - -```toml -micrometer-tracing = { module = "io.micrometer:micrometer-tracing", version.ref = "micrometer-tracing" } -``` - -- [ ] **Step 2: Add it as an optional dependency of driver-core** - -In `driver-core/build.gradle.kts`, directly after line 59 (`optionalImplementation(libs.micrometer.observation)`), add: - -```kotlin -optionalImplementation(libs.micrometer.tracing) -``` - -(The existing `"io.micrometer.*;resolution:=optional"` OSGi import on line 105 already covers the new package.) - -- [ ] **Step 3: Write the failing test** - -Create `driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java`: - -```java -/* - * Copyright 2008-present MongoDB, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.mongodb.internal.observability.micrometer; - -import io.micrometer.observation.ObservationRegistry; -import io.micrometer.tracing.test.simple.SimpleTracer; -import io.micrometer.tracing.handler.DefaultTracingObservationHandler; -import com.mongodb.observability.micrometer.MongodbObservation; -import org.junit.jupiter.api.Test; - -import java.util.regex.Pattern; - -import static org.junit.jupiter.api.Assertions.assertNotNull; -import static org.junit.jupiter.api.Assertions.assertNull; -import static org.junit.jupiter.api.Assertions.assertTrue; - -class MicrometerTraceParentTest { - private static final Pattern TRACEPARENT = - Pattern.compile("00-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}"); - - @Test - void returnsTraceParentForSampledSpan() { - ObservationRegistry registry = ObservationRegistry.create(); - SimpleTracer tracer = new SimpleTracer(); - registry.observationConfig().observationHandler(new DefaultTracingObservationHandler(tracer)); - - MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); - Span span = micrometerTracer.nextSpan(MongodbObservation.COMMAND_OBSERVATION, "find", null, null); - span.openScope(); - try { - String traceParent = span.context().traceParent(); - assertNotNull(traceParent); - assertTrue(TRACEPARENT.matcher(traceParent).matches(), traceParent); - } finally { - span.closeScope(); - span.end(); - } - } - - @Test - void returnsNullWhenNoTracingBridgeConfigured() { - ObservationRegistry registry = ObservationRegistry.create(); - MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); - Span span = micrometerTracer.nextSpan(MongodbObservation.COMMAND_OBSERVATION, "find", null, null); - span.openScope(); - try { - assertNull(span.context().traceParent()); - } finally { - span.closeScope(); - span.end(); - } - } -} -``` - -> Note: confirm the enum constant name in `MongodbObservation` (e.g. `COMMAND_OBSERVATION`); if it differs, use the actual command-span constant. `SimpleTracer`/`DefaultTracingObservationHandler` come from the test-scoped `micrometer-tracing-integration-test` dependency already present. - -- [ ] **Step 4: Run the test to verify it fails to compile/fails** - -Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.observability.micrometer.MicrometerTraceParentTest"` -Expected: FAIL — `MicrometerTraceContext` does not yet implement `traceParent()`. - -- [ ] **Step 5: Implement `traceParent()` in `MicrometerTraceContext`** - -In `MicrometerTracer.java`, add imports: - -```java -import io.micrometer.tracing.TraceContext; -import io.micrometer.tracing.handler.TracingObservationHandler; -``` - -> The driver's own type is also named `TraceContext`; reference the Micrometer one by its fully-qualified name in code to avoid the clash (do NOT add the conflicting import). Use `io.micrometer.tracing.TraceContext` inline. - -Replace the `MicrometerTraceContext` inner class with: - -```java - /** - * Represents a Micrometer-based trace context. - */ - private static class MicrometerTraceContext implements TraceContext { - @Nullable - private final Observation observation; - - MicrometerTraceContext(@Nullable final Observation observation) { - this.observation = observation; - } - - @Override - @Nullable - public String traceParent() { - if (observation == null) { - return null; - } - TracingObservationHandler.TracingContext tracingContext = - observation.getContextView().getOrNull(TracingObservationHandler.TracingContext.class); - if (tracingContext == null || tracingContext.getSpan() == null) { - return null; - } - io.micrometer.tracing.TraceContext ctx = tracingContext.getSpan().context(); - if (ctx == null || ctx.traceId() == null || ctx.spanId() == null) { - return null; - } - Boolean sampled = ctx.sampled(); - if (sampled == null || !sampled) { - return null; - } - return "00-" + ctx.traceId() + "-" + ctx.spanId() + "-01"; - } - } -``` - -> `traceId()`/`spanId()` from Micrometer Tracing are already lowercase hex of the correct length. We emit flags `01` (sampled) because we only propagate sampled spans, matching spec §3.3. - -Note: the `import io.micrometer.tracing.TraceContext;` added above is unused if you reference it fully-qualified — remove it and keep only the `TracingObservationHandler` import to satisfy the no-unused-import check. - -- [ ] **Step 6: Run the test to verify it passes** - -Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.observability.micrometer.MicrometerTraceParentTest"` -Expected: PASS (both tests). - -- [ ] **Step 7: Commit** - -```bash -git add gradle/libs.versions.toml driver-core/build.gradle.kts \ - driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java \ - driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java -git commit -m "DRIVERS-3454: extract W3C traceparent from Micrometer span" -``` - ---- - -## Task 3: Read `tracingSupport` from `hello` into `ConnectionDescription` - -**Files:** -- Modify: `driver-core/src/main/com/mongodb/connection/ConnectionDescription.java` -- Modify: `driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java` -- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java` - -> `ConnectionDescription` is public API. The change is **additive only** (new wither + getter; existing constructors keep delegating with `tracingSupport=false`), so it is binary-compatible. Do NOT change any existing public constructor signature. - -- [ ] **Step 1: Read the actual `ConnectionDescription` constructor chain and the `withServerType` wither** - -Run: `sed -n '40,210p' driver-core/src/main/com/mongodb/connection/ConnectionDescription.java` -Note the most-derived constructor (the one with `serviceId` + all fields) and how `withConnectionId`/`withServiceId`/`withServerType` build a copy. You will mirror that pattern exactly. - -- [ ] **Step 2: Write the failing test** - -Create `driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java`: - -```java -/* - * Copyright 2008-present MongoDB, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.mongodb.internal.connection; - -import com.mongodb.ServerAddress; -import com.mongodb.connection.ClusterConnectionMode; -import com.mongodb.connection.ConnectionDescription; -import com.mongodb.connection.ConnectionId; -import com.mongodb.connection.ServerId; -import org.bson.BsonDocument; -import org.junit.jupiter.api.Test; - -import static org.junit.jupiter.api.Assertions.assertFalse; -import static org.junit.jupiter.api.Assertions.assertTrue; - -class DescriptionHelperTracingTest { - private static final ConnectionId CONNECTION_ID = - new ConnectionId(new ServerId(new com.mongodb.connection.ClusterId(), new ServerAddress())); - - private static BsonDocument hello(final boolean tracing) { - BsonDocument doc = BsonDocument.parse( - "{ ok: 1, ismaster: true, maxWireVersion: 25, minWireVersion: 0," - + " maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, maxWriteBatchSize: 100000 }"); - if (tracing) { - doc.put("tracingSupport", org.bson.BsonBoolean.TRUE); - } - return doc; - } - - @Test - void parsesTracingSupportTrue() { - ConnectionDescription description = DescriptionHelper.createConnectionDescription( - ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(true)); - assertTrue(description.isTracingSupport()); - } - - @Test - void defaultsTracingSupportFalseWhenAbsent() { - ConnectionDescription description = DescriptionHelper.createConnectionDescription( - ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(false)); - assertFalse(description.isTracingSupport()); - } -} -``` - -> Confirm `createConnectionDescription`'s exact signature/visibility (Explore reported `static ConnectionDescription createConnectionDescription(ClusterConnectionMode, ConnectionId, BsonDocument)`). If the package-private method isn't visible, the test is already in the same `com.mongodb.internal.connection` package, so it is. - -- [ ] **Step 3: Run the test to verify it fails** - -Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.DescriptionHelperTracingTest"` -Expected: FAIL — `isTracingSupport()` does not exist. - -- [ ] **Step 4: Add the field, wither, and getter to `ConnectionDescription`** - -Add the field next to the other `private final` fields (after `logicalSessionTimeoutMinutes`): - -```java - private final boolean tracingSupport; -``` - -In the most-derived constructor (the one with `serviceId` and all fields), add `, false` initialization by assigning `this.tracingSupport = false;` at the end of its body. For all other delegating constructors no change is needed (they call the most-derived one). - -Add a wither (mirror `withServerType`) — it constructs a copy via the most-derived constructor and then sets the flag. Since the constructor sets `tracingSupport=false`, implement the wither by copying through a private all-args path. Concretely, add: - -```java - /** - * Returns a copy of this {@code ConnectionDescription} with the given tracing-support capability. - * - * @param tracingSupport whether the server advertised OpenTelemetry trace-context support - * @return the new connection description - */ - public ConnectionDescription withTracingSupport(final boolean tracingSupport) { - ConnectionDescription copy = new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, - maxBatchCount, maxDocumentSize, maxMessageSize, compressors, saslSupportedMechanisms, - logicalSessionTimeoutMinutes); - copy.tracingSupportOverride = tracingSupport; - return copy; - } - - /** - * @return whether the server advertised OpenTelemetry trace-context support in its hello response - */ - public boolean isTracingSupport() { - return tracingSupportOverride != null ? tracingSupportOverride : tracingSupport; - } -``` - -Because `tracingSupport` is `final`, add a separate non-final override field to keep the change additive without rewriting every constructor: - -```java - @Nullable - private Boolean tracingSupportOverride; -``` - -(`@Nullable` import `com.mongodb.lang.Nullable` already present in this file; verify and add if missing.) Remove the now-unnecessary `this.tracingSupport = false;` line and instead initialize the field inline at declaration: `private final boolean tracingSupport = false;` is illegal for a constructor-set final — so declare it `private final boolean tracingSupport;` and assign `this.tracingSupport = false;` in the most-derived constructor only. - -> Rationale: this keeps all five public constructors source- and binary-compatible. The override field carries the capability set post-construction by `withTracingSupport`. - -- [ ] **Step 5: Set the capability in `DescriptionHelper.createConnectionDescription`** - -Add a parser near the other private getters: - -```java - private static boolean getTracingSupport(final BsonDocument helloResult) { - return helloResult.getBoolean("tracingSupport", org.bson.BsonBoolean.FALSE).getValue(); - } -``` - -In `createConnectionDescription`, change the returned value to apply the wither. Find the `return connectionDescription;` (or the `new ConnectionDescription(...)` return) and wrap it: - -```java - return connectionDescription.withTracingSupport(getTracingSupport(helloResult)); -``` - -(If the method returns the `new ConnectionDescription(...)` directly, assign it to a local `ConnectionDescription connectionDescription = new ConnectionDescription(...);` first, then return the wither call.) - -- [ ] **Step 6: Run the test to verify it passes** - -Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.DescriptionHelperTracingTest"` -Expected: PASS (both tests). - -- [ ] **Step 7: Commit** - -```bash -git add driver-core/src/main/com/mongodb/connection/ConnectionDescription.java \ - driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java \ - driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java -git commit -m "DRIVERS-3454: parse hello tracingSupport into ConnectionDescription" -``` - ---- - -## Task 4: Carry `tracingSupported` through `MessageSettings` - -**Files:** -- Modify: `driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java` -- Modify: `driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java` - -- [ ] **Step 1: Add the field + builder method + getter to `MessageSettings`** - -Add next to `sessionSupported` (field after line 60): - -```java - private final boolean tracingSupported; -``` - -In the `Builder` (after `sessionSupported` field, line 82): - -```java - private boolean tracingSupported; -``` - -Add the builder setter (mirror `sessionSupported(...)`, after line 137): - -```java - public Builder tracingSupported(final boolean tracingSupported) { - this.tracingSupported = tracingSupported; - return this; - } -``` - -In the private `MessageSettings(final Builder builder)` constructor (line ~197), add: - -```java - this.tracingSupported = builder.tracingSupported; -``` - -Add the getter (mirror `getMaxWireVersion()`, near line 181): - -```java - public boolean isTracingSupported() { - return tracingSupported; - } -``` - -- [ ] **Step 2: Populate it in `ProtocolHelper`** - -In `ProtocolHelper.java` at the `MessageSettings.builder()` chain (line 231), directly after the existing `.sessionSupported(connectionDescription.getLogicalSessionTimeoutMinutes() != null)` line (line 237), add: - -```java - .tracingSupported(connectionDescription.isTracingSupport()) -``` - -- [ ] **Step 3: Compile** - -Run: `./gradlew :driver-core:compileJava` -Expected: SUCCESS. - -- [ ] **Step 4: Commit** - -```bash -git add driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java \ - driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java -git commit -m "DRIVERS-3454: thread tracingSupported into MessageSettings" -``` - ---- - -## Task 5: Write OP_MSG section kind 3 in `CommandMessage.writeOpMsg()` - -The section format matches the server POC's `kSecurityToken`/`kOtelTelemetryContext`: a single section-kind byte followed by a CString (no 4-byte length prefix). The server reads it with `readCStr()`. - -**Files:** -- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` - -- [ ] **Step 1: Add the section-kind constant** - -After `PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE` (line 82) add: - -```java - /** - * Specifies that the `OP_MSG` section payload is a W3C traceparent C-string (OpenTelemetry trace context). - */ - private static final byte PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT = 3; -``` - -- [ ] **Step 2: Add the import for the tracing Span** - -In the import block add: - -```java -import com.mongodb.internal.observability.micrometer.Span; -``` - -- [ ] **Step 3: Write the section in `writeOpMsg`, just before the flag bits are backpatched** - -In `writeOpMsg(...)`, immediately **before** the comment `// Write the flag bits` (line 280), insert: - -```java - writeOtelTraceContextSection(bsonOutput, operationContext); - -``` - -Then add the private method (place it after `writeOpMsg`, before `writeOpQuery`): - -```java - private void writeOtelTraceContextSection(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext) { - if (!getSettings().isTracingSupported()) { - return; - } - Span tracingSpan = operationContext.getTracingSpan(); - if (tracingSpan == null) { - return; - } - String traceParent = tracingSpan.context().traceParent(); - if (traceParent == null) { - return; - } - bsonOutput.writeByte(PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT); - bsonOutput.writeCString(traceParent); - } -``` - -> Rationale: at encode time only the operation-level span exists in `OperationContext` (the command span is created later, in `InternalStreamConnection`). The operation span is a valid parent for the server's RPC span. `context().traceParent()` returns `null` for no-op/unsampled spans, so a missing or unsampled span naturally omits the section (spec §3.3). Gating on `isTracingSupported()` ensures we never send the section to a server that would `uassert(40432)` (spec §4). - -- [ ] **Step 4: Compile** - -Run: `./gradlew :driver-core:compileJava` -Expected: SUCCESS. - -- [ ] **Step 5: Commit** - -```bash -git add driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java -git commit -m "DRIVERS-3454: write OP_MSG otel trace-context section (kind 3)" -``` - ---- - -## Task 6 (Phase 1 validation): OP_MSG round-trip encode test - -This is the automated proof of the wire contract: encode a command with a tracing-capable connection + a sampled span, then scan the produced bytes for section kind 3 + the exact traceparent C-string; assert it is absent when the capability is off or the span yields no traceparent. - -**Files:** -- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` - -- [ ] **Step 1: Write the failing test** - -Create `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java`: - -```java -/* - * Copyright 2008-present MongoDB, Inc. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -package com.mongodb.internal.connection; - -import com.mongodb.MongoNamespace; -import com.mongodb.ReadConcern; -import com.mongodb.ReadPreference; -import com.mongodb.ServerApi; -import com.mongodb.connection.ClusterConnectionMode; -import com.mongodb.connection.ServerType; -import com.mongodb.internal.TimeoutContext; -import com.mongodb.internal.connection.MessageSequences.EmptyMessageSequences; -import com.mongodb.internal.observability.micrometer.Span; -import com.mongodb.internal.observability.micrometer.TraceContext; -import com.mongodb.internal.session.SessionContext; -import com.mongodb.internal.validator.NoOpFieldNameValidator; -import org.bson.BsonDocument; -import org.bson.BsonString; -import org.bson.ByteBuf; -import org.junit.jupiter.api.Test; - -import java.nio.charset.StandardCharsets; -import java.util.List; - -import static com.mongodb.internal.mockito.MongoMockito.mock; -import static com.mongodb.internal.operation.ServerVersionHelper.LATEST_WIRE_VERSION; -import static org.junit.jupiter.api.Assertions.assertFalse; -import static org.junit.jupiter.api.Assertions.assertTrue; -import static org.mockito.Mockito.when; - -class CommandMessageOtelTraceContextTest { - private static final MongoNamespace NAMESPACE = new MongoNamespace("db.test"); - private static final BsonDocument COMMAND = new BsonDocument("find", new BsonString(NAMESPACE.getCollectionName())); - private static final String TRACE_PARENT = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"; - - private static CommandMessage commandMessage(final boolean tracingSupported) { - return new CommandMessage(NAMESPACE.getDatabaseName(), COMMAND, NoOpFieldNameValidator.INSTANCE, - ReadPreference.primary(), - MessageSettings.builder() - .maxWireVersion(LATEST_WIRE_VERSION) - .serverType(ServerType.REPLICA_SET_PRIMARY) - .sessionSupported(true) - .tracingSupported(tracingSupported) - .build(), - true, EmptyMessageSequences.INSTANCE, ClusterConnectionMode.MULTIPLE, (ServerApi) null); - } - - private static OperationContext operationContextWithSpan(final String traceParentOrNull) { - SessionContext sessionContext = mock(SessionContext.class, mock -> { - when(mock.getClusterTime()).thenReturn(null); - when(mock.hasSession()).thenReturn(false); - when(mock.getReadConcern()).thenReturn(ReadConcern.DEFAULT); - when(mock.notifyMessageSent()).thenReturn(true); - when(mock.hasActiveTransaction()).thenReturn(false); - when(mock.isSnapshot()).thenReturn(false); - }); - TimeoutContext timeoutContext = mock(TimeoutContext.class); - TraceContext traceContext = () -> traceParentOrNull; - Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); - return mock(OperationContext.class, mock -> { - when(mock.getSessionContext()).thenReturn(sessionContext); - when(mock.getTimeoutContext()).thenReturn(timeoutContext); - when(mock.getTracingSpan()).thenReturn(span); - }); - } - - private static byte[] encodeToBytes(final CommandMessage message, final OperationContext operationContext) { - try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { - message.encode(output, operationContext); - List buffers = output.getByteBuffers(); - byte[] bytes = new byte[output.getSize()]; - int pos = 0; - for (ByteBuf buf : buffers) { - int remaining = buf.remaining(); - buf.get(bytes, pos, remaining); - pos += remaining; - } - buffers.forEach(ByteBuf::release); - return bytes; - } - } - - private static boolean containsTraceParentSection(final byte[] message) { - // Section kind 3 byte immediately followed by the null-terminated traceparent. - byte[] needle = new byte[1 + TRACE_PARENT.length() + 1]; - needle[0] = 3; - byte[] tp = TRACE_PARENT.getBytes(StandardCharsets.UTF_8); - System.arraycopy(tp, 0, needle, 1, tp.length); - needle[needle.length - 1] = 0; - outer: - for (int i = 0; i + needle.length <= message.length; i++) { - for (int j = 0; j < needle.length; j++) { - if (message[i + j] != needle[j]) { - continue outer; - } - } - return true; - } - return false; - } - - @Test - void writesSectionWhenSupportedAndSampledSpanPresent() { - byte[] bytes = encodeToBytes(commandMessage(true), operationContextWithSpan(TRACE_PARENT)); - assertTrue(containsTraceParentSection(bytes), "expected OP_MSG section kind 3 with traceparent"); - } - - @Test - void omitsSectionWhenCapabilityAbsent() { - byte[] bytes = encodeToBytes(commandMessage(false), operationContextWithSpan(TRACE_PARENT)); - assertFalse(containsTraceParentSection(bytes), "must not send section to non-tracing server"); - } - - @Test - void omitsSectionWhenSpanHasNoTraceParent() { - byte[] bytes = encodeToBytes(commandMessage(true), operationContextWithSpan(null)); - assertFalse(containsTraceParentSection(bytes), "must omit section when span yields no traceparent"); - } -} -``` - -> If `ByteBufferBsonOutput` exposes a different accessor than `getByteBuffers()`/`getSize()`, mirror whatever the existing `CommandMessageTest`/`CommandMessageSpecification` uses to read encoded bytes (e.g. `getByteBuffers()` then `ByteBufNIO`); adjust `encodeToBytes` accordingly. The `TraceContext traceContext = () -> traceParentOrNull;` lambda works because `TraceContext` is now a single-abstract-method interface returning the traceparent. - -- [ ] **Step 2: Run to verify it fails** - -Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest"` -Expected: FAIL before Task 5 is implemented; after Task 5 it should pass. (If running tasks in order, this confirms PASS.) - -- [ ] **Step 3: Run to verify it passes** - -Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest"` -Expected: PASS (all three). - -- [ ] **Step 4: Commit** - -```bash -git add driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java -git commit -m "DRIVERS-3454: round-trip test for OP_MSG otel trace-context section" -``` - ---- - -## Task 7 (Phase 2, optional/manual): end-to-end runbook against the server POC - -Not automated / not CI. Documents how to confirm the server starts a linked child span. - -**Files:** -- Create: `docs/superpowers/runbooks/otel-opmsg-e2e.md` - -- [ ] **Step 1: Write the runbook** - -Create `docs/superpowers/runbooks/otel-opmsg-e2e.md`: - -```markdown -# Runbook: OTel OP_MSG propagation end-to-end (manual) - -Validates DRIVERS-3454 end to end: a sync-driver client span linked to a server child span. - -## Prerequisites -- A local build of the server POC branch (10gen/mongo PR #49930), which accepts OP_MSG - section kind 3 and starts a server span from it. NOTE: the POC does NOT yet advertise - `tracingSupport` in hello. Until SERVER-107128 adds it, temporarily force the driver - capability on for this manual run (see step 3). -- An OpenTelemetry collector / exporter the server POC is configured to export traces to - (e.g. Jaeger via OTLP), plus a Micrometer Tracing OTel bridge on the client. - -## Steps -1. Build & run the server POC `mongod` with tracing enabled and sampling at 100%. -2. Start a Jaeger all-in-one (or OTLP collector) and point both server and client exporters at it. -3. In a scratch sync-driver program, configure a `MongoClient` with an `ObservationRegistry` - that has an OTel-backed `DefaultTracingObservationHandler` (so `traceParent()` is non-null - and sampled). Run a `find`. - - Temporary capability override for the run (POC server lacks the hello flag): start an - OTel root span yourself, then run the command. Because the server POC accepts kind 3 - unconditionally, set the driver `tracingSupported` to true by connecting to a server - whose hello you patch to include `tracingSupport: true`, OR temporarily hardcode - `isTracingSupport()`/`tracingSupported` to `true` on the local branch for the manual run - only (revert before commit). -4. In Jaeger, confirm a single trace contains BOTH the client `find` span and a server span, - with the server span's parent = the client span id sent in the traceparent. - -## Pass criteria -- One trace, two+ spans, correct parent/child linkage across the client→server boundary. -- With the driver capability off (default), no server span is created (negative check). - -## Notes -- This proves the wire format end to end; the automated Phase 1 test - (`CommandMessageOtelTraceContextTest`) is the regression guard. -- Findings (any format mismatch, tracestate handling, flags) feed back into the spec. -``` - -- [ ] **Step 2: Commit** - -```bash -git add docs/superpowers/runbooks/otel-opmsg-e2e.md -git commit -m "DRIVERS-3454: add Phase 2 e2e validation runbook" -``` - ---- - -## Task 8: Full module verification - -- [ ] **Step 1: Format + static checks + tests for driver-core** - -Run: `./gradlew :driver-core:spotlessApply :driver-core:check` -Expected: BUILD SUCCESSFUL. Fix any spotless/checkstyle issues in the files you touched (copyright headers, import order, no unused imports — particularly the Micrometer `TraceContext` import note in Task 2). - -- [ ] **Step 2: Confirm the new tests ran and passed** - -Run: `./gradlew :driver-core:test --tests "*OtelTraceContext*" --tests "*MicrometerTraceParent*" --tests "*DescriptionHelperTracing*"` -Expected: PASS. - -- [ ] **Step 3: Final commit if spotless changed anything** - -```bash -git add -A && git commit -m "DRIVERS-3454: apply spotless formatting" || echo "nothing to commit" -``` - ---- - -## Self-Review (completed during planning) - -**Spec coverage:** -- §3.1 section kind 3 → Task 5. §3.2 W3C traceparent format → Task 2 (`00-…-01`). §3.3 request-only / sparse / sampled-only → Task 2 (null unless sampled) + Task 5 (omit when null). §4 `tracingSupport` negotiation → Tasks 3–5 (parse → MessageSettings → gate). §6.1 `TraceContext.traceParent()` → Task 1. §6.2 read capability → Task 3. §6.3 inject → Task 5. §6.4 Phase 1 automated → Task 6; Phase 2 manual → Task 7. §7 testing → Tasks 2,3,6 + Task 8 check. -- Out-of-scope items (reactive, mongos, tracestate beyond pass-through, sampling rework) intentionally have no task — matches spec §6.5. - -**Known follow-ups (not blockers for the POC):** -- Server `hello` `tracingSupport` flag does not exist yet (SERVER-107128). Phase 1 fully validates the driver without it; Phase 2 documents the temporary override. -- The `ConnectionDescription.tracingSupportOverride` approach (Task 3) keeps public constructors binary-compatible; a production implementation would likely fold the field into a new constructor + builder during the non-POC work. - -**Type consistency:** `traceParent()` (Task 1/2/5/6), `isTracingSupport()` on `ConnectionDescription` (Task 3/4), `isTracingSupported()`/`tracingSupported(...)` on `MessageSettings` (Task 4/5/6), `PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT` (Task 5) — names are used consistently across tasks. diff --git a/docs/superpowers/plans/2026-07-13-otel-telemetry-section-reference-impl.md b/docs/superpowers/plans/2026-07-13-otel-telemetry-section-reference-impl.md deleted file mode 100644 index 74cd117ccf8..00000000000 --- a/docs/superpowers/plans/2026-07-13-otel-telemetry-section-reference-impl.md +++ /dev/null @@ -1,695 +0,0 @@ -# DRIVERS-3454 Reference Implementation Plan — BSON Telemetry Section - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Convert the DRIVERS-3454 POC into the production-quality reference implementation matching the shipped server contract: OP_MSG section kind 3 carrying `{otel: {traceparent: "<55-char W3C>"}}` as BSON, gated on `maxWireVersion >= 29`. - -**Architecture:** All changes live in `driver-core` internal packages (shared by sync/reactive). The hello-capability plumbing from the POC is deleted; the gate becomes the existing `MessageSettings.getMaxWireVersion()` compared to a new `ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION = 29`. `MicrometerTracer` is tightened so `TraceContext.traceParent()` never yields a value the server would reject. Prose tests + an orchestration-wired e2e test verify server-span linkage. - -**Tech Stack:** Java 8 (driver-core baseline), JUnit 5, Micrometer Tracing test kit (`InMemoryOtelSetup`), Gradle Kotlin DSL. - -**Spec:** `docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md` (read it first). - -## Global Constraints - -- **LOCAL ONLY: never `git push`.** All commits stay on local branch `nabil_otel_context`. -- Java 8 source baseline in driver-core — no `var`, records, `Stream.toList()`, text blocks, etc. -- No public API changes. Everything under `com.mongodb.internal.*` except the *removal* of the not-yet-released POC additions to `com.mongodb.connection.ConnectionDescription` (added on this branch only, never released — safe to delete). -- Copyright header `Copyright 2008-present MongoDB, Inc.` on new files. No `System.out.println`; SLF4J only. -- Run `./gradlew spotlessApply` before each commit if formatting-sensitive files changed. -- Server contract (from merged 10gen/mongo#56646): section kind 3; payload BSON `{otel: {traceparent: }}`; traceparent exactly 55 chars `00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>`, non-zero trace-id/span-id, version != `ff`, **no tracestate**; max section 4096 bytes; at most one telemetry section per message. - ---- - -### Task 1: Branch prep — park unrelated noise, merge origin/main - -**Files:** -- No source files; git operations only. - -**Interfaces:** -- Produces: a working tree at `origin/main`-merged state with only DRIVERS-3454 content, on which all later tasks build. - -- [ ] **Step 1: Park unrelated local modifications (do NOT delete)** - -The working tree has unrelated noise. Stash tracked modifications with a label; leave untracked files (`prompt.txt`, `bson-kotlin/.../ReproJava6230Test.kt`) in place — they are ignored by later tasks and must not be committed. - -```bash -git status --porcelain -# reset the accidentally staged unrelated file, then stash all unrelated tracked changes -git restore --staged bson/src/test/unit/org/bson/BsonDocumentTest.java -git stash push -m "unrelated-local-noise-before-DRIVERS-3454" \ - bson/src/test/unit/org/bson/BsonDocumentTest.java \ - driver-scala/src/integrationTest/scala/tour/QuickTour.scala \ - driver-scala/src/test/scala/org/mongodb/scala/MongoClientSpec.scala \ - mongodb-crypt/build.gradle.kts -``` - -Expected: `git status --porcelain` shows only the untracked `prompt.txt`, `ReproJava6230Test.kt`, and the followups doc if still staged (`docs/superpowers/followups/...` — commit that separately if staged: `git commit -m "DRIVERS-3454: follow-up notes on mongos version skew" docs/superpowers/followups/2026-06-08-mongos-version-skew-and-negotiation.md`). - -- [ ] **Step 2: Merge origin/main** - -```bash -git fetch origin -git merge origin/main --no-edit -``` - -If conflicts arise, they will most likely be in `driver-core` files the POC touched (`ConnectionDescription.java`, `CommandMessage.java`, `gradle/libs.versions.toml`). Resolve keeping BOTH the upstream changes and the POC changes (the POC changes get reworked/deleted in later tasks, so a mechanically correct merge is enough). Commit the merge. - -- [ ] **Step 3: Sanity-build driver-core** - -```bash -./gradlew :driver-core:compileJava :driver-core:compileTestJava -q -``` - -Expected: BUILD SUCCESSFUL. If the merge broke compilation, fix and `git commit --amend` the merge resolution. - ---- - -### Task 2: `ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION = 29` - -**Files:** -- Modify: `driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java` - -**Interfaces:** -- Produces: `public static final int NINE_DOT_ZERO_WIRE_VERSION = 29;` — consumed by Task 4's gate in `CommandMessage`. - -- [ ] **Step 1: Add the constant** - -In `ServerVersionHelper`, immediately after `EIGHT_DOT_ZERO_WIRE_VERSION = 25` (line ~33): - -```java - // Server 9.0 (WIRE_VERSION_90 = 29). Minimum wire version for the OP_MSG telemetry - // section (OTel trace-context propagation, DRIVERS-3454). - public static final int NINE_DOT_ZERO_WIRE_VERSION = 29; -``` - -Do NOT change `LATEST_WIRE_VERSION` (it deliberately tracks the newest wire version the driver fully supports). - -- [ ] **Step 2: Compile and commit** - -```bash -./gradlew :driver-core:compileJava -q -git add driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java -git commit -m "DRIVERS-3454: add NINE_DOT_ZERO_WIRE_VERSION (29) constant" -``` - ---- - -### Task 3: Tighten `MicrometerTracer` traceparent production - -**Files:** -- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java` (inner class `MicrometerTraceContext.traceParent()`, ~line 122) -- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java` (javadoc only) -- Test: `driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java` (exists from POC — rework) - -**Interfaces:** -- Consumes: nothing from other tasks. -- Produces: `TraceContext.traceParent()` returns either `null` or a guaranteed-server-valid 55-char string. Behavior change vs POC: **unsampled contexts now return a traceparent with flags `00`** (previously `null`); malformed/zero ids return `null`. - -- [ ] **Step 1: Rework the unit test to the new contract** - -Replace the POC assertions in `MicrometerTraceParentTest` with a matrix (keep the file's existing test scaffolding/mocks for building Micrometer contexts; adjust to these cases): - -```java - @Test - void shouldFormatSampledTraceParent() { - // given a context with traceId "0af7651916cd43dd8448eb211c80319c", spanId "b7ad6b7169203331", sampled=true - assertEquals("00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01", traceParent); - assertEquals(55, traceParent.length()); - } - - @Test - void shouldFormatUnsampledTraceParentWithZeroFlags() { - // same ids, sampled=false (and also sampled=null) - assertEquals("00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-00", traceParent); - } - - @Test - void shouldReturnNullForInvalidIds() { - // each of these must yield null: - // traceId all zeros: "00000000000000000000000000000000" - // spanId all zeros: "0000000000000000" - // traceId wrong length: "abc" - // spanId wrong length: "abc" - // traceId uppercase hex: "0AF7651916CD43DD8448EB211C80319C" - // traceId null / spanId null - assertNull(traceParent); - } -``` - -- [ ] **Step 2: Run to verify the new cases fail** - -```bash -./gradlew :driver-core:test --tests 'com.mongodb.internal.observability.micrometer.MicrometerTraceParentTest' -q -``` - -Expected: FAIL (unsampled currently returns null; invalid ids currently pass through). - -- [ ] **Step 3: Implement** - -In `MicrometerTracer.MicrometerTraceContext`, replace the body of `traceParent()` from the `io.micrometer.tracing.TraceContext ctx` null-check down, and add the helper: - -```java - io.micrometer.tracing.TraceContext ctx = tracingContext.getSpan().context(); - if (ctx == null) { - return null; - } - String traceId = ctx.traceId(); - String spanId = ctx.spanId(); - if (!isValidNonZeroLowercaseHex(traceId, 32) || !isValidNonZeroLowercaseHex(spanId, 16)) { - return null; - } - Boolean sampled = ctx.sampled(); - return "00-" + traceId + "-" + spanId + (sampled != null && sampled ? "-01" : "-00"); - } - - /** - * The server ({@code validateW3CTraceparent}) rejects ids that are not exactly the expected - * length of lowercase hex, or that are all zeroes. Never emit a traceparent it would reject. - */ - private static boolean isValidNonZeroLowercaseHex(@Nullable final String value, final int expectedLength) { - if (value == null || value.length() != expectedLength) { - return false; - } - boolean nonZero = false; - for (int i = 0; i < value.length(); i++) { - char c = value.charAt(i); - if (!((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f'))) { - return false; - } - if (c != '0') { - nonZero = true; - } - } - return nonZero; - } -``` - -Update the javadoc of `TraceContext.traceParent()` to say: returns the 55-char W3C traceparent (`00-<32 hex>-<16 hex>-`; flags `01` sampled / `00` unsampled), or `null` when there is no valid context (no-op span, missing/zero/malformed ids). Never includes tracestate. - -- [ ] **Step 4: Run tests, verify pass** - -```bash -./gradlew :driver-core:test --tests 'com.mongodb.internal.observability.micrometer.MicrometerTraceParentTest' -q -``` - -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add driver-core/src/main/com/mongodb/internal/observability/micrometer/ driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/ -git commit -m "DRIVERS-3454: guarantee traceParent() is server-valid; propagate unsampled with flags 00" -``` - ---- - -### Task 4: `CommandMessage` — BSON telemetry section, wire-version gate - -**Files:** -- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` -- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` (exists from POC — rework) - -**Interfaces:** -- Consumes: `ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION` (Task 2); `TraceContext.traceParent()` contract (Task 3); existing `OperationContext.getTracingSpan()` (returns `@Nullable Span`). -- Produces: OP_MSG bytes with trailing section kind 3 whose payload is the BSON document `{otel: {traceparent: }}`. `getCommandDocument()` keeps ignoring the trailing non-sequence section (POC behavior retained). - -- [ ] **Step 1: Rework the round-trip test** - -Rework `CommandMessageOtelTraceContextTest` (keep its existing harness for encoding a `CommandMessage` and re-reading the produced buffer; the POC test asserted a C-string payload). New assertions: - -```java - @Test - void shouldWriteTelemetrySectionWhenWireVersionAtLeast29AndSpanActive() { - // settings: MessageSettings.builder().maxWireVersion(NINE_DOT_ZERO_WIRE_VERSION).build() - // operationContext with a tracing span whose context().traceParent() returns - // "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01" - // after encode(): scan sections after the body; expect one byte 0x03 followed by a BSON document - BsonDocument telemetry = readTelemetrySectionDocument(buffer); - assertEquals(new BsonDocument("otel", new BsonDocument("traceparent", - new BsonString("00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"))), telemetry); - } - - @Test - void shouldNotWriteTelemetrySectionWhenWireVersionBelow29() { - // identical setup but maxWireVersion = EIGHT_DOT_ZERO_WIRE_VERSION (25) => no section kind 3 - } - - @Test - void shouldNotWriteTelemetrySectionWhenNoSpan() { - // wire version 29, operationContext.getTracingSpan() == null => no section kind 3 - } - - @Test - void shouldNotWriteTelemetrySectionWhenTraceParentNull() { - // wire version 29, span present but context().traceParent() == null => no section kind 3 - } - - @Test - void getCommandDocumentIgnoresTelemetrySection() { - // encode with section present; CommandMessage.getCommandDocument(...) still returns the command - // (retains the POC regression test, renamed) - } - - @Test - void shouldWriteTelemetrySectionAfterDocumentSequences() { - // encode a command with a document-sequence payload AND an active span at wire version 29; - // assert sequence section(s) precede the kind-3 section and the command re-parses correctly - } -``` - -`readTelemetrySectionDocument` helper: walk sections after the body exactly like the production parser (kind byte, then for kind 1 skip `int32` size bytes, for kind 3 decode one BSON document via `new BsonDocumentCodec().decode(new BsonBinaryReader(...))`), fail if kind 3 absent. - -- [ ] **Step 2: Run to verify failures** - -```bash -./gradlew :driver-core:test --tests 'com.mongodb.internal.connection.CommandMessageOtelTraceContextTest' -q -``` - -Expected: FAIL (payload is still a C-string; gate is still `isTracingSupported()`). - -- [ ] **Step 3: Implement in `CommandMessage`** - -1. Rename the constant (~line 89) and update its javadoc: - -```java - /** - * Specifies that the `OP_MSG` section payload is a BSON telemetry document - * ({@code {otel: {traceparent: }}}). Mirrors the server's {@code kTelemetry = 3} - * section kind (DRIVERS-3454, 10gen/mongo#56646). - */ - private static final byte PAYLOAD_TYPE_3_TELEMETRY = 3; -``` - -2. Replace `writeOtelTraceContextSection` with: - -```java - private void writeTelemetryContextSection(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext) { - if (getSettings().getMaxWireVersion() < NINE_DOT_ZERO_WIRE_VERSION) { - return; - } - Span tracingSpan = operationContext.getTracingSpan(); - if (tracingSpan == null) { - return; - } - String traceParent = tracingSpan.context().traceParent(); - if (traceParent == null) { - return; - } - bsonOutput.writeByte(PAYLOAD_TYPE_3_TELEMETRY); - BsonDocument telemetry = new BsonDocument("otel", - new BsonDocument("traceparent", new BsonString(traceParent))); - new BsonDocumentCodec().encode(new BsonBinaryWriter(bsonOutput), telemetry, - EncoderContext.builder().build()); - } -``` - -Update the call site in `writeOpMsg()` (`writeOtelTraceContextSection(...)` → `writeTelemetryContextSection(...)`). Add imports: `org.bson.BsonString`, `org.bson.codecs.BsonDocumentCodec`, `org.bson.codecs.EncoderContext`, `static com.mongodb.internal.operation.ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION`. Remove imports of `OtelTracePropagationTestToggle`. Keep the `getCommandDocument()` handling of trailing non-sequence sections (update its comment to say `PAYLOAD_TYPE_3_TELEMETRY`). - -- [ ] **Step 4: Run tests, verify pass** - -```bash -./gradlew :driver-core:test --tests 'com.mongodb.internal.connection.CommandMessageOtelTraceContextTest' -q -``` - -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java -git commit -m "DRIVERS-3454: write BSON telemetry section gated on maxWireVersion >= 29" -``` - ---- - -### Task 5: Delete POC capability plumbing - -**Files:** -- Modify: `driver-core/src/main/com/mongodb/connection/ConnectionDescription.java` (revert POC additions: `tracingSupport` field, constructor param, `withTracingSupport`, `isTracingSupported`, equals/hashCode/toString entries) -- Modify: `driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java` (remove `withTracingSupport(...)` call at ~line 72 and `getTracingSupport` helper at ~line 175) -- Modify: `driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java` (remove `.tracingSupported(...)` at ~line 238) -- Modify: `driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java` (remove `tracingSupported` field, builder method, getter) -- Delete: `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java` -- Delete: `driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java` - -**Interfaces:** -- Consumes: Task 4 must be done first (CommandMessage no longer references the toggle or `isTracingSupported()`). -- Produces: clean tree — `git grep -i tracingSupport` and `git grep OtelTracePropagationTestToggle` return nothing. - -- [ ] **Step 1: Remove all plumbing** - -Use `git diff origin/main -- ` on each of the four modified files to see exactly what the POC added, and revert those hunks (the cleanest method: `git checkout origin/main -- driver-core/src/main/com/mongodb/connection/ConnectionDescription.java driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java` — valid because the POC changes are the ONLY branch changes to these files; verify with `git diff origin/main --stat` first, and re-apply by hand if upstream merge conflicts made the files diverge otherwise). Then: - -```bash -git rm driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java -git rm driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java -``` - -- [ ] **Step 2: Verify nothing references the removed API** - -```bash -git grep -il 'tracingSupport' -- '*.java' ; git grep -l 'OtelTracePropagationTestToggle' -- '*.java' -./gradlew :driver-core:compileJava :driver-core:compileTestJava -q -``` - -Expected: no grep hits; BUILD SUCCESSFUL. Fix any leftover references (e.g. `DescriptionHelperTest`, `ConnectionDescription` unit tests touched by the POC). - -- [ ] **Step 3: Run the affected test suites** - -```bash -./gradlew :driver-core:test --tests '*CommandMessage*' --tests '*ConnectionDescription*' --tests '*DescriptionHelper*' --tests '*MessageSettings*' -q -``` - -Expected: PASS. - -- [ ] **Step 4: Commit** - -```bash -git add -A driver-core -git commit -m "DRIVERS-3454: remove hello-capability plumbing and test toggle (wire-version gate supersedes)" -``` - ---- - -### Task 6: Full driver-core validation - -**Files:** none (verification gate). - -- [ ] **Step 1: Static checks + full driver-core unit tests** - -```bash -./gradlew spotlessApply -q -./gradlew :driver-core:check -q -``` - -Expected: BUILD SUCCESSFUL. Fix anything that fails (checkstyle on new code, unused imports, etc.). - -- [ ] **Step 2: Commit any fixups** - -```bash -git status --porcelain # if formatting changed files: -git add -A && git commit -m "DRIVERS-3454: formatting/static-check fixups" -``` - ---- - -### Task 7: Prose-test definitions document - -**Files:** -- Create: `docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md` - -**Interfaces:** -- Produces: the prose-test text that Task 8's Java test implements (test names must match the prose numbering). - -- [ ] **Step 1: Write the document** - -Content (verbatim; structured like the existing OTel spec prose tests README so it can be lifted into the future `mongodb/specifications` PR): - -```markdown -# Trace-Context Propagation Prose Tests (DRIVERS-3454) - -These tests complement the OpenTelemetry spec prose tests. Tests 1–4 are unit-level and -MUST NOT require a server. Test 5 requires a MongoDB 9.0+ server (maxWireVersion >= 29) -started with OTel tracing enabled and the OTLP file exporter -(`--setParameter opentelemetryTraceDirectory=` plus the tracing feature flag and -sampling parameters), running on the same host as the test. - -## 1. Telemetry section is attached when supported and traced - -With tracing enabled and an active operation span, encode a command targeting a -connection whose `maxWireVersion` is >= 29. Assert the resulting OP_MSG contains exactly -one section of kind 3 whose payload is a BSON document of the form -`{otel: {traceparent: }}`, where `traceparent` is 55 characters, -`00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>`, and the trace-id/span-id match -the active span's context. - -## 2. Telemetry section is omitted for older servers - -Same as (1) but with `maxWireVersion` < 29. Assert no section of kind 3 is present. - -## 3. Telemetry section is omitted without an active span - -With tracing disabled (or no active span, e.g. monitoring/auth commands), encode a -command at `maxWireVersion` >= 29. Assert no section of kind 3 is present. - -## 4. Malformed trace context is never sent - -With an active span whose context cannot produce a valid W3C traceparent (zero trace-id, -zero span-id, wrong-length or non-lowercase-hex ids), assert no section of kind 3 is -present (drivers MUST omit the section rather than send an invalid traceparent). - -## 5. End-to-end server span linkage - -With tracing enabled, run a CRUD operation (e.g. `find`) against the configured 9.0+ -server. Capture the driver's finished command span (client side). Then read the server's -exported OTLP JSON from the trace directory and assert a server span exists whose -`traceId` equals the client command span's trace-id and whose `parentSpanId` equals the -client command span's span-id. Allow for the server's batch export interval when polling. -``` - -- [ ] **Step 2: Commit** - -```bash -git add docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md -git commit -m "DRIVERS-3454: prose-test definitions for trace-context propagation" -``` - -Note: prose tests 1–4 are implemented by `CommandMessageOtelTraceContextTest` (Task 4) and `MicrometerTraceParentTest` (Task 3); add a javadoc line to each test class referencing this document and the prose test numbers. - ---- - -### Task 8: E2E server-span linkage test - -**Files:** -- Create: `driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java` - -**Interfaces:** -- Consumes: `AbstractMicrometerProseTest` patterns (`InMemoryOtelSetup`, `getMongoClientSettingsBuilder()`, `ObservabilitySettings.micrometerBuilder()`), env-var helper `setEnv` if needed for enabling tracing. -- Produces: prose test 5 implementation, gated on system property `org.mongodb.test.otel.trace.dir`. - -- [ ] **Step 1: Write the test** - -```java -/* - * Copyright 2008-present MongoDB, Inc. - * ... (standard Apache header, copy from AbstractMicrometerProseTest) - */ -package com.mongodb.client.observability; - -import com.mongodb.MongoClientSettings; -import com.mongodb.client.MongoClient; -import com.mongodb.client.MongoClients; -import com.mongodb.client.MongoCollection; -import com.mongodb.observability.ObservabilitySettings; -import io.micrometer.observation.ObservationRegistry; -import io.micrometer.tracing.exporter.FinishedSpan; -import io.micrometer.tracing.test.reporter.inmemory.InMemoryOtelSetup; -import org.bson.Document; -import org.junit.jupiter.api.AfterEach; -import org.junit.jupiter.api.BeforeEach; -import org.junit.jupiter.api.DisplayName; -import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.condition.EnabledIfSystemProperty; - -import java.io.IOException; -import java.nio.charset.StandardCharsets; -import java.nio.file.Files; -import java.nio.file.Path; -import java.nio.file.Paths; -import java.util.List; -import java.util.stream.Collectors; -import java.util.stream.Stream; - -import static com.mongodb.ClusterFixture.getDefaultDatabaseName; -import static com.mongodb.client.Fixture.getMongoClientSettingsBuilder; -import static org.junit.jupiter.api.Assertions.assertTrue; -import static org.junit.jupiter.api.Assertions.fail; - -/** - * Prose test 5 (docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md): - * end-to-end server span linkage via the server's OTLP file exporter. - * - *

Requires a MongoDB 9.0+ (maxWireVersion >= 29) server on the same host, started with - * {@code --setParameter opentelemetryTraceDirectory=

} (plus tracing feature flag and - * sampling parameters), and the test run with - * {@code -Dorg.mongodb.test.otel.trace.dir=}. Skipped otherwise.

- */ -@EnabledIfSystemProperty(named = "org.mongodb.test.otel.trace.dir", matches = ".+") -public class ServerSpanLinkageProseTest { - - private static final long EXPORT_POLL_TIMEOUT_MS = 30_000; - private static final long EXPORT_POLL_INTERVAL_MS = 1_000; - - private final ObservationRegistry observationRegistry = ObservationRegistry.create(); - private InMemoryOtelSetup memoryOtelSetup; - private InMemoryOtelSetup.Builder.OtelBuildingBlocks inMemoryOtel; - - @BeforeEach - void setUp() { - memoryOtelSetup = InMemoryOtelSetup.builder().register(observationRegistry); - inMemoryOtel = memoryOtelSetup.getBuildingBlocks(); - } - - @AfterEach - void tearDown() { - memoryOtelSetup.close(); - } - - @Test - @DisplayName("Prose test 5: server emits a child span of the driver command span") - void testServerSpanLinkage() throws Exception { - MongoClientSettings clientSettings = getMongoClientSettingsBuilder() - .observabilitySettings(ObservabilitySettings.micrometerBuilder() - .observationRegistry(observationRegistry) - .build()) - .build(); - - try (MongoClient client = MongoClients.create(clientSettings)) { - MongoCollection collection = - client.getDatabase(getDefaultDatabaseName()).getCollection("serverSpanLinkage"); - collection.find().first(); - } - - List clientSpans = inMemoryOtel.getFinishedSpans(); - assertTrue(clientSpans.size() >= 2, "expected operation + command client spans, got: " + clientSpans); - // command span is the innermost (first finished) span; see AbstractMicrometerProseTest ordering - FinishedSpan commandSpan = clientSpans.get(0); - String traceId = commandSpan.getTraceId(); - String commandSpanId = commandSpan.getSpanId(); - - Path traceDir = Paths.get(System.getProperty("org.mongodb.test.otel.trace.dir")); - long deadline = System.currentTimeMillis() + EXPORT_POLL_TIMEOUT_MS; - while (System.currentTimeMillis() < deadline) { - if (serverSpanLinked(traceDir, traceId, commandSpanId)) { - return; - } - Thread.sleep(EXPORT_POLL_INTERVAL_MS); - } - fail("no server span found in " + traceDir + " with traceId=" + traceId - + " and parentSpanId=" + commandSpanId); - } - - /** - * Scans OTLP JSON export files for a span with the given traceId whose parentSpanId is the - * driver command span. OTLP file exports contain resourceSpans[].scopeSpans[].spans[] objects - * with hex-encoded traceId/spanId/parentSpanId fields; a simple containment check on the two - * hex ids in the same file line is sufficient and avoids a protobuf/JSON-schema dependency. - */ - private static boolean serverSpanLinked(final Path traceDir, final String traceId, final String parentSpanId) - throws IOException { - if (!Files.isDirectory(traceDir)) { - return false; - } - try (Stream files = Files.walk(traceDir)) { - List exportFiles = files.filter(Files::isRegularFile).collect(Collectors.toList()); - for (Path file : exportFiles) { - for (String line : Files.readAllLines(file, StandardCharsets.UTF_8)) { - if (line.contains(traceId) && line.contains("\"parentSpanId\":\"" + parentSpanId + "\"")) { - return true; - } - } - } - } - return false; - } -} -``` - -Adaptation note for the implementer: verify the exact accessor names on `FinishedSpan` (`getTraceId()`/`getSpanId()`) and the client-span ordering against `AbstractMicrometerProseTest` (it asserts `get(0)` is the command span, e.g. `"find"`); also check how that class enables tracing — if tracing is env-var-gated (`ENV_OBSERVABILITY_ENABLED`), reuse its `setEnv` helper in `@BeforeEach`/`@AfterEach` the same way existing tests do. Match whatever compiles against the merged main. - -- [ ] **Step 2: Verify compile + skip behavior** - -```bash -./gradlew :driver-sync:compileFunctionalJava -q 2>/dev/null || ./gradlew :driver-sync:compileTestJava -q -# run without the property; the test must be SKIPPED, not failed (check the task's test report) -``` - -Expected: compiles; test skipped when property absent. (Full e2e execution happens in Task 9's runbook flow — it needs the special server.) - -- [ ] **Step 3: Commit** - -```bash -git add driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java -git commit -m "DRIVERS-3454: e2e server-span linkage prose test (OTLP file exporter)" -``` - ---- - -### Task 9: Runbook update + manual e2e run - -**Files:** -- Modify: `docs/superpowers/runbooks/otel-opmsg-e2e.md` - -**Interfaces:** -- Consumes: Task 8's test and its `org.mongodb.test.otel.trace.dir` property. - -- [ ] **Step 1: Rewrite the runbook** - -Replace its POC content with the current workflow: - -1. Download a server 9.0 binary containing 10gen/mongo#56646 (merged 2026-07-06) from Evergreen (any master waterfall compile task after that date; note the artifact URL used). -2. Start it locally, e.g.: - -```bash -mkdir -p /tmp/otel-e2e/{db,traces} -/mongod --dbpath /tmp/otel-e2e/db --port 27017 \ - --setParameter opentelemetryTraceDirectory=/tmp/otel-e2e/traces \ - --setParameter featureFlagOtelTracing=true \ - --setParameter 'opentelemetrySamplingRates={"default": 1.0}' -``` - -(Implementer: confirm the exact feature-flag and sampling parameter names from the downloaded binary via `mongod --help` / `getParameter '*'` — they live in `src/mongo/otel/traces/tracing_feature_flags.idl` and `trace_sampling_parameters.idl` on the server; record the working invocation in the runbook.) - -3. Run the e2e test: - -```bash -./gradlew :driver-sync:test --tests 'com.mongodb.client.observability.ServerSpanLinkageProseTest' \ - -Dorg.mongodb.test.uri="mongodb://localhost:27017" \ - -Dorg.mongodb.test.otel.trace.dir=/tmp/otel-e2e/traces -``` - -4. Keep the previous Jaeger flow as an optional "visual verification" appendix (`opentelemetryHttpEndpoint=http://localhost:4318/v1/traces`), removing all references to `OtelTracePropagationTestToggle` and `tracingSupport`. - -- [ ] **Step 2: Execute the runbook end-to-end** - -Actually perform steps 1–3 and record results. Expected: `ServerSpanLinkageProseTest` PASSES against the real server. If the server rejects or ignores the section, debug with `superpowers:systematic-debugging` before touching driver code — likely causes: feature flag name, sampling off, wire version reported < 29, or traceparent validation mismatch. - -- [ ] **Step 3: Commit** - -```bash -git add docs/superpowers/runbooks/otel-opmsg-e2e.md -git commit -m "DRIVERS-3454: update e2e runbook for 9.0 server and OTLP file exporter" -``` - ---- - -### Task 10: Supersede old spec + final validation - -**Files:** -- Modify: `docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md` (status header only) - -- [ ] **Step 1: Mark the old design superseded** - -Change its `- **Status:**` line to: - -```markdown -- **Status:** SUPERSEDED by `2026-07-13-otel-telemetry-section-reference-impl-design.md` (payload is now a BSON document; gating is by maxWireVersion >= 29, not a hello flag) -``` - -- [ ] **Step 2: Final full validation** - -```bash -./gradlew spotlessApply docs check -q -``` - -Expected: BUILD SUCCESSFUL. (Skip `scalaCheck` unless Scala files were touched; integration tests only with a running server per AGENTS.md.) - -- [ ] **Step 3: Commit** - -```bash -git add -A -git commit -m "DRIVERS-3454: mark POC design superseded; final validation fixups" -``` - -**Do not push. Everything stays local until Nabil says otherwise.** diff --git a/docs/superpowers/plans/2026-07-18-command-span-propagation.md b/docs/superpowers/plans/2026-07-18-command-span-propagation.md deleted file mode 100644 index 942f3e0189b..00000000000 --- a/docs/superpowers/plans/2026-07-18-command-span-propagation.md +++ /dev/null @@ -1,353 +0,0 @@ -# Command-Span Trace Propagation Implementation Plan (Option A) - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Propagate the traceparent of the **command** span (not the operation span) in the OP_MSG telemetry section, by creating the command span before message encoding. - -**Architecture:** `TracingManager.createTracingSpan` stops depending on encoded bytes (it reads the raw command document that `CommandMessage` already holds); `InternalStreamConnection` hoists span creation above `encode()` in both sync and async paths; `CommandMessage` gains an `encode` overload carrying the command span, consumed by `writeTelemetryContextSection`. Spec branch, prose docs, and e2e assertions flip from operation-span to command-span parentage. - -**Tech Stack:** Java 8 (driver-core), JUnit 5, GFM specs. - -**Spec:** `docs/superpowers/specs/2026-07-18-command-span-propagation-design.md` — read it first. - -## Global Constraints - -- **LOCAL ONLY: never `git push`** (parent repo branch `nabil_otel_context`; spec submodule branch `DRIVERS-3454`). -- Java 8 source; no public API changes; everything under `com.mongodb.internal.*`. -- `docs/superpowers/` is gitignored in the parent repo — use `git add -f` for docs. -- Do not change `RequestMessage.encode(ByteBufferBsonOutput, OperationContext)`'s signature (other message types use it). -- The operation span REMAINS the command span's parent (`TracingManager.createTracingSpan` line ~191-192) — only the *propagated* context changes. -- Behavior contract after this change: telemetry section carries the command span's traceparent; null command span (tracing disabled / sensitive command / no-span paths) ⇒ **no section**, even if an operation span exists. - ---- - -### Task 1: `TracingManager.createTracingSpan` reads the raw command document - -**Files:** -- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/TracingManager.java:174-186` -- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` (add package-private raw-command accessor) -- Modify: `driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java:448-455, 624-631` (call sites — argument change only in this task; reorder happens in Task 3) - -**Interfaces:** -- Produces: `public Span createTracingSpan(CommandMessage message, OperationContext operationContext, Predicate isSensitiveCommand, Supplier serverAddressSupplier, Supplier connectionIdSupplier)` — the `Supplier commandDocumentSupplier` parameter is REMOVED; the method reads `message.getRawCommandDocument()`. -- Produces: `CommandMessage`: `BsonDocument getRawCommandDocument()` (package-private is impossible across packages — TracingManager is in a different package, so make it `public` on the internal class, javadoc'd as internal) returning the `command` field (`CommandMessage.java:96`). - -- [ ] **Step 1: Add the accessor to `CommandMessage`** - -Next to the other getters: - -```java - /** - * The raw (pre-encoding) command document this message was constructed with. Unlike - * {@link #getCommandDocument(ByteBufferBsonOutput)}, this does not include fields added during encoding - * ({@code $db}, {@code $readPreference}) nor document-sequence fields, but its first key is always the - * command name, and it is available before {@code encode()} runs. - */ - public BsonDocument getRawCommandDocument() { - return command; - } -``` - -- [ ] **Step 2: Change `createTracingSpan`** - -In `TracingManager.java`, remove the `commandDocumentSupplier` parameter and replace its use: - -```java - public Span createTracingSpan(final CommandMessage message, - final OperationContext operationContext, - final Predicate isSensitiveCommand, - final Supplier serverAddressSupplier, - final Supplier connectionIdSupplier - ) { - - if (!isEnabled()) { - return null; - } - BsonDocument command = message.getRawCommandDocument(); - String commandName = command.getFirstKey(); - ... -``` - -The rest of the body is unchanged — `command.containsKey("getMore")` / `command.getInt64("getMore")` now read the raw document, where the `getMore` field is identically present. - -- [ ] **Step 3: Update the two call sites in `InternalStreamConnection` (drop the supplier argument only)** - -At `:448` and `:624`, delete the `() -> message.getCommandDocument(bsonOutput),` argument line. Do NOT move the calls yet (Task 3). - -- [ ] **Step 4: Compile and run tracing tests** - -Run: `./gradlew :driver-core:compileJava :driver-core:test --tests 'com.mongodb.internal.observability.micrometer.*' -q` -Expected: BUILD SUCCESSFUL, tests pass. - -- [ ] **Step 5: Commit** - -```bash -git add driver-core/src/main -git commit -m "DRIVERS-3454: createTracingSpan reads the raw command document (no encoded-bytes dependency)" -``` - ---- - -### Task 2: `CommandMessage.encode` overload carrying the command span - -**Files:** -- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java:250, 295, 302-315` -- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` - -**Interfaces:** -- Consumes: existing `RequestMessage.encode(ByteBufferBsonOutput, OperationContext)` (untouched). -- Produces: `public void encode(ByteBufferBsonOutput bsonOutput, OperationContext operationContext, @Nullable Span commandSpan)` on `CommandMessage`. The 2-arg `encode` inherited from `RequestMessage` now results in NO telemetry section (span defaults to null). - -- [ ] **Step 1: Rework the unit test to the new contract (TDD)** - -In `CommandMessageOtelTraceContextTest`, change every `message.encode(output, operationContext)` call in the positive-path tests to `message.encode(output, operationContext, span)`, and repurpose `buildOperationContext(span)` → `buildOperationContext()` (the operation context no longer needs to carry the span for these tests; keep the method if other setup depends on it, passing null). Update the test matrix: - -```java - // Prose test 1 — section present: encode(output, operationContext, span) with wire version 29 => kind-3 section - // whose traceparent matches the span passed to encode (the COMMAND span). - // Prose test 2 — wire version 25 + span passed => no section. - // Prose test 3 — encode(output, operationContext, null) at wire version 29 => no section - // (rename shouldNotWriteTelemetrySectionWhenNoSpan; note: this now also covers "operation span exists but - // command span is null", since the operation context's span is irrelevant to the section). - // Prose test 4 — span whose context().traceParent() returns null => no section. - // getCommandDocumentIgnoresTelemetrySection — unchanged semantics, 3-arg encode with span. - // shouldWriteTelemetrySectionAfterDocumentSequences — 3-arg encode with span. - // NEW: shouldNotWriteTelemetrySectionViaTwoArgEncode — message.encode(output, operationContext) (2-arg) - // never writes a section even at wire version 29 (guards monitoring/compression/other callers). -``` - -- [ ] **Step 2: Run to verify failures** - -Run: `./gradlew :driver-core:test --tests 'com.mongodb.internal.connection.CommandMessageOtelTraceContextTest' -q` -Expected: FAIL (no 3-arg encode exists). - -- [ ] **Step 3: Implement in `CommandMessage`** - -Add a transient field + overload (a `CommandMessage` is encoded at most once per send attempt, single-threaded, so a field set for the duration of encode is safe; the overload documents that): - -```java - @Nullable - private Span commandSpanForEncoding; - - /** - * Encodes the message, attaching the OP_MSG telemetry section with {@code commandSpan}'s trace context when - * the gating conditions hold (see writeTelemetryContextSection). The 2-arg {@code encode} never attaches the - * section. Not thread-safe: a message must be encoded by one thread at a time (already the case — a - * CommandMessage is built and encoded per send attempt). - */ - public void encode(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext, - @Nullable final Span commandSpan) { - this.commandSpanForEncoding = commandSpan; - try { - encode(bsonOutput, operationContext); - } finally { - this.commandSpanForEncoding = null; - } - } -``` - -Change `writeTelemetryContextSection` (drop the `operationContext` parameter — no longer used; update its call at `:295`): - -```java - private void writeTelemetryContextSection(final ByteBufferBsonOutput bsonOutput) { - if (getSettings().getMaxWireVersion() < NINE_DOT_ZERO_WIRE_VERSION) { - return; - } - Span commandSpan = this.commandSpanForEncoding; - if (commandSpan == null) { - return; - } - String traceParent = commandSpan.context().traceParent(); - if (traceParent == null) { - return; - } - bsonOutput.writeByte(PAYLOAD_TYPE_3_TELEMETRY); - BsonDocument telemetry = new BsonDocument("otel", - new BsonDocument("traceparent", new BsonString(traceParent))); - new BsonDocumentCodec().encode(new BsonBinaryWriter(bsonOutput), telemetry, - EncoderContext.builder().build()); - } -``` - -Remove the now-unused `operationContext.getTracingSpan()` read here (do NOT remove `OperationContext.getTracingSpan()` itself — `TracingManager` uses it for parenting). - -- [ ] **Step 4: Run tests, verify pass** - -Run: `./gradlew :driver-core:test --tests 'com.mongodb.internal.connection.CommandMessageOtelTraceContextTest' -q` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java -git commit -m "DRIVERS-3454: encode overload carries the command span for the telemetry section" -``` - ---- - -### Task 3: Hoist span creation above encode in `InternalStreamConnection` - -**Files:** -- Modify: `driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java` sync `sendAndReceiveInternal` (~`:441-480`) and async equivalent (~`:615-650`). The one-way `send` path (`:508`) and compression paths (`:540`, `:681`) keep the 2-arg encode (no command span there today). - -**Interfaces:** -- Consumes: Task 1's `createTracingSpan` (no supplier param); Task 2's `encode(bsonOutput, operationContext, commandSpan)`. - -- [ ] **Step 1: Reorder the sync path** - -Current shape (`:443-455`): `encode(...)` then `tracingSpan = createTracingSpan(...)`. New shape: - -```java - CommandEventSender commandEventSender; - Span tracingSpan; - try (ByteBufferBsonOutput bsonOutput = new ByteBufferBsonOutput(this)) { - tracingSpan = operationContext - .getTracingManager() - .createTracingSpan(message, - operationContext, - cmdName -> SECURITY_SENSITIVE_COMMANDS.contains(cmdName) - || SECURITY_SENSITIVE_HELLO_COMMANDS.contains(cmdName), - () -> getDescription().getServerAddress(), - () -> getDescription().getConnectionId() - ); - try { - message.encode(bsonOutput, operationContext, tracingSpan); - } catch (RuntimeException | Error e) { - if (tracingSpan != null) { - tracingSpan.error(e); - tracingSpan.end(); - } - throw e; - } - ...rest unchanged (isLoggingCommandNeeded, getCommandDocument hydration, setQueryText, openScope)... -``` - -The existing failure handling after this point (`:484-486` `error/closeScope/end`) is unchanged — the new catch only covers the window where the span exists but the established handlers don't yet. - -- [ ] **Step 2: Reorder the async path identically** - -Same transformation at ~`:618-631` — `tracingSpan = createTracingSpan(...)` first, then `message.encode(bsonOutput, operationContext, tracingSpan)` inside a try/catch that calls `tracingSpan.error(e); tracingSpan.end();` before rethrowing/propagating through the existing async error handling (match how that block reports errors — it already has a `try` whose catch handles cleanup; ensure the span ends exactly once: if the surrounding catch already ends the span when non-null, rely on it instead of a nested catch — inspect and pick the single-end structure that fits the existing block). - -- [ ] **Step 3: Run driver-core connection + tracing tests** - -Run: `./gradlew :driver-core:test --tests 'com.mongodb.internal.connection.*' --tests 'com.mongodb.internal.observability.micrometer.*' -q` -Expected: PASS. - -- [ ] **Step 4: Prose tests against local server (functional)** - -Requires a running MongoDB at localhost:27017. Run: -`./gradlew :driver-sync:test --tests 'com.mongodb.client.observability.MicrometerProseTest' -Dorg.mongodb.test.uri="mongodb://localhost:27017" -q` -Expected: PASS (span counts/names unchanged — only creation timing moved). - -- [ ] **Step 5: Commit** - -```bash -git add driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java -git commit -m "DRIVERS-3454: create command span before encode; propagate its context" -``` - ---- - -### Task 4: Flip e2e + prose docs to command-span parentage - -**Files:** -- Modify: `driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java` (assertion comments/javadoc — the span it selects, `getFinishedSpans().get(0)` named e.g. "find", IS the command span per `AbstractMicrometerProseTest` ordering; verify and update the javadoc/comments that currently say "operation span", and any variable names like `operationSpanId`) -- Modify: `docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md` (tests 1 and 5: trace-id/span-id of the **command span**) -- Modify: `docs/superpowers/runbooks/otel-opmsg-e2e.md` (any operation-span wording) - -**Interfaces:** -- Consumes: Tasks 1-3 (behavior now propagates the command span). - -- [ ] **Step 1: Update the e2e test's span selection and wording** - -Read the test first. If it currently picks the span by name "find" via `get(0)` — that IS the command span (`AbstractMicrometerProseTest` asserts `get(0).getName() == "find"` = command span, `get(1)` = operation span "find db.test"); then the previous operation-span behavior was asserted via `getParentId()` or similar adaptation made in the earlier fix — align the selection so the asserted parent is the command span's own span-id, and fix all comments/javadoc that explain operation-span semantics. - -- [ ] **Step 2: Update prose-test definitions and runbook wording** - -In `2026-07-13-otel-telemetry-section-prose-tests.md`: test 1's "trace-id/span-id match the active span's context" → "match the **command span**'s context"; test 5's "parentSpanId equals the client command span's span-id" (already command-oriented from the original draft — make sure any 2026-07-14-era rewording to "operation span" is reverted to command span). Same sweep in the runbook. - -- [ ] **Step 3: Run the e2e test against the local 9.0 server (the real proof)** - -Start the wire-29 server (from the runbook; binary at `~/MongoDB/otel-e2e-server/dist-test`, Docker image `otel-poc-mongod:latest`) with the OTLP file exporter + `featureFlagOtelTraceSampling=true` + `openTelemetryTracingSampling={defaultSampling: {samplingFactor: 1.0}}` and a trace directory volume-mounted to the host. Then: - -```bash -./gradlew :driver-sync:test --tests 'com.mongodb.client.observability.ServerSpanLinkageProseTest' \ - -Dorg.mongodb.test.uri="mongodb://localhost:27017" \ - -Dorg.mongodb.test.otel.trace.dir= --rerun -``` - -Expected: PASS with the server span's `parentSpanId` == the client **command** span id. Stop and remove the container afterwards. - -- [ ] **Step 4: Commit** - -```bash -git add -f driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md docs/superpowers/runbooks/otel-opmsg-e2e.md -git commit -m "DRIVERS-3454: e2e + prose docs assert command-span parentage" -``` - ---- - -### Task 5: Flip the spec branch to command span - -**Files:** -- Modify: `testing/resources/specifications/source/open-telemetry/open-telemetry.md` (branch `DRIVERS-3454` in the submodule) - -**Interfaces:** -- Consumes: nothing from other tasks (text-only), but merge AFTER Task 3 proves the behavior. - -- [ ] **Step 1: Amend the propagation subsection** - -On submodule branch `DRIVERS-3454` (verify `git branch --show-current`), in the "Propagating Trace Context to the Server" section, replace the operation-span paragraph and condition 3: - -Current text (post-review): "`traceparent` is the ... value of the **operation span**: the propagated context MUST be that of the operation span, not the command span; server spans therefore join the trace as children of the operation span. (In practice, drivers attach the trace context when encoding the wire message, before the command span is created.)" and "3. A valid `traceparent` value is available from the active operation span." - -New text: - -```markdown -`traceparent` is the [W3C traceparent](https://www.w3.org/TR/trace-context/#traceparent-header) value of the **command -span**: the propagated context MUST be that of the command span for the command being sent, so server spans join the -trace as children of the exact command (and retry attempt) that produced them. -``` - -and condition 3: "3. A valid `traceparent` value is available from the command span for the command being sent." Also update the monitoring/auth sentence if it references "operation span" ("Connections that carry no operation span" → "Commands that carry no command span (for example server monitoring, authentication, and security-sensitive commands) naturally send no section, since condition 3 cannot hold."). - -- [ ] **Step 2: Amend the OTel-spec commit and verify hooks** - -```bash -cd testing/resources/specifications -git add source/open-telemetry/open-telemetry.md -git commit --amend --no-edit # amends "DRIVERS-3454: specify trace context propagation to the server" -pre-commit run --files source/open-telemetry/open-telemetry.md -``` - -Expected: hooks pass (if mdformat rewrites, re-add and re-amend until clean). If the OTel commit is not HEAD, use the reset-soft rebuild pattern from the ledger instead of amend. NOTE: the user's fork branch `nabil/nh/otel/DRIVERS-3454` will need another force push (their call — do not push). - -- [ ] **Step 3: Update the parent-repo design docs** - -In the parent repo: `docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md` — add one line under §3.1 noting the 2026-07-18 amendment (command span propagated; see `2026-07-18-command-span-propagation-design.md`). Commit with `git add -f`. - -```bash -git add -f docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md -git commit -m "DRIVERS-3454: note command-span amendment in reference-impl design" -``` - ---- - -### Task 6: Full validation - -**Files:** none. - -- [ ] **Step 1: Full driver-core check (no concurrent Gradle builds!)** - -```bash -./gradlew spotlessApply :driver-core:check -q -``` - -Expected: BUILD SUCCESSFUL, except the known pre-existing environmental failure `TypeMqlValuesFunctionalTest.asStringTestNested` (fails identically on origin/main vs the local server — ignore it; anything else failing must be investigated). - -- [ ] **Step 2: Commit any spotless fixups** - -```bash -git status --porcelain # if formatting changed files: -git add -A driver-core && git commit -m "DRIVERS-3454: formatting fixups" -``` diff --git a/docs/superpowers/plans/2026-07-18-drivers-3454-spec-change.md b/docs/superpowers/plans/2026-07-18-drivers-3454-spec-change.md deleted file mode 100644 index 8228a6855d6..00000000000 --- a/docs/superpowers/plans/2026-07-18-drivers-3454-spec-change.md +++ /dev/null @@ -1,235 +0,0 @@ -# DRIVERS-3454 Minimal Spec Change Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Draft the DRIVERS-3454 normative spec change (OP_MSG Payload Type 3 + OTel trace-context propagation) on a local branch of the `mongodb/specifications` submodule. - -**Architecture:** Two markdown edits in the spec submodule at `testing/resources/specifications`: the wire-format layer lands in `source/message/OP_MSG.md`, the driver-behavior layer in `source/open-telemetry/open-telemetry.md`. No test files, no tooling regeneration. - -**Tech Stack:** GitHub Flavored Markdown (120-char line width), git. - -**Spec:** `docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md` (in the parent repo) — read it first. - -## Global Constraints - -- **LOCAL ONLY: never `git push`**, never update the parent repo's submodule pointer. -- All work happens inside `/Users/nabil.hachicha/MongoDB/mongo-java-driver/nabil_otel_context/testing/resources/specifications` on branch `DRIVERS-3454` based on `origin/master` (repo default; there is no `main`). -- Markdown at **120-character line width**; match each file's existing formatting exactly (note: `OP_MSG.md` uses `### Changelog`, `open-telemetry.md` uses `## Changelog`). -- Changelog entries dated **2026-07-18**, prepended as the first bullet. -- Normative text only: no changes under `tests/`, no Future Work section additions. -- Normative content must match the shipped server contract: payload `{ otel: { traceparent: } }`; gate `maxWireVersion >= 29` (MongoDB 9.0); traceparent exactly 55 chars `00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>`, non-zero ids; omit-on-invalid; no tracestate; unsampled propagated with flags `00`; at most one section per message; request-only; **operation span** is what's propagated. - ---- - -### Task 1: Branch setup - -**Files:** none (git only), all commands run from `testing/resources/specifications`. - -**Interfaces:** -- Produces: local branch `DRIVERS-3454` at `origin/master`, on which Tasks 2–3 commit. - -- [ ] **Step 1: Fetch and branch** - -```bash -cd /Users/nabil.hachicha/MongoDB/mongo-java-driver/nabil_otel_context/testing/resources/specifications -git fetch origin master -git checkout -b DRIVERS-3454 origin/master -git log --oneline -1 # note the base commit in your report -``` - -Expected: new branch `DRIVERS-3454` checked out, clean status. (The submodule was previously in detached-HEAD state; that is normal. Do NOT touch the parent repo.) - ---- - -### Task 2: `source/message/OP_MSG.md` — Payload Type 3 - -**Files:** -- Modify: `testing/resources/specifications/source/message/OP_MSG.md` (Section struct ~line 46-55; payload-type rules ~line 74-75; payload-type detail list ~line 168-186; `### Changelog` ~line 406) - -**Interfaces:** -- Produces: the OP_MSG-side definition that Task 3's cross-link relies on (anchor: the Payload Type 3 paragraphs added here). - -- [ ] **Step 1: Extend the `Section` struct** - -In the `struct Section` code block, add one union member after the `sequence` struct: - -```c - document telemetry; // payloadType == 3 -``` - -- [ ] **Step 2: Add the payload-type rules** - -Immediately after the paragraph ending "…MUST do so when the batch contains more than one entry." (~line 75), insert: - -```markdown - -Each `OP_MSG` request MAY additionally contain **at most one** section with `Payload Type 3`, carrying telemetry -context. `Payload Type 3` is request-only: drivers MAY send it, and it MUST NOT appear in server replies. Its payload -is a single BSON document whose contents are defined by the -[OpenTelemetry specification](../open-telemetry/open-telemetry.md). Servers and intermediaries that do not recognize a -section kind fail the message (see below), so senders MUST gate emission of this section on server support as defined -by the OpenTelemetry specification. - -> [!NOTE] -> `Payload Type 2` is reserved for server-internal use (a security-token section populated by the server and -> infrastructure components, never by drivers), so the next payload type available for driver-emitted content is 3. -``` - -(If the file does not use `> [!NOTE]` callouts elsewhere, render the note as a plain paragraph starting with -"Note:" instead — check with `grep -n '\[!NOTE\]' source/message/OP_MSG.md` and match house style.) - -- [ ] **Step 3: Add the payload detail entry** - -After the "When the Payload Type is 1, the content of the payload is:" block (~line 174-180) and before the "Any -unknown Payload Types MUST result in an error…" paragraph (~line 182), insert: - -```markdown -When the Payload Type is 3, the content of the payload is: - -- document — a single BSON document containing telemetry context, as defined by the - [OpenTelemetry specification](../open-telemetry/open-telemetry.md). -``` - -Also update the sentence at ~line 185 "A fully constructed `OP_MSG` MUST contain exactly one `Payload Type 0`, and -optionally any number of `Payload Type 1`" so it reads: - -```markdown -A fully constructed `OP_MSG` MUST contain exactly one `Payload Type 0`, optionally any number of `Payload Type 1`, -and optionally at most one `Payload Type 3` -``` - -(keeping the remainder of that sentence/paragraph intact). - -- [ ] **Step 4: Changelog entry** - -Prepend under `### Changelog` (~line 406): - -```markdown -- 2026-07-18: Add `Payload Type 3` (telemetry context BSON document; DRIVERS-3454). -``` - -- [ ] **Step 5: Verify formatting and commit** - -```bash -awk 'length > 120 {print FILENAME": "NR" ("length")"}' source/message/OP_MSG.md # expect no output -git add source/message/OP_MSG.md -git commit -m "DRIVERS-3454: define OP_MSG Payload Type 3 (telemetry context)" -``` - ---- - -### Task 3: `source/open-telemetry/open-telemetry.md` — propagation behavior - -**Files:** -- Modify: `testing/resources/specifications/source/open-telemetry/open-telemetry.md` (new `####` subsection at the end of Implementation Requirements, i.e. after the `##### Exceptions` block that precedes `## Motivation for Change` ~line 325; Design Rationale ~line 415; `## Changelog` ~line 426) - -**Interfaces:** -- Consumes: Task 2's OP_MSG Payload Type 3 definition (cross-link `../message/OP_MSG.md`). - -- [ ] **Step 1: Add the propagation subsection** - -Insert immediately before `## Motivation for Change` (so it is the last `####` inside Implementation Requirements): - -```markdown -#### Propagating Trace Context to the Server - -Drivers MUST propagate the active trace context to servers that support it, so that server-generated spans join the -same distributed trace as the driver's spans. - -The trace context is carried in an `OP_MSG` section with -[`Payload Type 3`](../message/OP_MSG.md), whose payload is a single BSON document with the following schema: - -```json -{ "otel": { "traceparent": "" } } -``` - -`traceparent` is the [W3C traceparent](https://www.w3.org/TR/trace-context/#traceparent-header) value of the -**operation span**. Drivers attach the trace context when encoding the wire message, before the command span is -created; server spans therefore join the trace as children of the operation span. - -Drivers MUST attach the section to a command if and only if all of the following hold: - -1. Tracing is enabled on the `MongoClient`. -2. The connection's `maxWireVersion` is greater than or equal to 29 (MongoDB 9.0). -3. A valid `traceparent` value is available from the active operation span. - -A `traceparent` value is valid if and only if it is exactly 55 characters of the form -`00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>` and neither the trace-id nor the parent-id is all zeroes. -This mirrors the server-side validation. If no valid value is available, drivers MUST omit the section entirely -rather than send an invalid or truncated value. Drivers MUST NOT append a `tracestate` suffix. Unsampled trace -contexts are propagated with trace-flags `00`. - -A message MUST NOT contain more than one telemetry section. Connections that carry no operation span (for example -server monitoring and authentication) naturally send no section, since condition 3 cannot hold. - -No tracing data is returned in server responses as part of this feature. -``` - -Note on the nested code fence: the JSON block inside this section uses a standard triple-backtick fence; when -inserting, make sure the surrounding document structure stays valid (the snippet above shows the intended rendered -content — reproduce it with correct fencing in the file). - -- [ ] **Step 2: Add Design Rationale entries** - -After the `### No URI options` block (before `## Changelog`), insert: - -```markdown -### Wire protocol version gate instead of a hello capability - -An alternative was for servers to advertise support via a `hello` response field. Gating on `maxWireVersion` was -chosen instead: the wire protocol version acts as the schema contract for the telemetry payload, so drivers know -exactly which fields a server accepts, and any future payload additions require a wire version bump rather than a -second, parallel negotiation mechanism. - -### BSON document instead of a bare traceparent string - -Carrying the traceparent inside a BSON document allows future propagation fields to be added to the same section -without redesigning the payload format. -``` - -- [ ] **Step 3: Changelog entry** - -Prepend under `## Changelog`: - -```markdown -- 2026-07-18: Add trace context propagation to the server via the `OP_MSG` telemetry section (DRIVERS-3454). -``` - -- [ ] **Step 4: Verify formatting and commit** - -```bash -awk 'length > 120 {print FILENAME": "NR" ("length")"}' source/open-telemetry/open-telemetry.md # expect no output -git add source/open-telemetry/open-telemetry.md -git commit -m "DRIVERS-3454: specify trace context propagation to the server" -``` - ---- - -### Task 4: Repo checks - -**Files:** none. - -- [ ] **Step 1: Run available repo tooling** - -```bash -cd /Users/nabil.hachicha/MongoDB/mongo-java-driver/nabil_otel_context/testing/resources/specifications -command -v pre-commit && pre-commit run --files source/message/OP_MSG.md source/open-telemetry/open-telemetry.md || echo "pre-commit unavailable" -command -v mkdocs && mkdocs build --strict 2>&1 | tail -5 || echo "mkdocs unavailable" -``` - -If a hook rewrites formatting, review the diff, re-verify content is intact, and amend the relevant commit. If -tooling is unavailable, note that in the report; the `awk` width checks from Tasks 2-3 plus a markdown render -sanity-check (e.g. `grep -c '^#### Propagating'`) suffice. - -- [ ] **Step 2: Final state check** - -```bash -git log --oneline origin/master..HEAD # expect exactly the 2 commits from Tasks 2-3 -git status --porcelain # expect clean -cd /Users/nabil.hachicha/MongoDB/mongo-java-driver/nabil_otel_context && git status --porcelain -- testing/resources/specifications -``` - -The parent repo will show the submodule as modified (new HEAD); do NOT commit that pointer change — leave it, and -say so in the report. - -**Do not push anything. Everything stays local.** diff --git a/docs/superpowers/runbooks/otel-opmsg-e2e.md b/docs/superpowers/runbooks/otel-opmsg-e2e.md deleted file mode 100644 index f19151a1a6b..00000000000 --- a/docs/superpowers/runbooks/otel-opmsg-e2e.md +++ /dev/null @@ -1,237 +0,0 @@ -# Runbook: OTel OP_MSG propagation end-to-end (manual) - -Validates DRIVERS-3454 end to end against a real MongoDB 9.0/master server: a -sync-driver client trace linked to a server-side span created from the OP_MSG -telemetry section (section kind 3, `{otel: {traceparent: }}`), via the -server's OTLP file exporter. - -This supersedes the old POC-branch flow (10gen/mongo PR #49930). The server -feature landed as [10gen/mongo#56646](https://github.com/10gen/mongo/pull/56646) -(SERVER-129959, merged 2026-07-06) — no more `OtelTracePropagationTestToggle` / -`tracingSupport` hello-capability hacks are needed; the driver sends the -telemetry section whenever `maxWireVersion >= 29` (9.0+) and an active tracing -span yields a valid traceparent. - -Last executed successfully: 2026-07-14, against -`gitVersion ee3a67f8f8735b5e4aedd2b66ccb700e92d55205` -(`9.0.0-alpha0-ee3a67f8`, mongodb-mongo-master mainline of 2026-07-14 — -after the #56646 merge), linux arm64 dist-test binary run in Docker on a macOS -arm64 host. `ServerSpanLinkageProseTest` PASSED. - -Re-verified 2026-07-21 against a `9.0.0-alpha0` server in Docker container -`otel-opta` after the driver was changed to inject the traceparent from the -command span (rather than the operation span): `ServerSpanLinkageProseTest` -PASSED, with the server's `find` span `parentSpanId` equal to the client -command span's own span id. - -## Prerequisites - -- A `mongod` built from `10gen/mongo` `master` at or after the 2026-07-06 merge - of #56646. Verify with `mongod --version` → `gitVersion` must be a master - commit at/after that date; pre-merge binaries silently lack the telemetry - section support. -- The automated test this runbook exercises: - `driver-sync/src/test/functional/com/mongodb/client/observability/ServerSpanLinkageProseTest.java` - (prose test 5 in - `docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md`). - It is gated by `@EnabledIfSystemProperty(named = "org.mongodb.test.otel.trace.dir", ...)` - and skipped unless that property is set. - -## Step 1: Get a server binary (what actually worked) - -Do NOT submit new Evergreen patch builds for this — a no-op -`enterprise-macos-arm64` patch (and the same task on mainline) failed in -`bazel compile`, i.e. macOS `archive_dist_test` was broken on master at the -time of writing, and patch builds burn CI resources. Use an already-built -mainline artifact instead: - -1. Authenticate REST calls with the CLI's OAuth token against the **corp** - host (the public host rejects both static API keys and this token): - - ```bash - TOKEN=$(evergreen client get-oauth-token) - API=https://evergreen.corp.mongodb.com/api/rest/v2 - ``` - - (If the CLI complains it is too old, `evergreen get-update` and use the - downloaded binary.) - -2. List recent mainline versions and pick a revision after 2026-07-06: - - ```bash - curl -s -H "Authorization: Bearer $TOKEN" \ - "$API/projects/mongodb-mongo-master/versions?limit=40" - ``` - -3. macOS arm64 variants are rarely activated on mainline; the reliably-green - arm64 compile variant is `amazon-linux2023-arm64-static-compile`. Fetch its - `archive_dist_test` task (task id pattern - `mongodb_mongo_master_amazon_linux2023_arm64_static_compile_archive_dist_test__`, - timestamp = the version's `create_time`) and take the presigned URL of the - "Binaries" artifact (`mongo-.tgz`, ~1 GB, unpacks to `dist-test/`): - - ```bash - curl -s -H "Authorization: Bearer $TOKEN" "$API/tasks/" # → artifacts[].url - curl -L -o mongodb-binaries.tgz "" - tar -xzf mongodb-binaries.tgz # → dist-test/bin/mongod (aarch64 linux ELF) - ``` - - The binary used for the recorded run is kept at - `~/MongoDB/otel-e2e-server/dist-test/bin/mongod` - (from `mongodb-binaries-ee3a67f8.tgz` in the same directory). - -4. Verify: - - ```bash - docker run --rm -v ~/MongoDB/otel-e2e-server/dist-test:/opt/dist-test \ - otel-poc-mongod:latest /opt/dist-test/bin/mongod --version - # db version v9.0.0-alpha0-ee3a67f8, gitVersion ee3a67f8... - ``` - - `otel-poc-mongod:latest` is a local `amazonlinux:2023` image with the - runtime shared libs the dynamically-linked dist-test mongod needs - (`openldap-compat` for `libldap_r`, cyrus-sasl, krb5, net-snmp, …); its - Dockerfile is at `~/MongoDB/otel-poc-server/Dockerfile`. - -## Step 2: Start mongod with tracing + the OTLP file exporter - -Parameter names confirmed both from source -(`src/mongo/otel/traces/tracing_feature_flags.idl`, `trace_settings.idl`, -`trace_sampling_parameters.idl`) and by a live run — the plan's original -guesses (`featureFlagOtelTracing`, `opentelemetrySamplingRates`) were wrong. -Span creation requires **two** feature flags -(`src/mongo/otel/traces/tracing_enablement.cpp`): `featureFlagTracing` -(default true) AND `featureFlagOtelTraceSampling` (default false — set it). - -Working invocation (Docker; note the host port remap — anything already -listening on host 27017, e.g. a local dev mongod, will otherwise shadow the -container because it binds 127.0.0.1 while Docker binds `*`): - -```bash -mkdir -p /tmp/otel-e2e/db /tmp/otel-e2e/traces - -docker run -d --name otel-e2e-mongod \ - -p 27227:27017 \ - -v ~/MongoDB/otel-e2e-server/dist-test:/opt/dist-test \ - -v /tmp/otel-e2e/db:/data/db \ - -v /tmp/otel-e2e/traces:/data/traces \ - otel-poc-mongod:latest \ - /opt/dist-test/bin/mongod --dbpath /data/db --port 27017 --bind_ip_all \ - --setParameter opentelemetryTraceDirectory=/data/traces \ - --setParameter featureFlagOtelTraceSampling=true \ - --setParameter 'openTelemetryTracingSampling={defaultSampling: {samplingFactor: 1.0}}' -``` - -(For a native binary, drop the Docker wrapper and point -`opentelemetryTraceDirectory` straight at `/tmp/otel-e2e/traces`.) - -Sanity checks: - -```bash -mongosh --quiet --port 27227 --eval ' - print(db.version()); // 9.0.0-alpha0-... - print(db.adminCommand({hello:1}).maxWireVersion); // 29 - printjson(db.adminCommand({getParameter:1, featureFlagOtelTraceSampling:1})); - // -> currentlyEnabled: true -' -``` - -Notes: -- `openTelemetryTracingSampling` takes an `OpenTelemetryTracingSamplingConfig` - BSON document, not a raw double; `samplingFactor: 1.0` samples everything - (the default is 0.000045). -- The client's inbound traceparent is treated as an *externally sampled* - trace, governed by `openTelemetryExternalTracing` (token bucket, defaults - refillRate 1/s, maxTokens 10 — ample for a test run). -- Do NOT delete files under the trace directory while mongod runs: the file - exporter keeps its `.jsonl` open and further exports go to the deleted - inode. Restart the container (or mongod) after cleaning the directory. - -## Step 3: Run the e2e test - -```bash -./gradlew :driver-sync:test --tests 'com.mongodb.client.observability.ServerSpanLinkageProseTest' \ - -Dorg.mongodb.test.uri="mongodb://localhost:27227" \ - -Dorg.mongodb.test.otel.trace.dir=/tmp/otel-e2e/traces -``` - -Expected: PASSED. The test runs an instrumented `find`, then polls the trace -directory for an exported server span whose `traceId` equals the client trace -id and whose `parentSpanId` equals the driver's **command** span id. - -Important semantics: the driver injects the traceparent of the *command* span, -which is created before encoding and passed explicitly to -`CommandMessage.encode(bsonOutput, operationContext, commandSpan)` in -`InternalStreamConnection` (`OperationContext.getTracingSpan()` still returns the -*operation* span and is used only to parent the command span). -The server `find` span is therefore a *direct child* of the client command -span. (An earlier iteration of the driver created the command span after -encoding, so the traceparent carried the operation span instead and the -server span was a sibling of the command span; the prose test was adjusted -accordingly at the time but has since been reverted to assert command-span -linkage now that the command span is created before encoding.) - -Debugging order if it fails: -1. `maxWireVersion >= 29`? If lower, the driver never attaches the section - (`NINE_DOT_ZERO_WIRE_VERSION` gate in `CommandMessage`). Beware of another - local mongod shadowing the port (see Step 2). -2. `featureFlagOtelTraceSampling` currentlyEnabled? -3. Traceparent sampled flag (`-01` suffix) set? Unsampled parents are dropped. -4. mongod log: `BadValue` / `UnknownOpMsgSectionKind` on the telemetry section - would indicate a wire-format mismatch. -5. Inspect the export files directly (Step 4). - -## Step 4: OTLP file export format (verified) - -The exporter writes `mongod---trace.jsonl` in the trace -directory: NDJSON, one single-line `{"resourceSpans":[...]}` blob per batch -export. A span's `traceId`, `spanId`, `parentSpanId` (lowercase hex, no -dashes) all appear on the same line, so the test's line-containment matcher -(`serverSpanLinked`) is valid as written. If a future server switches to -pretty-printed JSON, the matcher must parse whole documents instead. - -Example exported span: - -```json -{"resourceSpans":[{"resource":{...service.name="mongod"...},"scopeSpans":[{"scope":{"name":"mongodb"}, - "spans":[{"name":"find","traceId":"c3778d7ccab7feb73a1e82af44140371", - "spanId":"100d03d5a6795500","parentSpanId":"461e3f3509da712b","kind":1,"flags":1,...}]}]}]} -``` - -## Step 5: Clean up - -```bash -docker rm -f otel-e2e-mongod -rm -rf /tmp/otel-e2e -``` - -Keep the downloaded binary (`~/MongoDB/otel-e2e-server/`) for future runs. - ---- - -## Appendix: optional visual verification via Jaeger - -For interactive confirmation, point the server at an OTLP HTTP collector -instead of the file exporter: - -```bash -docker run --rm -d --name jaeger -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one:latest - - ... \ - --setParameter opentelemetryHttpEndpoint=http://:4318/v1/traces \ - --setParameter featureFlagOtelTraceSampling=true \ - --setParameter 'openTelemetryTracingSampling={defaultSampling: {samplingFactor: 1.0}}' -``` - -Run the same instrumented `find` and check http://localhost:16686 for a single -trace containing the client operation span, the client command span (a child -of the operation span), and the server span (a child of the command span). - -## Notes - -- `CommandMessageOtelTraceContextTest` is the regression guard for the - send-path BSON encoding; `ServerSpanLinkageProseTest` (this runbook) is the - regression guard for full linkage against a real server. -- Findings (parent-span semantics, tracestate handling, flags) feed back into - the spec — in particular the operation-vs-command parent-span semantics - above. diff --git a/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md b/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md deleted file mode 100644 index 537f4cb6fcc..00000000000 --- a/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md +++ /dev/null @@ -1,221 +0,0 @@ -# Design: OpenTelemetry Trace-Context Propagation over OP_MSG - -- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) — Support trace context propagation to the server -- **Status:** SUPERSEDED by `2026-07-13-otel-telemetry-section-reference-impl-design.md` (payload is now a BSON document; gating is by maxWireVersion >= 29, not a hello flag) -- **Author:** Nabil Hachicha -- **Date:** 2026-06-01 -- **Related:** [DRIVERS-719](https://jira.mongodb.org/browse/DRIVERS-719) (client-side OTel tracing), [SERVER-107128](https://jira.mongodb.org/browse/SERVER-107128) (define trace context in OP_MSG), server POC [10gen/mongo#49930](https://github.com/10gen/mongo/pull/49930) - ---- - -## 1. Summary - -DRIVERS-719 added client-side OpenTelemetry spans to the MongoDB drivers, but those -spans stop at the client boundary: the server starts a fresh, disconnected trace. This -work closes the gap by **propagating the client's active trace context to the server on -each wire message**, so the server can start a *child* span and produce one continuous -client → server timeline. - -Two deliverables, sequenced so the POC validates the spec: - -1. **Spec proposal** — defines the wire contract and negotiation mechanism, written so it - can become a `mongodb/specifications` PR shepherded through the DRIVERS process. -2. **Java driver POC** — a minimal, `driver-sync`-only implementation that exercises every - claim in the spec. Ambiguities in the spec should surface as failing tests or awkward - code; findings feed back into the spec. The POC is internal/throwaway quality and - touches no public API. - ---- - -## 2. Background & constraints - -- **Why transport layer, not command BSON.** Per SERVER-107128, the trace context is - carried in an OP_MSG *section* rather than inside the command document, so the server - can read it before parsing the command — enabling tracing of command-parse failures and - early network handling. -- **The server POC is a prototype.** PR #49930 is explicitly throwaway ("USING AI DO NOT - COPY"), intra-server focused, and unconditionally accepts the new section. It is the - authoritative reference for the **wire format only**, not for negotiation or production - behavior. -- **Process.** Spec changes must flow through DRIVERS tickets; the drivers team shepherds - them. This document is the basis for that proposal. -- **Status of dependencies (June 2026).** No OP_MSG-propagation spec exists yet. - SERVER-107128 is Open/Unassigned. The server `hello` capability flag described below is - **not** in the POC and must be added server-side for the negotiation to work end to end. - ---- - -## 3. Wire contract - -### 3.1 OP_MSG section - -A new OP_MSG section kind is defined: - -| Kind | Name | Status | -|------|-----------------------|------------| -| 0 | Body | existing | -| 1 | Document Sequence | existing | -| 2 | Security Token | existing | -| **3**| **OTel Trace Context**| **new** | - -This matches `kOtelTelemetryContext = 3` in the server POC -(`src/mongo/rpc/op_msg.cpp`). - -### 3.2 Payload format - -The section payload is a single **null-terminated C-string** containing a **W3C -`traceparent`** value: - -``` -00-<32 hex trace-id>-<16 hex span-id>-<2 hex trace-flags>[-] -``` - -- `00` — W3C version. -- trace-id — 16 bytes, 32 lowercase hex chars. -- span-id — 8 bytes, 16 lowercase hex chars (the client span that created the RPC). -- trace-flags — 1 byte, 2 hex chars (e.g. `01` = sampled). -- Base length is **55 chars**; an optional `-` suffix may follow - (server POC appends `span_context.trace_state()->ToHeader()`). - -The string is encoded with the BSON CString convention (UTF-8 bytes + trailing `\0`), -consistent with how the server reads it via `readCStr()`. - -### 3.3 Direction & optionality - -- **Request-only.** No trace data is added to responses; server spans are exported - independently via the OTel pipeline and correlated by trace-id. -- **Sparse.** The section is present only when the client has an **active, sampled** span. - When there is no span, or the span is not sampled, the section is omitted entirely. -- Server behavior (informational): if a trace context is present, the server starts a span - subject to its own external tracing rate limiter. - ---- - -## 4. Negotiation (the key addition over the POC) - -Older or mixed-version servers `uassert(40432)` ("Unknown section kind") on an -unrecognized OP_MSG section. The driver therefore must **not** send section kind 3 to a -server that does not understand it. - -**Mechanism:** the server advertises support in its `hello` response: - -``` -{ ..., "tracingSupport": true } -``` - -Rules: - -- The driver reads `tracingSupport` from each `hello`/handshake response and stores it as a - per-connection / per-server capability (default **false** when absent). -- The driver sends the section **only** on connections whose server advertised - `tracingSupport: true`. -- The capability is re-evaluated on reconnect/failover; a server that stops advertising it - (downgrade) stops receiving the section. - -> **Open question / dependency.** This flag is not in server POC #49930. SERVER-107128 must -> add it. Until then, the POC validates the section via a controlled negotiation -> (see §6). An alternative considered was gating on a `maxWireVersion` bump; rejected for -> the spec because it couples the feature to a wire-version release and is coarser than an -> explicit capability. To be confirmed with the server team. - ---- - -## 5. Edge cases (spec-level, noted; not all in POC scope) - -- **Invalid/empty traceparent.** If the driver cannot produce a valid 55-char traceparent, - it omits the section (never sends a malformed one). The server treats a sub-55-char or - malformed value as absent. -- **tracestate size.** Pass-through only in the POC; the spec should reference W3C limits - (512 chars) and define truncation/drop behavior — to be finalized with the server team. -- **Compression / OP_COMPRESSED.** The section is part of the OP_MSG body that gets - compressed; no special handling expected, but the spec calls this out for confirmation. -- **mongos / load-balanced passthrough.** Out of scope for this driver work; the server is - responsible for forwarding context to downstream services (noted only). - ---- - -## 6. Java POC (driver-sync only) - -Four focused changes in `driver-core`. All internal; no public API change. - -### 6.1 Expose the trace context - -`com.mongodb.internal.observability.micrometer.TraceContext` is currently an empty marker -interface, and `Span.context()` returns it. Extend it minimally: - -```java -public interface TraceContext { - TraceContext EMPTY = new TraceContext() { ... }; - @Nullable String traceParent(); // W3C traceparent, or null if unavailable/unsampled -} -``` - -- Implement in `MicrometerTraceContext`/`MicrometerSpan` by reading `traceId()`, - `spanId()`, and `sampled()` from the underlying Micrometer `io.micrometer.tracing.TraceContext` - and formatting the `00-…` string. Return `null` when the span is no-op or not sampled. -- `TraceContext.EMPTY` and `Span.EMPTY` return `null`. - -### 6.2 Read the capability - -- In `DescriptionHelper`, parse `tracingSupport` from the `hello` result. -- Store on `ServerDescription` (and `ConnectionDescription` as needed) with an - `isTracingSupport()` accessor, following the existing `helloOk` / `maxWireVersion` - patterns. - -### 6.3 Inject the section - -- In `CommandMessage`, add `PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT = 3`. -- In `writeOpMsg()`, after the body/sequence sections, write the new section **iff**: - 1. the connection's server advertised `tracingSupport`, **and** - 2. the active operation `Span` yields a non-null (sampled) `traceParent()`. -- The send path (`InternalStreamConnection.sendAndReceiveInternal`) already has both the - `Span` and the connection description; thread the capability + traceparent into - `CommandMessage` encoding via the existing plumbing. - -### 6.4 Validation — phased - -- **Phase 1 (in-repo, automated — the real proof):** - - Positive: build the OP_MSG bytes for a command with an active sampled span on a - tracing-capable connection; re-parse the sections and assert section kind 3 is present - and its traceparent round-trips and matches the span's trace-id/span-id. - - Negative: capability absent ⇒ no section; no/unsampled span ⇒ no section. -- **Phase 2 (optional, manual runbook — not CI):** build the server POC branch (#49930) - locally with an OTel collector, point the sync driver at it, and confirm the server - emits a child span linked to the client trace. - -### 6.5 Scope guards (YAGNI) - -Explicitly **out of scope** for the POC: reactive/async send path, any public API, -tracestate handling beyond pass-through, sampling rework, and `mongos`/load-balanced -propagation. - ---- - -## 7. Testing summary - -- Unit tests for traceparent formatting (`MicrometerTraceContext.traceParent()`), including - unsampled and no-op cases. -- Unit test for `tracingSupport` parsing in `DescriptionHelper`. -- OP_MSG encode/parse round-trip tests in `CommandMessage` (positive + negative), per §6.4. -- All new code follows `driver-core` conventions (Java 8 baseline, SLF4J, copyright header, - `@Nullable`/`@NonNull`). No reduction in coverage. - ---- - -## 8. Deliverable order - -1. Write & socialize this spec proposal (basis for the DRIVERS spec PR + SERVER-107128 - coordination on the `hello` flag). -2. Build the Java `driver-sync` POC (§6) and run Phase 1 validation. -3. Feed POC findings back into the spec; optionally run Phase 2 end-to-end. - ---- - -## 9. Open questions - -- Server `hello` `tracingSupport` flag ownership/timing (SERVER-107128 is unassigned). -- Final `tracestate` size/truncation policy. -- Whether this section should be designed as a special case or as the first user of the - broader "client-side telemetry section" effort (retry metadata + OTel + client config). -- Confirmation that the W3C `traceparent` (vs. a BSON-structured payload) is the final - on-wire encoding the server team will commit to. diff --git a/docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md b/docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md deleted file mode 100644 index 7bd19fe5562..00000000000 --- a/docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md +++ /dev/null @@ -1,218 +0,0 @@ -# Design: End-to-end OTel trace visualization (driver toggle + Spring Boot + Jaeger) - -- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) — trace context propagation to the server -- **Status:** Design -- **Author:** Nabil Hachicha -- **Date:** 2026-06-02 -- **Builds on:** `docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md` (the OP_MSG kind-3 POC, already staged) and the running server POC (`~/MongoDB/otel-poc-server/`, mongod `gitVersion b2cb2bf`). - ---- - -## 1. Goal - -Make a client→server trace **visible end to end in Jaeger**: a Spring Boot app issues a MongoDB -operation; the driver propagates its sampled W3C `traceparent` to the server via the OP_MSG kind-3 -section; the server starts a child span; both client and server spans export to one Jaeger instance -and appear under a single trace. - -The POC server does **not** advertise `tracingSupport` in `hello`, so the driver (which gates on -that capability) needs a temporary, removable switch to send the section anyway. - -## 2. Components - -1. **Driver test-only force toggle** — bypasses the server-capability gate only. -2. **Local Maven publish** — `5.9.0-SNAPSHOT` consumable by the app. -3. **Jaeger** (Docker) — single OTLP collector + UI. -4. **mongod re-config** — export server spans to Jaeger. -5. **Spring Boot app** (host JVM) — Spring Data MongoDB + Micrometer→OTel→Jaeger, wired to the - driver's internal tracing, with a REST trigger. - ---- - -## 3. Driver: force-propagation toggle - -New file `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java`: - -```java -public final class OtelTracePropagationTestToggle { - // TEST-ONLY (DRIVERS-3454): force sending the OP_MSG OTel trace-context section even when the - // server did not advertise tracingSupport in hello. Remove before any production use. - public static volatile boolean FORCE_PROPAGATION = false; - - private OtelTracePropagationTestToggle() { - } -} -``` - -`CommandMessage.writeOtelTraceContextSection` gate changes from: - -```java -if (!getSettings().isTracingSupported()) { - return; -} -``` - -to: - -```java -if (!getSettings().isTracingSupported() && !OtelTracePropagationTestToggle.FORCE_PROPAGATION) { - return; -} -``` - -The remaining gates are unchanged: a `null` tracing span or a `null`/unsampled `traceParent()` still -omits the section. So the toggle only relaxes the capability check. - -**Test** (`CommandMessageOtelTraceContextTest`): new case — `tracingSupported=false` + -`FORCE_PROPAGATION=true` + sampled span ⇒ section IS written. Set/reset `FORCE_PROPAGATION` in a -`try/finally` so the static does not leak into other tests. - -**Removal path:** delete the toggle class and revert the one `&&` clause. - -## 4. Publish to Maven Local - -From the repo root: - -```bash -./gradlew publishToMavenLocal -PskipCryptVerify=true -``` - -Publishes all modules (`bson`, `bson-record-codec`, `driver-core`, `driver-sync`, …) as -`org.mongodb:*:5.9.0-SNAPSHOT`. The app consumes `5.9.0-SNAPSHOT` from `mavenLocal()`. - -## 5. Jaeger (Docker) - -`jaegertracing/all-in-one:1.62.0` on a dedicated network `otel-poc-net`, OTLP enabled, host ports: - -| Port | Purpose | -|------|---------| -| 16686 | Jaeger UI | -| 4317 | OTLP gRPC | -| 4318 | OTLP HTTP | - -Run: - -```bash -docker network create otel-poc-net 2>/dev/null || true -docker run -d --name jaeger --network otel-poc-net \ - -e COLLECTOR_OTLP_ENABLED=true \ - -p 16686:16686 -p 4317:4317 -p 4318:4318 \ - jaegertracing/all-in-one:1.62.0 -``` - -## 6. mongod: export server spans to Jaeger - -Restart the existing POC mongod container **on `otel-poc-net`** so it can reach Jaeger by name, and -point its OTLP exporter at Jaeger: - -```bash -docker rm -f otel-poc-mongod-run -docker run -d --name otel-poc-mongod-run --platform linux/arm64 --network otel-poc-net \ - -v "$HOME/MongoDB/otel-poc-server/dist-test:/opt/mongo:ro" \ - -v "$HOME/MongoDB/otel-poc-server/data:/data/db" \ - -p 27017:27017 \ - otel-poc-mongod \ - /opt/mongo/bin/mongod --dbpath /data/db --bind_ip_all --port 27017 \ - --setParameter opentelemetryHttpEndpoint=http://jaeger:4318/v1/traces \ - --setParameter openTelemetryExportIntervalMillis=1000 -``` - -`featureFlagTracing` is already enabled in this build. **Verification & fallback:** confirm via -server logs that the OTLP endpoint is accepted and exports succeed; if the exact endpoint form is -rejected, fall back to the already-working file exporter -(`opentelemetryTraceDirectory=/data/db/otel-traces`) and note that server spans are then inspected as -JSONL rather than in the Jaeger UI. - -## 7. Spring Boot app (host JVM) - -Maven project at `~/MongoDB/otel-poc-server/otel-poc-client/` — kept outside the driver repo so it is -not entangled with the driver build. - -**Stack:** Spring Boot 3.3.x, Java 17, Maven wrapper. - -**Dependencies:** `spring-boot-starter-web`, `spring-boot-starter-data-mongodb`, -`spring-boot-starter-actuator`, `micrometer-tracing-bridge-otel`, `opentelemetry-exporter-otlp`. -`pom.xml` adds `mavenLocal()` (via a ``) and overrides `5.9.0-SNAPSHOT`. - -**Critical wiring bean** — activates the driver's *internal* tracing (the path that sets -`operationContext.tracingSpan`, which `traceParent()` reads): - -```java -@Bean -MongoClientSettingsBuilderCustomizer tracingCustomizer(ObservationRegistry registry) { - return builder -> builder.observabilitySettings( - MicrometerObservabilitySettings.builder() - .observationRegistry(registry) - .build()); -} -``` - -**Toggle activation** — set before any operation runs: - -```java -@PostConstruct -void enableForcedPropagation() { - OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; -} -``` - -**Trigger** — a `@RestController`: - -```java -@GetMapping("/ping") -String ping() { - mongoTemplate.getCollection("ping").insertOne(new Document("at", new Date())); - long n = mongoTemplate.getCollection("ping").countDocuments(); - return "ok, count=" + n; -} -``` - -Hitting `/ping` creates an HTTP server span (root, sampled), under which the driver creates a mongo -command span (exported to Jaeger; its context is the propagated `traceparent`), and the server -creates its child span (exported to Jaeger). - -**`application.properties`:** - -```properties -spring.application.name=otel-poc-client -spring.data.mongodb.uri=mongodb://localhost:27017/test -management.tracing.sampling.probability=1.0 -management.otlp.tracing.endpoint=http://localhost:4318/v1/traces -# Suppress Spring's auto Mongo command instrumentation so the driver's internal tracer is the -# ONLY source of Mongo command spans (avoids duplicate/competing spans). -management.metrics.enable.mongodb=false -spring.autoconfigure.exclude=org.springframework.boot.actuate.autoconfigure.metrics.mongo.MongoMetricsAutoConfiguration -``` - -**Suppressing Spring's auto Mongo instrumentation (firm decision).** Spring Boot's actuator -auto-registers a Mongo command listener via `MongoMetricsAutoConfiguration`. To guarantee the -driver's own internal tracer is the single source of Mongo command spans (and the one performing -propagation), we **exclude** `org.springframework.boot.actuate.autoconfigure.metrics.mongo.MongoMetricsAutoConfiguration` -and set `management.metrics.enable.mongodb=false`. Only our `MongoClientSettingsBuilderCustomizer` -(§7 wiring) then contributes Mongo observability. - -## 8. End-to-end run & verification - -1. Start Jaeger (§5). 2. Restart mongod on the network with OTLP (§6). 3. Publish the driver (§4). -4. `./mvnw spring-boot:run` (§7). 5. `curl localhost:8080/ping`. 6. Open **http://localhost:16686**, -select service `otel-poc-client`, open the latest trace. - -**Pass criteria:** one trace contains the HTTP span, the driver mongo command span, **and** a -`mongod` server span; the server span's `traceId` equals the client trace, and its parent is the -client mongo span's id. Negative check: with `FORCE_PROPAGATION=false`, the server span is absent -from the client trace (server starts its own unrelated trace, if any). - -## 9. Scope guards - -- Test-only toggle; not a public API; removed before any real merge. -- Single mongod (no replica set/sharding), `find`/`insert` only. -- No auth/TLS on the local mongod. -- The app lives outside the driver repo; it is not committed to the driver repo. - -## 10. Open questions / risks - -- Exact accepted form of `opentelemetryHttpEndpoint` (full `/v1/traces` URL vs base) — verify via - logs; file-exporter fallback documented (§6). -- (Resolved) Spring's auto Mongo instrumentation is suppressed via autoconfigure-exclude + metric disable (§7). -- Spring Boot 3.3.x pins an older driver; overriding `mongodb.version` to `5.9.0-SNAPSHOT` assumes API - compatibility (it is, the API is unchanged by this POC). diff --git a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md deleted file mode 100644 index 34ef2c488a5..00000000000 --- a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md +++ /dev/null @@ -1,43 +0,0 @@ -# Trace-Context Propagation Prose Tests (DRIVERS-3454) - -These tests complement the OpenTelemetry spec prose tests. Tests 1–4 are unit-level and -MUST NOT require a server. Test 5 requires a MongoDB 9.0+ server (maxWireVersion >= 29) -started with OTel tracing enabled and the OTLP file exporter -(`--setParameter opentelemetryTraceDirectory=` plus the tracing feature flag and -sampling parameters), running on the same host as the test. - -## 1. Telemetry section is attached when supported and traced - -With tracing enabled and an active operation span, encode a command targeting a -connection whose `maxWireVersion` is >= 29. Assert the resulting OP_MSG contains exactly -one section of kind 3 whose payload is a BSON document of the form -`{otel: {traceparent: }}`, where `traceparent` is 55 characters, -`00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>`, and the trace-id/span-id match -the **command span**'s context. - -## 2. Telemetry section is omitted for older servers - -Same as (1) but with `maxWireVersion` < 29. Assert no section of kind 3 is present. - -## 3. Telemetry section is omitted without an active span - -With tracing disabled (or no active span, e.g. monitoring/auth commands), encode a -command at `maxWireVersion` >= 29. Assert no section of kind 3 is present. - -## 4. Malformed trace context is never sent - -With an active span whose context cannot produce a valid W3C traceparent (zero trace-id, -zero span-id, wrong-length or non-lowercase-hex ids), assert no section of kind 3 is -present (drivers MUST omit the section rather than send an invalid traceparent). - -## 5. End-to-end server span linkage - -With tracing enabled, run a CRUD operation (e.g. `find`) against the configured 9.0+ -server. Capture the driver's finished spans (client side). Then read the server's -exported OTLP JSON from the trace directory and assert a server span exists whose -`traceId` equals the client trace-id and whose `parentSpanId` equals the client -**command** span's span-id (the traceparent is that of the command span, passed to -`CommandMessage.encode(...)` per the 2026-07-18 command-span propagation design; -`OperationContext.getTracingSpan()` still returns the operation span, used only for -parenting the command span — so the server span is a direct child of the client command -span). Allow for the server's batch export interval when polling. diff --git a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md b/docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md deleted file mode 100644 index dacaf0358d8..00000000000 --- a/docs/superpowers/specs/2026-07-13-otel-telemetry-section-reference-impl-design.md +++ /dev/null @@ -1,171 +0,0 @@ -# Design: DRIVERS-3454 Reference Implementation — OTel Trace-Context Propagation (BSON telemetry section) - -- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) — Support trace context propagation to the server -- **Status:** Approved design (reference implementation) -- **Author:** Nabil Hachicha -- **Date:** 2026-07-13 -- **Supersedes:** `2026-06-01-otel-opmsg-propagation-design.md` (POC design; wire contract and negotiation have since changed) -- **Related:** [DRIVERS-719](https://jira.mongodb.org/browse/DRIVERS-719) (client-side OTel tracing), - [SERVER-107128](https://jira.mongodb.org/browse/SERVER-107128), - server implementation [10gen/mongo#56646](https://github.com/10gen/mongo/pull/56646) (merged 2026-07-06), - [Technical Design: Drivers OTEL Trace Context Propagation to the Server](https://docs.google.com/document/d/1mLFw3yvLW3GPq8GLU82tNIHmsNiS5UvDmMJ4FjB80R4/edit) - ---- - -## 1. Summary - -Turn the existing POC on this branch into the production-quality **reference implementation** -for propagating the client's OTel trace context to the server, matching the final -driver/server contract that the server has now shipped (10gen/mongo#56646). The final -contract differs from the POC in three material ways: - -1. **Payload is a BSON document**, not a bare traceparent C-string: - `{ otel: { traceparent: "<55-char W3C traceparent>" } }`. -2. **Gating is by `maxWireVersion >= 29`** (server 9.0), not a `hello` capability flag. - The hello-flag approach was explicitly rejected (meeting 2026-06-16, Nabil/Jeff/Didier). -3. The server **strictly validates** the traceparent and enforces a 4 KB section size cap. - -Deliverables: production internal implementation in `driver-core` (shared by sync and -reactive), prose-test definitions upstreamable to the future DRIVERS spec PR, unit tests, -and an automated end-to-end test that asserts server-span linkage via the server's OTLP -file exporter. - -## 2. Server contract (authoritative, from merged PR #56646) - -- OP_MSG section kind **3** (`kTelemetry`), payload is a single BSON document. -- Schema (IDL `TelemetryContextSection`, `strict: false` at both levels — unknown fields - ignored): - - ``` - { otel: { traceparent: } } - ``` - -- Max section size **4096 bytes** (`kMaxTelemetrySectionSize`); larger payloads are - rejected with `BSONObjectTooLarge`. -- `traceparent` is validated by `otel::traces::validateW3CTraceparent`: - - exactly **55 characters**: `"---"`; - - lowercase hex only; `-` delimiters at fixed positions; - - trace-id and parent-id must **not** be all zeroes; - - version `ff` is forbidden; - - **no tracestate suffix is accepted** (a >55-char value is rejected — this differs from - the earlier server POC #49930, which appended tracestate). -- Multiple telemetry sections in one message are rejected. -- The server writes the section before the body when serializing, but the **parser is - order-agnostic**; the driver may write it after the body. -- Consumption: `service_entry_point_shard_role.cpp` starts server spans from the received - context; mongos forwards it on downstream remote calls. -- Server spans are exported via startup parameters - `opentelemetryTraceDirectory` (OTLP JSON files) or `opentelemetryHttpEndpoint` - (OTLP/HTTP), subject to the server tracing feature flag and sampling parameters. - -## 3. Driver-side design - -### 3.1 Gating rule - -Attach the telemetry section to an outgoing OP_MSG **iff**: - -1. `MessageSettings.getMaxWireVersion() >= ServerVersionHelper.NINE_DOT_ZERO_WIRE_VERSION` - (new constant, value **29**, matching `WIRE_VERSION_90 = 29` on the server); and -2. the `OperationContext` carries an active tracing `Span` whose context yields a - **valid** W3C traceparent (non-null). - -Tracing-disabled clients, monitoring connections, and auth commands never have an active -span (per the DRIVERS-719 implementation), so those exclusions fall out of condition 2 -without extra plumbing. `MessageSettings` already carries `maxWireVersion`, so no new -capability plumbing is needed. - -> **2026-07-18 amendment:** condition 2 now requires the **command span** rather than the -> operation span — see `2026-07-18-command-span-propagation-design.md` for the rationale -> and the updated spec text. - -### 3.2 Wire encoding (`CommandMessage`) - -In `writeOpMsg()`, after the body and any document-sequence sections, when the gating rule -passes, write: - -- one byte `PAYLOAD_TYPE_3_TELEMETRY = 3`; -- the BSON document `{ otel: { traceparent: "" } }` (raw BSON, no length prefix - beyond the document's own). - -Placement after the body is kept (simpler for our encoder; server parser is -order-agnostic). The existing POC change to `getCommandDocument()` (tolerating a trailing -non-sequence section when re-parsing our own message) is retained and adapted. - -### 3.3 Traceparent production (`MicrometerTracer` / `TraceContext`) - -`TraceContext.traceParent()` remains the internal accessor, tightened to guarantee it -never returns a value the server would reject: - -- format `00-<32 lowercase hex>-<16 lowercase hex>-<02 flags>`, exactly 55 chars; -- return `null` for no-op spans, zero trace-id or span-id, or any malformed component - (section is then omitted entirely — the driver never sends a malformed traceparent); -- unsampled contexts are propagated with flags `00` (the server applies its own - sampling); no tracestate is ever appended. - -### 3.4 Removals (POC artifacts) - -- `OtelTracePropagationTestToggle` (force-send toggle) — deleted. -- `MessageSettings.tracingSupported` and its builder — deleted. -- `ConnectionDescription` / `DescriptionHelper` `tracingSupport` hello parsing — reverted. -- POC-era tests reworked to the new contract. - -No public API changes; everything stays under `com.mongodb.internal.*`. Sync and reactive -drivers both inherit the behavior through the shared `driver-core` send path. - -## 4. Testing - -### 4.1 Unit tests (run everywhere) - -- **Traceparent formatting matrix** (`MicrometerTracer`): valid sampled/unsampled spans, - no-op span, zero ids, length exactly 55, lowercase hex. -- **`CommandMessage` OP_MSG round-trip** (rework of `CommandMessageOtelTraceContextTest`): - - positive: wire version ≥ 29 + active span ⇒ section kind 3 present; payload BSON - parses to `{otel: {traceparent: ...}}` matching the span's trace-id/span-id; - - negative: wire version < 29 ⇒ no section; no span ⇒ no section; invalid traceparent ⇒ - no section; interaction with document-sequence sections preserved. - -### 4.2 Prose tests (upstreamable) - -A markdown prose-test definition (structured for the future `mongodb/specifications` -tracing spec PR) covering the gating matrix and the e2e linkage test, plus Java -implementations following the `AbstractMicrometerProseTest` pattern. - -### 4.3 End-to-end server-span linkage test - -- New integration test (sync, `AbstractMicrometerProseTest`-style) using - `InMemoryOtelSetup` to capture the client command span (expected trace-id/span-id). -- Enabled only when `-Dorg.mongodb.test.otel.trace.dir=` is set; skipped otherwise. -- Requires the test's MongoDB deployment (same host, as with mongo-orchestration in CI) to - be a **server 9.0 build started with** - `--setParameter opentelemetryTraceDirectory=` plus the tracing feature flag and - sampling parameters. -- The test runs a traced CRUD operation, polls the directory past the export interval - (`openTelemetryTracingBatchExportIntervalMillis`, default 1 s), parses the OTLP JSON - span files, and asserts a server span exists with `traceId` equal to the client span's - trace-id and `parentSpanId` equal to the client command span's span-id. -- The local runbook (`docs/superpowers/runbooks/otel-opmsg-e2e.md`) is updated for the - Evergreen 9.0 artifact and file-exporter workflow (Jaeger flow kept as an optional - visual check). - -## 5. Branch strategy - -Merge `origin/main` into `nabil_otel_context` (currently 11 commits behind), keeping POC -history; implement the production version as new, logically separated commits. Remove -unrelated local noise (e.g. `prompt.txt`, stray modified test files) before starting. - -## 6. Out of scope - -- `baggage` / `tracestate` propagation (future payload evolution; requires wire bump). -- Unified test format runner extensions. -- Evergreen YAML task wiring for the e2e test (follow-up). -- Public API surface changes and mongos/load-balancer forwarding (server-side concern). - -## 7. Risks / open items - -- **Wire version N = 29** matches server master today; if the final DRIVERS spec picks a - different gate before 9.0 GA, only the `ServerVersionHelper` constant changes. -- The server deploys parsing behind a feature flag (design Decision 5); a 9.0 server with - the flag off still parses the section (parse path is unconditional in #56646), so the - driver needs no flag awareness. -- Intermediaries (Atlas Proxy, Envoy, mongobetween) compatibility is tracked server-side - under SERVER-128017; not a driver concern for this implementation. diff --git a/docs/superpowers/specs/2026-07-18-command-span-propagation-design.md b/docs/superpowers/specs/2026-07-18-command-span-propagation-design.md deleted file mode 100644 index d2349260535..00000000000 --- a/docs/superpowers/specs/2026-07-18-command-span-propagation-design.md +++ /dev/null @@ -1,85 +0,0 @@ -# Design: Propagate the Command Span's Trace Context (Option A) - -- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) -- **Status:** Approved design (chosen in discussion 2026-07-18; "split creation from decoration" variant) -- **Author:** Nabil Hachicha -- **Date:** 2026-07-18 -- **Amends:** `2026-07-13-otel-telemetry-section-reference-impl-design.md` (which recorded operation-span - propagation as a limitation) and the spec branch `DRIVERS-3454` in `testing/resources/specifications`. - -## Problem - -The telemetry section currently propagates the **operation** span's traceparent because -`InternalStreamConnection.sendAndReceiveInternal` encodes the message (`:445`) before creating the command span -(`:448`), and `TracingManager.createTracingSpan` eagerly re-parses the **encoded** bytes -(`commandDocumentSupplier.get()` → `message.getCommandDocument(bsonOutput)`) just to obtain the command name, -sensitive-command verdict, and `getMore` cursor id. Server spans therefore parent to the operation span — siblings -of the command span instead of children — losing per-command / per-retry attribution. - -## Key insight - -All *eager* inputs of `createTracingSpan` are available from the **raw, un-encoded** command document that -`CommandMessage` already stores (`CommandMessage.java:96`): first key = command name; sensitive check needs only -the name; `getMore` cursor id is a field of the raw document. The only encode-dependent input, the folded command -document for `db.query.text`, is already consumed lazily *after* span creation (`setQueryText`). So span creation -can move before `encode()` with no loss. - -In the Micrometer Observation model, span-id allocation and observation start are the same event, so -"pre-allocating ids" and "starting the span earlier" are the same change; decoration (attributes, scope, query -text) stays where it is today. - -## Changes (driver-core, internal only) - -1. **`TracingManager.createTracingSpan`** — replace the `Supplier commandDocumentSupplier` - parameter with the raw command document (or have the method read it from `message`). Body otherwise unchanged - (name, sensitive check, namespace resolution, observation-context population, `getMore` cursor id, session - fields). -2. **`CommandMessage`** — expose the raw command document (package-private accessor) for (1); add an `encode` - overload `encode(ByteBufferBsonOutput, OperationContext, @Nullable Span commandSpan)`. The base - `RequestMessage.encode(bsonOutput, operationContext)` signature is untouched (`CompressedMessage` and other - message types keep it). -3. **`writeTelemetryContextSection`** — read the traceparent from the passed `commandSpan` instead of - `operationContext.getTracingSpan()`. Null command span (tracing disabled, sensitive command, monitoring/auth) - ⇒ section omitted. The operation span is still used for *parenting* the command span in `TracingManager`. -4. **`InternalStreamConnection`** — in both the sync (`:445`) and async (`:620`) paths: create the command span - *before* `encode()`, pass it to the new overload, and wrap `encode` so that an encode failure ends the span - with error status (today encode failures happen span-less). The one-way `send` path (`:508`) has no command - span today; it passes `null` (no section — matches current behavior since w:0 fire-and-forget commands carry - no command span). - -## Behavior changes (all intentional) - -- Server spans become children of the **command** span. Each retry attempt / each `getMore` gets its own command - span, so server spans attribute to the exact wire command that caused them. -- The command span's duration now includes BSON serialization (unavoidable: ids ⇒ started; and defensible — - serialization is work done for that command). Microseconds against a network round trip. -- Security-sensitive commands can never carry a telemetry section (null command span), even where an operation - span exists — strictly safer than today. - -## Spec/doc updates (same effort) - -- **Spec branch `DRIVERS-3454`** (`testing/resources/specifications`, `source/open-telemetry/open-telemetry.md`): - flip the normative text from operation span to **command span** ("the propagated context MUST be that of the - command span; server spans join the trace as children of the command span"), and drop the encode-ordering - parenthetical. Changelog wording stays under the same 2026-07-18 entry (branch not yet merged). -- **Prose-test definitions** (`docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md`): test 1 - and test 5 assert the *command* span's ids. -- **E2E test** (`ServerSpanLinkageProseTest`): assert `parentSpanId == command span id` - (`getFinishedSpans().get(0)`, per `AbstractMicrometerProseTest` ordering). -- **Unit tests**: `CommandMessageOtelTraceContextTest` passes the span via the new `encode` overload; add a case - that a null command span omits the section even when an operation span exists. - -## Testing - -- Rework unit tests as above; run `:driver-core:test` affected suites. -- Micrometer prose tests (`AbstractMicrometerProseTest`) must stay green (span counts/names unchanged; only start - timing moves). -- Re-run the e2e server-span linkage test against the local 9.0 Docker server (runbook) and confirm the server - span's `parentSpanId` equals the **command** span id. - -## Out of scope - -- `Tracer.nextSpan()`-based true id pre-allocation (requires abandoning the Observation pipeline / new public - API). -- Option B (append-section-after-encode with header backpatch). -- Any public API change. diff --git a/docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md b/docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md deleted file mode 100644 index 81bda87c548..00000000000 --- a/docs/superpowers/specs/2026-07-18-drivers-3454-spec-change-design.md +++ /dev/null @@ -1,109 +0,0 @@ -# Design: DRIVERS-3454 Minimal Spec Change (mongodb/specifications) - -- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) -- **Status:** Approved design -- **Author:** Nabil Hachicha -- **Date:** 2026-07-18 -- **Based on:** the reference implementation on branch `nabil_otel_context` (see - `2026-07-13-otel-telemetry-section-reference-impl-design.md`) and the shipped server contract - ([10gen/mongo#56646](https://github.com/10gen/mongo/pull/56646)). - -## Goal - -Draft the minimal normative spec change reflecting DRIVERS-3454 (OTel trace-context propagation to the server), -split across the two specs that own each layer, in the `mongodb/specifications` repo (worked on locally in the -`testing/resources/specifications` submodule). - -## Where the work happens - -- Branch **`DRIVERS-3454`** in `testing/resources/specifications`, based on **`origin/master`** (the repo's default - branch; there is no `main`). -- Local only; nothing pushed. The parent repo's submodule pointer is NOT updated (the parent repo treats the - submodule as do-not-modify; this work is on an explicit user instruction and stays on a local branch). -- Conventions per the repo's `AGENTS.md`: GitHub Flavored Markdown, 120-character line width, MongoDB docs style, - dated changelog entries prepended inside each spec's `## Changelog`. - -## Change 1 — `source/message/OP_MSG.md` (wire layer) - -1. Extend the `Section` struct's union: - - ```c - document telemetry; // payloadType == 3 - ``` - -2. Add normative rules next to the existing payload-type rules: - - An `OP_MSG` MAY contain **at most one** section with Payload Type 3. - - Payload Type 3 is **request-only**: drivers MAY send it; it MUST NOT appear in server replies. - - Its payload is a single BSON document whose contents are defined by the - [OpenTelemetry specification](../open-telemetry/open-telemetry.md) (cross-link). - - Receivers that do not recognize a section kind fail the message (existing behavior), therefore senders MUST - gate emission on server support as defined by the OpenTelemetry specification. -3. Include a brief explanatory line on numbering: **Payload Type 2 is reserved for server-internal use** (a - security-token section populated by the server/infrastructure components, never by drivers), so the next - available payload type for driver-emitted content is 3. Kind 2 itself remains undocumented here (status quo, - out of scope). -4. Changelog entry dated 2026-07-18. - -## Change 2 — `source/open-telemetry/open-telemetry.md` (driver behavior) - -1. New subsection under Implementation Requirements: **"Propagating Trace Context to the Server"**: - - Payload schema: `{ otel: { traceparent: } }`, where `traceparent` is the W3C traceparent value of the - **operation span** (stated explicitly: drivers attach the context at message-encode time, before the command - span exists; server spans therefore join the trace as children of the operation span). - - Drivers attach the section **iff**: tracing is enabled, the connection's `maxWireVersion >= 29` (MongoDB 9.0), - and a valid traceparent exists. - - Validity mirrors server-side enforcement: exactly 55 characters, - `00-<32 lowercase hex>-<16 lowercase hex>-<2 hex flags>`, non-zero trace-id and parent-id. Drivers MUST omit - the section rather than send an invalid value; MUST NOT append `tracestate`; unsampled contexts are propagated - with trace-flags `00`. - - At most one telemetry section per message. Connections without an active span (monitoring, authentication) - naturally send no section (informative note). -2. **Design Rationale** additions (brief): - - Wire-version gate instead of a `hello` capability flag: the wire protocol version acts as the schema contract - for the payload; future fields require a wire-version bump (decision of 2026-06-16). - - BSON document instead of a bare traceparent string: extensibility without redesigning the payload format. -3. Changelog entry dated 2026-07-18. -4. **No Future Work addition** (explicitly out of scope per review). - -## Out of scope - -- Test-plan / prose-test additions (`tests/README.md`) — normative text only; tests follow in a later PR. -- Unified test format schema changes; YAML/JSON test files (so no `make -C source` regeneration needed). -- Documenting Payload Type 2 beyond the one-line reservation note. -- Pushing the branch or updating the parent repo's submodule pointer. - -## Validation - -- Run available repo checks locally (`pre-commit run --files ` and/or `mkdocs build --strict`) if the - tooling is installed; otherwise verify 120-char width and markdown lint manually and note it. - ---- - -## Decision record (2026-07-21): spec prose test — chosen candidate, deferred - -**Decision:** the spec PR remains normative-text-only for now; no prose test was added to -`source/open-telemetry/tests/README.md`. When the DRIVERS review asks for a test plan (the PR template expects -one), the agreed candidate is a single appended prose test (*Test 3*, numbering is load-bearing — existing tests -1–2 must not be renumbered): - -> **Test 3: Trace Context Propagation Does Not Affect Command Execution.** With tracing enabled, run CRUD -> operations across the standard server-version/topology matrix, on both sides of the wire-version gate -> (`maxWireVersion` < 29 and >= 29), and assert the operations succeed and spans are emitted. - -**Why this candidate (and not the alternatives):** cross-driver prose tests can only rely on observables every -driver has. The telemetry section is invisible to command monitoring/APM (transport-layer), and server spans are -observable only with special mongod startup parameters (OTLP exporter + feature flag) that no driver CI matrix -provisions. That rules out prescribing (a) wire-level gating-matrix assertions (driver-internal byte inspection) -and (b) the full server-span-linkage e2e (infra burden other driver teams would reject). The chosen test instead -uses the **server's strict validation as the oracle**: a 9.0+ server `uasserts` on any malformed traceparent and -a pre-9.0 server rejects unknown OP_MSG section kinds, so plain command success across the matrix simultaneously -verifies the wire-version gate and the well-formedness of anything sent — and directly tests the spec's key -operational property that telemetry must never become an availability failure. It runs on existing CI with zero -new infrastructure. - -**Known limitation (accepted):** the test cannot prove the section is ever *sent* — a driver that never attaches -the section passes trivially (false-negative tolerant). Positive proof of emission/omission stays in -driver-internal unit tests (in the Java reference implementation: the `CommandMessageOtelTraceContextTest` -OP_MSG round-trip matrix and the `ServerSpanLinkageProseTest` e2e, see -`docs/superpowers/specs/2026-07-13-otel-telemetry-section-prose-tests.md`); the spec test text should recommend — -not mandate — equivalent driver-level verification. From 8f3db54896cfdf3e6c943ea2e13acb7963156195 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Thu, 23 Jul 2026 12:41:37 +0100 Subject: [PATCH 43/48] DRIVERS-3454: revert pairing-guard test in LoggingCommandEventSenderSpecification (kept locally) --- ...gingCommandEventSenderSpecification.groovy | 29 ------------------- 1 file changed, 29 deletions(-) diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/LoggingCommandEventSenderSpecification.groovy b/driver-core/src/test/unit/com/mongodb/internal/connection/LoggingCommandEventSenderSpecification.groovy index 00faee01f0a..e6f6afb02e0 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/connection/LoggingCommandEventSenderSpecification.groovy +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/LoggingCommandEventSenderSpecification.groovy @@ -91,35 +91,6 @@ class LoggingCommandEventSenderSpecification extends Specification { debugLoggingEnabled << [true, false] } - def 'should not send failed or succeeded events if the started event was not completed'() { - given: - def connectionDescription = new ConnectionDescription(new ServerId(new ClusterId(), new ServerAddress())) - def database = 'test' - def messageSettings = MessageSettings.builder().maxWireVersion(LATEST_WIRE_VERSION).build() - def commandListener = new TestCommandListener() - def commandDocument = new BsonDocument('ping', new BsonInt32(1)) - def replyDocument = new BsonDocument('ok', new BsonInt32(1)) - def message = new CommandMessage(database, commandDocument, - NoOpFieldNameValidator.INSTANCE, ReadPreference.primary(), messageSettings, MULTIPLE, null) - def bsonOutput = new ByteBufferBsonOutput(new SimpleBufferProvider()) - message.encode(bsonOutput, new OperationContext(IgnorableRequestContext.INSTANCE, NoOpSessionContext.INSTANCE, - Stub(TimeoutContext), null)) - def logger = Stub(Logger) { - isDebugEnabled() >> false - } - def sender = new LoggingCommandEventSender([] as Set, [] as Set, connectionDescription, commandListener, - OPERATION_CONTEXT, message, message.getCommandDocument(bsonOutput), - new StructuredLogger(logger), LoggerSettings.builder().build()) - - when: 'terminal events are sent without a completed started event' - sender.sendFailedEvent(new MongoInternalException('failure!')) - sender.sendSucceededEvent(MessageHelper.buildSuccessfulReply(message.getId(), replyDocument.toJson())) - sender.sendSucceededEventForOneWayCommand() - - then: 'no events are delivered' - commandListener.getEvents().isEmpty() - } - def 'should log events'() { given: def serverId = new ServerId(new ClusterId(), new ServerAddress()) From f1a4d02ca8217aca9fbce18a01a10b867d181e37 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Thu, 23 Jul 2026 12:45:36 +0100 Subject: [PATCH 44/48] DRIVERS-3454: drop explanatory comment from sendAndReceiveInternal --- .../mongodb/internal/connection/InternalStreamConnection.java | 4 ---- 1 file changed, 4 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java b/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java index 34accfc332d..b1baf434124 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java +++ b/driver-core/src/main/com/mongodb/internal/connection/InternalStreamConnection.java @@ -441,10 +441,6 @@ private T sendAndReceiveInternal(final CommandMessage message, final Decoder final OperationContext operationContext) { CommandEventSender commandEventSender = new NoOpCommandEventSender(); Span tracingSpan = null; - // Single owner of the span and event sender until the message is handed off to the receive phase: - // any failure from span creation through sendCommandMessage ends the span exactly once and emits at - // most one failed event (LoggingCommandEventSender suppresses terminal events until the started event - // has completed, and Span.closeScope() is a no-op if no scope was opened). try (ByteBufferBsonOutput bsonOutput = new ByteBufferBsonOutput(this)) { tracingSpan = operationContext .getTracingManager() From ca3beb02e0a3f6c0ed2b09c242372be664d5e91b Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Thu, 23 Jul 2026 12:47:43 +0100 Subject: [PATCH 45/48] DRIVERS-3454: revert LoggingCommandEventSender pairing guard (kept locally) --- .../connection/LoggingCommandEventSender.java | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/LoggingCommandEventSender.java b/driver-core/src/main/com/mongodb/internal/connection/LoggingCommandEventSender.java index 6da708d0518..1972e2caaab 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/LoggingCommandEventSender.java +++ b/driver-core/src/main/com/mongodb/internal/connection/LoggingCommandEventSender.java @@ -72,13 +72,6 @@ class LoggingCommandEventSender implements CommandEventSender { private final String commandName; private volatile BsonDocument commandDocument; private final boolean redactionRequired; - /** - * Guards event pairing: terminal (failed/succeeded) events and logs are emitted only after the started - * event has completed, so callers may invoke {@link #sendFailedEvent} unconditionally from any error path - * without pairing knowledge. Volatile because the started and terminal events may fire on different - * threads in the async path. - */ - private volatile boolean startedEventCompleted; LoggingCommandEventSender(final Set securitySensitiveCommands, final Set securitySensitiveHelloCommands, final ConnectionDescription description, @@ -124,15 +117,11 @@ public void sendStartedEvent() { // the buffer underlying the command document may be released after the started event, so set to null to ensure it's not used // when sending the failed or succeeded event commandDocument = null; - startedEventCompleted = true; } @Override public void sendFailedEvent(final Throwable t) { - if (!startedEventCompleted) { - return; - } Throwable commandEventException = t; if (t instanceof MongoCommandException && redactionRequired) { commandEventException = MongoCommandExceptionUtils.redacted((MongoCommandException) t); @@ -168,9 +157,6 @@ public void sendSucceededEventForOneWayCommand() { } private void sendSucceededEvent(final BsonDocument reply) { - if (!startedEventCompleted) { - return; - } long elapsedTimeNanos = System.nanoTime() - startTimeNanos; if (loggingRequired()) { From 4f417ffcc114330212b1124f52c17bacd91bed5c Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Thu, 23 Jul 2026 12:49:44 +0100 Subject: [PATCH 46/48] update doc --- .../com/mongodb/internal/observability/micrometer/Span.java | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/Span.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/Span.java index 91902057db4..f0821f2375d 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/Span.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/Span.java @@ -90,7 +90,7 @@ public MongodbObservationContext getMongodbObservationContext() { /** * Opens a scope for this span, making it the current observation on the thread. - * A successful {@code openScope()} must eventually be followed by {@link #closeScope()}. + * A successful {@code openScope()} must be followed by {@link #closeScope()}. */ void openScope(); From 2ce9337b4e1b12ba3b2f5fc8626b1fa7ea582790 Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Thu, 23 Jul 2026 15:26:03 +0100 Subject: [PATCH 47/48] Bump to wire protocol 29 and Copilot feedback --- .../com/mongodb/connection/ServerDescription.java | 2 +- .../internal/connection/CommandMessage.java | 15 ++++++++------- .../internal/operation/ServerVersionHelper.java | 2 +- .../operation/OperationUnitSpecification.groovy | 3 ++- gradle/libs.versions.toml | 4 ++-- 5 files changed, 14 insertions(+), 12 deletions(-) diff --git a/driver-core/src/main/com/mongodb/connection/ServerDescription.java b/driver-core/src/main/com/mongodb/connection/ServerDescription.java index 2d675bce217..fd82f1dcb16 100644 --- a/driver-core/src/main/com/mongodb/connection/ServerDescription.java +++ b/driver-core/src/main/com/mongodb/connection/ServerDescription.java @@ -65,7 +65,7 @@ public class ServerDescription { * The maximum supported driver wire version * @since 3.8 */ - public static final int MAX_DRIVER_WIRE_VERSION = 25; + public static final int MAX_DRIVER_WIRE_VERSION = 29; private static final int DEFAULT_MAX_DOCUMENT_SIZE = 0x1000000; // 16MB diff --git a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java index 08160ff2bb8..56b66314d63 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +++ b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java @@ -38,8 +38,6 @@ import org.bson.BsonValue; import org.bson.ByteBuf; import org.bson.FieldNameValidator; -import org.bson.codecs.BsonDocumentCodec; -import org.bson.codecs.EncoderContext; import org.bson.io.BsonOutput; import java.io.ByteArrayOutputStream; @@ -218,7 +216,7 @@ BsonDocument getCommandDocument(final ByteBufferBsonOutput bsonOutput) { } /** - * The command name (first key of the command document. Available before {@code encode()} runs, unlike + * The command name (the first key of the command document). Available before {@code encode()} runs, unlike * {@link #getCommandDocument(ByteBufferBsonOutput)}; identical to the encoded document's first * key, since encoding only appends fields. */ @@ -353,10 +351,13 @@ private void writeTelemetryContextSection(final ByteBufferBsonOutput bsonOutput) return; } bsonOutput.writeByte(PAYLOAD_TYPE_3_TELEMETRY); - BsonDocument telemetry = new BsonDocument("otel", - new BsonDocument("traceparent", new BsonString(traceParent))); - new BsonDocumentCodec().encode(new BsonBinaryWriter(bsonOutput), telemetry, - EncoderContext.builder().build()); + // {otel: {traceparent: }} + BsonBinaryWriter writer = new BsonBinaryWriter(bsonOutput); + writer.writeStartDocument(); + writer.writeStartDocument("otel"); + writer.writeString("traceparent", traceParent); + writer.writeEndDocument(); + writer.writeEndDocument(); } private int writeOpQuery(final ByteBufferBsonOutput bsonOutput) { diff --git a/driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java b/driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java index 655ab416490..482458c8615 100644 --- a/driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java +++ b/driver-core/src/main/com/mongodb/internal/operation/ServerVersionHelper.java @@ -34,7 +34,7 @@ public final class ServerVersionHelper { // Server 9.0 (WIRE_VERSION_90 = 29). Minimum wire version for the OP_MSG telemetry // section (OTel trace-context propagation, DRIVERS-3454). public static final int NINE_DOT_ZERO_WIRE_VERSION = 29; - public static final int LATEST_WIRE_VERSION = EIGHT_DOT_ZERO_WIRE_VERSION; + public static final int LATEST_WIRE_VERSION = NINE_DOT_ZERO_WIRE_VERSION; public static boolean serverIsAtLeastVersionFourDotFour(final ConnectionDescription description) { return description.getMaxWireVersion() >= FOUR_DOT_FOUR_WIRE_VERSION; diff --git a/driver-core/src/test/unit/com/mongodb/internal/operation/OperationUnitSpecification.groovy b/driver-core/src/test/unit/com/mongodb/internal/operation/OperationUnitSpecification.groovy index ec5cb74156f..fb5fa791eda 100644 --- a/driver-core/src/test/unit/com/mongodb/internal/operation/OperationUnitSpecification.groovy +++ b/driver-core/src/test/unit/com/mongodb/internal/operation/OperationUnitSpecification.groovy @@ -65,7 +65,8 @@ class OperationUnitSpecification extends Specification { [6, 2]: 19, [6, 3]: 20, [7, 0]: 21, - [9, 0]: 25, + [8, 0]: 25, + [9, 0]: 29, ] static Integer getMaxWireVersionForServerVersion(List serverVersion) { diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 2e67d80b43b..848ae670815 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -187,8 +187,8 @@ reactive-streams-tck = { module = " org.reactivestreams:reactive-streams-tck", v reflections = { module = "org.reflections:reflections", version.ref = "reflections" } micrometer-tracing = { module = "io.micrometer:micrometer-tracing", version.ref = "micrometer-tracing" } -micrometer-tracing-integration-test-bom = { module = " io.micrometer:micrometer-tracing-bom", version.ref = "micrometer-tracing" } -micrometer-tracing-integration-test = { module = " io.micrometer:micrometer-tracing-integration-test" } +micrometer-tracing-integration-test-bom = { module = "io.micrometer:micrometer-tracing-bom", version.ref = "micrometer-tracing" } +micrometer-tracing-integration-test = { module = "io.micrometer:micrometer-tracing-integration-test" } [bundles] aws-java-sdk-v1 = ["aws-java-sdk-v1-core", "aws-java-sdk-v1-sts"] From 73cf17ba8aee9133b6470a5a1e1f9c64a0a39f2a Mon Sep 17 00:00:00 2001 From: Nabil Hachicha Date: Thu, 23 Jul 2026 15:43:47 +0100 Subject: [PATCH 48/48] review feedback --- .../internal/connection/CommandMessage.java | 65 ++++++++++--------- 1 file changed, 33 insertions(+), 32 deletions(-) diff --git a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java index 56b66314d63..074c0129cfc 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +++ b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java @@ -172,41 +172,42 @@ BsonDocument getCommandDocument(final ByteBufferBsonOutput bsonOutput) { byteBuf.position(firstDocumentPosition); ByteBufBsonDocument byteBufBsonDocument = createOne(byteBuf); - // If true, there are more sections after the `PAYLOAD_TYPE_0_DOCUMENT` section: either one or more + // There may be more sections after the `PAYLOAD_TYPE_0_DOCUMENT` section: either one or more // `PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE` sections, and/or a trailing `PAYLOAD_TYPE_3_TELEMETRY` section. - if (byteBuf.hasRemaining()) { - BsonDocument commandBsonDocument = byteBufBsonDocument.toBaseBsonDocument(); - - // Each loop iteration processes one Document Sequence - // When there are no more bytes remaining, there are no more Document Sequences - while (byteBuf.hasRemaining()) { - byte payloadType = byteBuf.get(); - // Document-sequence sections always precede any trailing non-sequence section (e.g. - // PAYLOAD_TYPE_3_TELEMETRY), and such sections carry no command-document fields, so stop here. - if (payloadType != PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE) { - break; - } - int sequenceStart = byteBuf.position(); - int sequenceSizeInBytes = byteBuf.getInt(); - int sectionEnd = sequenceStart + sequenceSizeInBytes; - - String fieldName = getSequenceIdentifier(byteBuf); - // If this assertion fires, it means that the driver has started using document sequences for nested fields. If - // so, this method will need to change in order to append the value to the correct nested document. - assertFalse(fieldName.contains(".")); - - ByteBuf documentsByteBufSlice = byteBuf.duplicate().limit(sectionEnd); - try { - commandBsonDocument.append(fieldName, new BsonArray(createList(documentsByteBufSlice))); - } finally { - documentsByteBufSlice.release(); - } - byteBuf.position(sectionEnd); + // The command document is materialized lazily, only when a document sequence has to be folded into + // it; otherwise (no sections, or only a telemetry section) the cheap lazy document is returned. + BsonDocument commandBsonDocument = null; + + // Each loop iteration processes one Document Sequence + // When there are no more bytes remaining, there are no more Document Sequences + while (byteBuf.hasRemaining()) { + byte payloadType = byteBuf.get(); + // Document-sequence sections always precede any trailing non-sequence section (e.g. + // PAYLOAD_TYPE_3_TELEMETRY), and such sections carry no command-document fields, so stop here. + if (payloadType != PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE) { + break; } - return commandBsonDocument; - } else { - return byteBufBsonDocument; + if (commandBsonDocument == null) { + commandBsonDocument = byteBufBsonDocument.toBaseBsonDocument(); + } + int sequenceStart = byteBuf.position(); + int sequenceSizeInBytes = byteBuf.getInt(); + int sectionEnd = sequenceStart + sequenceSizeInBytes; + + String fieldName = getSequenceIdentifier(byteBuf); + // If this assertion fires, it means that the driver has started using document sequences for nested fields. If + // so, this method will need to change in order to append the value to the correct nested document. + assertFalse(fieldName.contains(".")); + + ByteBuf documentsByteBufSlice = byteBuf.duplicate().limit(sectionEnd); + try { + commandBsonDocument.append(fieldName, new BsonArray(createList(documentsByteBufSlice))); + } finally { + documentsByteBufSlice.release(); + } + byteBuf.position(sectionEnd); } + return commandBsonDocument != null ? commandBsonDocument : byteBufBsonDocument; } finally { byteBuf.release(); }