Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,19 @@ title: "Advanced Tracing (OTEL)"
description: "Explore manual context propagation, custom decorators, and sampling techniques for real-world async, multi-service, and high-volume tracing scenarios."
---

## What it is
## About

**Advanced Tracing** covers OpenTelemetry patterns that go beyond basic span creation. You'll learn how to propagate trace context manually across async tasks, threads, and microservices; build custom decorators that instrument functions without modifying their logic; and write custom samplers that selectively record or drop spans based on attributes like user ID or span name.
Basic span creation works for synchronous, single-service code. But real applications run async tasks, communicate across microservices, and generate more telemetry than needed. Advanced tracing covers the OpenTelemetry patterns for these scenarios: manual context propagation across async tasks, threads, and services; custom decorators for function-level instrumentation; and custom samplers to control which spans are recorded.

## Use cases
---

## When to use

- **Async tracing**Manually pass and attach context in Python `async/await` or JS `Promise`-based code where automated propagation doesn't suffice.
- **Multi-service tracing** Inject and extract trace context from HTTP headers to link spans across microservices into a single distributed trace.
- **Concurrent thread tracing**Capture context in the main thread and propagate it to worker threads so all tasks remain associated with the parent trace.
- **Function-level instrumentation**Write a custom decorator that starts a span, records inputs and outputs, and ends the span without touching the function body.
- **Selective sampling** Drop spans for specific users or conditions to reduce telemetry volume and cost while keeping high-value traces.
- **Async tracing**: Manually pass and attach context in Python `async/await` or JS `Promise`-based code where automated propagation does not work.
- **Multi-service tracing**: Inject and extract trace context from HTTP headers to link spans across microservices into a single distributed trace.
- **Concurrent thread tracing**: Capture context in the main thread and propagate it to worker threads so all tasks stay linked to the parent trace.
- **Function-level instrumentation**: Write a custom decorator that starts a span, records inputs and outputs, and ends the span without modifying the function body.
- **Selective sampling**: Drop spans for specific users or conditions to reduce telemetry volume and cost while keeping high-value traces.

---

Expand Down Expand Up @@ -95,15 +97,15 @@ description: "Explore manual context propagation, custom decorators, and samplin
<Tab title="Microservices">
When making HTTP calls to another microservice, inject the current trace context into request headers in Service A and extract it in Service B to link spans across services.

**Service A**inject context into outgoing request headers:
**Service A**:inject context into outgoing request headers:

<CodeGroup>

```python Python
import requests
from opentelemetry import trace
# from opentelemetry.context import Context # Not strictly needed for inject but good for awareness
from opentelemetry.propagators.textmap import DefaultTextMapPropagator
from opentelemetry.propagate import inject, extract

tracer = trace.get_tracer(__name__)

Expand All @@ -112,7 +114,7 @@ description: "Explore manual context propagation, custom decorators, and samplin
with tracer.start_as_current_span("llm_service_a") as span:
# Prepare headers
headers = {}
DefaultTextMapPropagator().inject(carrier=headers) # Inject the current context
inject(carrier=headers) # Inject the current context

# Make the request with the injected headers
response = requests.get("http://localhost:5001/endpoint", headers=headers) # Assuming Python Service B runs on 5001
Expand Down Expand Up @@ -181,14 +183,14 @@ description: "Explore manual context propagation, custom decorators, and samplin

</CodeGroup>

**Service B**extract context from incoming request headers:
**Service B**:extract context from incoming request headers:

<CodeGroup>

```python Python
from flask import Flask, request
from opentelemetry import trace
from opentelemetry.propagators.textmap import DefaultTextMapPropagator
from opentelemetry.propagate import inject, extract
# Minimal OTel setup for console output if not already configured globally
# from opentelemetry.sdk.trace import TracerProvider
# from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
Expand All @@ -201,7 +203,7 @@ description: "Explore manual context propagation, custom decorators, and samplin
@app.route("/endpoint")
def endpoint():
# Extract the context from incoming request
context_from_propagator = DefaultTextMapPropagator().extract(carrier=dict(request.headers))
context_from_propagator = extract(carrier=dict(request.headers))

# Create a new span as child
with tracer.start_as_current_span("python_service_b_processing", context=context_from_propagator) as span:
Expand Down Expand Up @@ -395,7 +397,7 @@ description: "Explore manual context propagation, custom decorators, and samplin
</Tab>

<Tab title="Custom Decorators">
A custom decorator starts a span before the function call, records function arguments and return values as span attributes, and ends the spanwithout modifying the function body.
A custom decorator starts a span before the function call, records function arguments and return values as span attributes, and ends the span:without modifying the function body.

<CodeGroup>

Expand Down Expand Up @@ -661,16 +663,16 @@ description: "Explore manual context propagation, custom decorators, and samplin

## Key concepts

- **`attach()` / `detach()`**Python functions to manually bind a captured context to the current thread or async task. Always call `detach(token)` in a `finally` block to avoid context leaks.
- **`context.with(ctx, fn)`**JS/TS equivalent of `attach`/`detach`. Runs `fn` with the specified context active, then restores the previous context automatically.
- **`propagation.inject()` / `propagation.extract()`**Serialize the current trace context into HTTP headers (inject) and deserialize it from incoming headers (extract) to link spans across services.
- **Custom decorators**Wrap functions with span start/end logic so every call is traced automatically. Use `functools.wraps` in Python to preserve the original function's metadata.
- **`Sampler` interface**Implement `should_sample()` (Python) or `shouldSample()` (JS/TS) to return `DROP`, `RECORD_ONLY`, or `RECORD_AND_SAMPLE` based on span name, kind, or attributes.
- **`SamplingResult`**The object returned by a sampler. Set `decision` to control recording and optionally attach additional attributes (e.g., a sampling reason).
- **`attach()` / `detach()`**:Python functions to manually bind a captured context to the current thread or async task. Always call `detach(token)` in a `finally` block to avoid context leaks.
- **`context.with(ctx, fn)`**:JS/TS equivalent of `attach`/`detach`. Runs `fn` with the specified context active, then restores the previous context automatically.
- **`propagation.inject()` / `propagation.extract()`**:Serialize the current trace context into HTTP headers (inject) and deserialize it from incoming headers (extract) to link spans across services.
- **Custom decorators**:Wrap functions with span start/end logic so every call is traced automatically. Use `functools.wraps` in Python to preserve the original function's metadata.
- **`Sampler` interface**:Implement `should_sample()` (Python) or `shouldSample()` (JS/TS) to return `DROP`, `RECORD_ONLY`, or `RECORD_AND_SAMPLE` based on span name, kind, or attributes.
- **`SamplingResult`**:The object returned by a sampler. Set `decision` to control recording and optionally attach additional attributes (e.g., a sampling reason).

---

## What you can do next
## Next Steps

<CardGroup cols={2}>
<Card title="Set Up Tracing" icon="gear" href="/docs/observe/features/manual-tracing/set-up-tracing">
Expand Down