[HELP] Trace explorer contains a "Missing span ID xxx" entry

[michaeloliverx]

Hi there, I have been experiementing with adding OpenTelemetry to my CloudRun application. It is a FastAPI powered application:

# main.py
...
app = FastAPI()

setup_opentelemetry(app)
setup_logging()

# setup_opentelemetry.py
import os

from fastapi import FastAPI
from opentelemetry import metrics, trace
from opentelemetry.exporter.cloud_monitoring import (
    CloudMonitoringMetricsExporter,
)
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.instrumentation.logging import LoggingInstrumentor
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.cloud_trace_propagator import (
    CloudTraceFormatPropagator,
)
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import SERVICE_INSTANCE_ID, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
)

import galactic_api.database


def setup_opentelemetry(app: FastAPI) -> None:
    # https://google-cloud-opentelemetry.readthedocs.io/en/stable/index.html

    # This will use the GooglecloudResourceDetector under the covers.
    resource = Resource.create(
        attributes={
            # Use the PID as the service.instance.id to avoid
            # duplicate timeseries from different Gunicorn worker processes.
            SERVICE_INSTANCE_ID: f"worker-{os.getpid()}",
        }
    )

    # Tracing
    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(
        BatchSpanProcessor(CloudTraceSpanExporter())
    )
    trace.set_tracer_provider(tracer_provider)

    # Metrics
    metrics.set_meter_provider(
        MeterProvider(
            metric_readers=[
                PeriodicExportingMetricReader(CloudMonitoringMetricsExporter())
            ],
            resource=resource,
        )
    )

    # Set the X-Cloud-Trace-Context header
    set_global_textmap(CloudTraceFormatPropagator())

    # Instrumentors
    FastAPIInstrumentor.instrument_app(app, tracer_provider=tracer_provider)
    RequestsInstrumentor().instrument(tracer_provider=tracer_provider)
    HTTPXClientInstrumentor().instrument()
    SQLAlchemyInstrumentor().instrument(
        engine=galactic_api.database.engine,
    )
    OpenAIInstrumentor().instrument()

    # Enable OpenTelemetry Logging Instrumentation
    LoggingInstrumentor().instrument()

The exporter is running and I can see trace data in the Trace Explorer, the problem is for each request there is always (Missing span ID xxx):

Also does this output seem correct here? Why is there a span /concepts and a sub span OPTIONS /concepts

Do I need to include set_global_textmap(CloudTraceFormatPropagator())? Does Cloud Run auto inject a X-Cloud-Trace-Context header to incoming requests?

---

[aabmass]

I'd recommend migrating to the Telemetry (OTLP) API endpoint https://docs.cloud.google.com/trace/docs/migrate-to-otlp-endpoints, instead of using the CloudTraceSpanExporter in this repo.

Do I need to include set_global_textmap(CloudTraceFormatPropagator())? Does Cloud Run auto inject a X-Cloud-Trace-Context header to incoming requests?

No, Cloud Run should be using traceparent to propagate which is included by default.

But the problem is most likely Cloud Run insert spans which aren't showing up. Are you exporting spans to the same project as where Cloud Run instances are living?