Skip to content

feat(spanner): support user-provided OpenTelemetry for client metrics export#13741

Open
rahul2393 wants to merge 1 commit into
mainfrom
feat/spanner-custom-otel-metrics
Open

feat(spanner): support user-provided OpenTelemetry for client metrics export#13741
rahul2393 wants to merge 1 commit into
mainfrom
feat/spanner-custom-otel-metrics

Conversation

@rahul2393

Copy link
Copy Markdown
Contributor

Summary

Adds Client Metrics: a new, opt-in feature that exports Spanner's client-side metrics to a caller-provided OpenTelemetry pipeline (OTLP, Prometheus, any exporter), including on Spanner Omni where the existing Cloud Monitoring metrics are unavailable.

Fully opt-in and fully decoupled from the existing built-in (Cloud Monitoring) metrics. Default behavior is unchanged: without a client-metrics provider, nothing new happens, and built-in metrics continue to export to Cloud Monitoring exactly as before.

Scope note: This change was split out of a larger PR into two focused PRs. This PR contains only the Client Metrics export feature. The optional endpoint metric attribute for location-aware routing is a separate follow-up: #13740.

Two independent features

  • Built-in metrics (existing, unchanged): export to Google Cloud Monitoring. Controlled by setBuiltInMetricsEnabled and the SPANNER_DISABLE_BUILTIN_METRICS environment variable, exactly as today. Not available on Spanner Omni.
  • Client metrics (new): export the client instruments to a caller-owned OpenTelemetry. Controlled solely by setClientMetricsProvider(...) — a CustomOpenTelemetryMetricsProvider turns it on; NoopMetricsProvider (or no provider) turns it off. Works on all instance types, Omni included.

The two are decoupled: setBuiltInMetricsEnabled and the env var affect only the built-in Cloud Monitoring sink and have no effect on client metrics; the client-metrics provider affects only the caller-owned sink and has no effect on built-in metrics. Under the hood these are the same client instruments, exported under the distinct spanner/client namespace — the difference is the export path, not the metrics.

Emulator handling

Client metrics are not recorded when Spanner is pointed at the emulator. To make that reliable when the emulator is configured programmatically (via setEmulatorHost(...)) and not only through the SPANNER_EMULATOR_HOST environment variable, this PR broadens emulator detection: SpannerOptions.isEmulatorEnabled() now also recognizes the builder-configured emulator host, and the connection-check error message is generalized to describe both the environment-variable and programmatic configuration paths. This gates both the client and Cloud Monitoring metrics sinks off against the emulator using a single detection predicate. Called out explicitly here for reviewer visibility.

Motivation

Client metrics currently export only to Cloud Monitoring, which is unavailable on Spanner Omni. Customers running on Omni (or who standardize on their own observability stack) had no way to receive these metrics. This lets them route the metrics to any OpenTelemetry exporter, independently of the built-in Cloud Monitoring configuration.

API

SdkMeterProviderBuilder meterProviderBuilder = SdkMeterProvider.builder();
SpannerMetrics.configureMeterProviderBuilder(meterProviderBuilder);
// ... attach your exporter to meterProviderBuilder ...
OpenTelemetry otel =
    OpenTelemetrySdk.builder().setMeterProvider(meterProviderBuilder.build()).build();

SpannerOptions options =
    SpannerOptions.newBuilder()
        .setClientMetricsProvider(new CustomOpenTelemetryMetricsProvider(otel))
        .build();

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for exporting Cloud Spanner client metrics to a caller-owned OpenTelemetry destination using a new MetricsProvider API, allowing users to configure custom pipelines independently of the default Google Cloud Monitoring export. It also updates documentation, adds comprehensive integration tests, and integrates routed endpoint attributes for location-aware routing. The review feedback highlights a bug in the newly added isEmulatorEnabled() method, where a mismatched protocol prefix (e.g., http://) between the configured host and the emulator host can cause the emulator detection to incorrectly return false.

Comment on lines +3052 to +3057
public boolean isEmulatorEnabled() {
return getChannelProvider() == null
&& emulatorHost != null
&& getHost() != null
&& getHost().equals(emulatorHost);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The isEmulatorEnabled() method checks getHost().equals(emulatorHost). However, when setEmulatorHost is called with a value like "localhost:1234", the builder prepends "http://" to this.host (making it "http://localhost:1234"), but builder.emulatorHost remains "localhost:1234". This causes getHost().equals(emulatorHost) to return false because of the mismatched "http://" prefix.

To make this robust and handle both cases (with or without the protocol prefix), consider normalizing the comparison or checking if getHost() ends with emulatorHost or starts with it after stripping the protocol. Note that a null check on getHost() is unnecessary here as the host would have already been validated.

  @InternalApi
  public boolean isEmulatorEnabled() {
    if (getChannelProvider() != null || emulatorHost == null) {
      return false;
    }
    String normalizedHost = getHost().startsWith("http://") ? getHost().substring(7) : getHost();
    String normalizedEmulatorHost = emulatorHost.startsWith("http://") ? emulatorHost.substring(7) : emulatorHost;
    return normalizedHost.equals(normalizedEmulatorHost);
  }
References
  1. A null check on a parsed URL component (like the host) is unnecessary if other parts of the constructor would have already failed with a malformed URL (e.g., one missing a scheme).

… export

Adds the MetricsProvider family (Default/Noop/CustomOpenTelemetry),
SpannerOptions.Builder#setClientMetricsProvider, and
SpannerMetrics#configureMeterProviderBuilder so client metrics can be
recorded on a caller-owned OpenTelemetry in the spanner/client namespace,
decoupled from the Cloud Monitoring built-in metrics flag. Also broadens
the emulator gate (SpannerOptions#isEmulatorEnabled, enablegRPCMetrics
overload) so no client metrics are recorded against the emulator.
The emulator gate normalizes the http(s) scheme away before comparing
the configured host with the emulator host.
@rahul2393 rahul2393 force-pushed the feat/spanner-custom-otel-metrics branch from 818893b to 0d0ab92 Compare July 14, 2026 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant