Skip to content

feat(spanner): support user-provided OpenTelemetry for client metrics export#13679

Closed
rahul2393 wants to merge 1 commit into
googleapis:mainfrom
rahul2393:feat/spanner-custom-otel-metrics
Closed

feat(spanner): support user-provided OpenTelemetry for client metrics export#13679
rahul2393 wants to merge 1 commit into
googleapis:mainfrom
rahul2393:feat/spanner-custom-otel-metrics

Conversation

@rahul2393

@rahul2393 rahul2393 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds Client Metrics: a new, opt-in feature that exports Spanner's client-side metrics to a caller-provided OpenTelemetry pipeline (OTLP, Prometheus, any exporter), including on Spanner Omni where the existing Cloud Monitoring metrics are unavailable.

Fully opt-in and fully decoupled from the existing built-in (Cloud Monitoring) metrics. Default behavior is unchanged: without a client-metrics provider, nothing new happens, and built-in metrics continue to export to Cloud Monitoring exactly as before.

Scope note: This change was split out of a larger PR into two focused PRs. This PR contains only the Client Metrics export feature. The optional endpoint metric attribute for location-aware routing is a separate follow-up: #13740.

Two independent features

  • Built-in metrics (existing, unchanged): export to Google Cloud Monitoring. Controlled by setBuiltInMetricsEnabled and the SPANNER_DISABLE_BUILTIN_METRICS environment variable, exactly as today. Not available on Spanner Omni.
  • Client metrics (new): export the client instruments to a caller-owned OpenTelemetry. Controlled solely by setClientMetricsProvider(...) — a CustomOpenTelemetryMetricsProvider turns it on; NoopMetricsProvider (or no provider) turns it off. Works on all instance types, Omni included.

The two are decoupled: setBuiltInMetricsEnabled and the env var affect only the built-in Cloud Monitoring sink and have no effect on client metrics; the client-metrics provider affects only the caller-owned sink and has no effect on built-in metrics. Under the hood these are the same client instruments, exported under the distinct spanner/client namespace — the difference is the export path, not the metrics.

Emulator handling

Client metrics are not recorded when Spanner is pointed at the emulator. To make that reliable when the emulator is configured programmatically (via setEmulatorHost(...)) and not only through the SPANNER_EMULATOR_HOST environment variable, this PR broadens emulator detection: SpannerOptions.isEmulatorEnabled() now also recognizes the builder-configured emulator host, and the connection-check error message is generalized to describe both the environment-variable and programmatic configuration paths. This gates both the client and Cloud Monitoring metrics sinks off against the emulator using a single detection predicate. Called out explicitly here for reviewer visibility.

Motivation

Client metrics currently export only to Cloud Monitoring, which is unavailable on Spanner Omni. Customers running on Omni (or who standardize on their own observability stack) had no way to receive these metrics. This lets them route the metrics to any OpenTelemetry exporter, independently of the built-in Cloud Monitoring configuration.

API

SdkMeterProviderBuilder meterProviderBuilder = SdkMeterProvider.builder();
SpannerMetrics.configureMeterProviderBuilder(meterProviderBuilder);
// ... attach your exporter to meterProviderBuilder ...
OpenTelemetry otel =
    OpenTelemetrySdk.builder().setMeterProvider(meterProviderBuilder.build()).build();

SpannerOptions options =
    SpannerOptions.newBuilder()
        .setClientMetricsProvider(new CustomOpenTelemetryMetricsProvider(otel))
        .build();

@rahul2393 rahul2393 requested review from a team as code owners July 7, 2026 17:32

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the MetricsProvider interface and its implementations (DefaultMetricsProvider, NoopMetricsProvider, and CustomOpenTelemetryMetricsProvider) to allow customization of how built-in client-side metrics are collected and exported in the Cloud Spanner Java client. This enables recording metrics on a caller-provided OpenTelemetry instance rather than exporting them directly to Google Cloud Monitoring, which is particularly useful for Spanner Omni clients. Additionally, the changes add support for registering built-in metric views on a custom meter provider, introduce an endpoint attribute for location-aware routing, and include comprehensive integration and unit tests to validate these new features. There are no review comments, so I have no feedback to provide.

@rahul2393 rahul2393 force-pushed the feat/spanner-custom-otel-metrics branch from 124a9c7 to 383daa2 Compare July 8, 2026 07:18
@rahul2393

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a customizable MetricsProvider interface for the Cloud Spanner Java client, allowing users to configure how built-in metrics are collected and exported. It adds support for a CustomOpenTelemetryMetricsProvider to record metrics on a user-provided OpenTelemetry instance, alongside the default Google Cloud Monitoring export and a NoopMetricsProvider to disable metrics entirely. The changes also include labeling attempt metrics with routed endpoints on the location-aware fastpath, and extensive integration tests verifying the dual-sink and gRPC-layer metrics behavior. Feedback on the code changes highlights a potential UnsupportedOperationException in BuiltInMetricsConstant.java due to mutating a set returned by Collectors.toSet(), with a suggestion to use Collectors.toCollection(HashSet::new) to guarantee mutability.

@rahul2393 rahul2393 force-pushed the feat/spanner-custom-otel-metrics branch from f7bd207 to 5a63c9e Compare July 8, 2026 09:17
@rahul2393

Copy link
Copy Markdown
Contributor Author

Current failures in CI pipeline are because of #13584
cc: @lqiu96

@rahul2393 rahul2393 force-pushed the feat/spanner-custom-otel-metrics branch from 72b0be7 to 03c7825 Compare July 8, 2026 14:56
@lqiu96

lqiu96 commented Jul 8, 2026

Copy link
Copy Markdown
Member

Current failures in CI pipeline are because of #13584 cc: @lqiu96

Sorry, we will fix this upstream. I think you should be able to proceed without this check if you want

@rahul2393 rahul2393 force-pushed the feat/spanner-custom-otel-metrics branch from b5fa19a to 1add8f1 Compare July 8, 2026 17:16
throws IOException {
// Only do the check if the emulator environment variable has been set to localhost.
if (isEmulatorEnabled(options, emulatorHost)) {
String resolvedEmulatorHost = emulatorHost != null ? emulatorHost : options.getEndpoint();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we doing this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Earlier only env was checked, now this is updated to also check if emulator is enabled programmatically via SpannerOptions, to log that in case it's set we are adding this

rahul2393 added a commit to rahul2393/google-cloud-java that referenced this pull request Jul 14, 2026
… export

Adds the MetricsProvider family (Default/Noop/CustomOpenTelemetry),
SpannerOptions.Builder#setClientMetricsProvider, and
SpannerMetrics#configureMeterProviderBuilder so client metrics can be
recorded on a caller-owned OpenTelemetry in the spanner/client
namespace, decoupled from the Cloud Monitoring built-in metrics flag.
Also broadens the emulator gate (SpannerOptions#isEmulatorEnabled,
enablegRPCMetrics overload) so no client metrics are recorded against
the emulator.

Split from PR googleapis#13679 (client-metrics half).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rahul2393 rahul2393 force-pushed the feat/spanner-custom-otel-metrics branch from af8fd4a to bfc4aea Compare July 14, 2026 05:57
@rahul2393 rahul2393 changed the title feat(spanner): support user-provided OpenTelemetry for built-in client metrics and endpoint attribute for location-aware routing feat(spanner): support user-provided OpenTelemetry for client metrics export Jul 14, 2026
… export

Adds the MetricsProvider family (Default/Noop/CustomOpenTelemetry),
SpannerOptions.Builder#setClientMetricsProvider, and
SpannerMetrics#configureMeterProviderBuilder so client metrics can be
recorded on a caller-owned OpenTelemetry in the spanner/client namespace,
decoupled from the Cloud Monitoring built-in metrics flag. Also broadens
the emulator gate (SpannerOptions#isEmulatorEnabled, enablegRPCMetrics
overload) so no client metrics are recorded against the emulator.
@rahul2393 rahul2393 force-pushed the feat/spanner-custom-otel-metrics branch from bfc4aea to 818893b Compare July 14, 2026 06:06
rahul2393 added a commit to rahul2393/google-cloud-java that referenced this pull request Jul 14, 2026
…location-aware routing

Adds the ENDPOINT_KEY metric attribute: KeyAwareChannel labels each
attempt with the endpoint the location-aware fastpath routed it to, and
the custom-export attempt views allow the label through their attribute
filter. Cloud Monitoring views never include it.

Stacked follow-up to the client metrics export change (googleapis#13679); until
that merges this commit also carries the client metrics export work.
Merge after googleapis#13679 lands.
@rahul2393 rahul2393 closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants