Skip to content

fix(spanner): derive built-in metrics project from database client #13262

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rahul2393 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix(spanner): derive built-in metrics project from database client #13262

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3ef313c
fix(spanner): derive built-in metrics project from database client
rahul2393 May 25, 2026
52234f4
address review comments
rahul2393 May 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 38 additions & 2 deletions ...r/google-cloud-spanner/src/main/java/com/google/cloud/spanner/BuiltInMetricsProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
import com.google.cloud.opentelemetry.detection.AttributeKeys;
import com.google.cloud.opentelemetry.detection.DetectedPlatform;
import com.google.cloud.opentelemetry.detection.GCPPlatformDetector;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Strings;
import com.google.common.hash.HashFunction;
import com.google.common.hash.Hashing;
Expand Down Expand Up @@ -75,10 +76,12 @@ final class BuiltInMetricsProvider {
private static final String default_location = "global";

private OpenTelemetry openTelemetry;
private String projectId;
private boolean mismatchedProjectIdLogged;

private BuiltInMetricsProvider() {}

OpenTelemetry getOrCreateOpenTelemetry(
synchronized OpenTelemetry getOrCreateOpenTelemetry(
String projectId,
@Nullable Credentials credentials,
@Nullable String monitoringHost,
Expand All @@ -88,7 +91,7 @@ OpenTelemetry getOrCreateOpenTelemetry(
SdkMeterProviderBuilder sdkMeterProviderBuilder = SdkMeterProvider.builder();
BuiltInMetricsView.registerBuiltinMetrics(
SpannerCloudMonitoringExporter.create(
projectId, credentials, monitoringHost, universeDomain),
this::getProjectId, credentials, monitoringHost, universeDomain),
Copy link
Copy Markdown
Contributor

@surbhigarg92 surbhigarg92 May 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rahul2393 I did not understand this solution. getOrCreateOpenTelemetry is called from GapicSpannerRPC while creating SpannerClient. At the time projectId shared here could be the projectId of GKE instance for example.

So in this case we will be initialising SpannerCloudMonitoringExporter with null projectId ? As by this time setProjectIdIfAbsent won't be called, it is called later during database init.

Copy link
Copy Markdown
Contributor Author

@rahul2393 rahul2393 May 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So flow is:

  1. SpannerClient init → OpenTelemetry/exporter may be created, project supplier returns null
  2. no export happens yet because no database project is known
  3. getDatabaseClient(DatabaseId) → database project is set once
  4. future metric exports use that database project

Copy link
Copy Markdown
Contributor

@surbhigarg92 surbhigarg92 May 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are also passing the projectId in next line to create OpenTelemetry Resource

Copy link
Copy Markdown
Contributor Author

@rahul2393 rahul2393 May 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also do

monitoredResourceBuilder.putLabels(PROJECT_ID_KEY.getKey(), projectId);

So the resource created during SDK initialization may contain the early/default project, but before sending
createServiceTimeSeries, we overwrite the monitored resource label with the database project from the exporter supplier.

sdkMeterProviderBuilder);
sdkMeterProviderBuilder.setResource(Resource.create(createResourceAttributes(projectId)));
SdkMeterProvider sdkMeterProvider = sdkMeterProviderBuilder.build();
Expand All @@ -106,6 +109,39 @@ OpenTelemetry getOrCreateOpenTelemetry(
}
}

synchronized void setProjectIdIfAbsent(String projectId) {
if (this.projectId == null) {
this.projectId = projectId;
} else if (!this.projectId.equals(projectId) && !mismatchedProjectIdLogged) {
mismatchedProjectIdLogged = true;
logger.log(
Level.WARNING,
"Built-in metrics are already initialized for project {0}. Metrics for project {1} will"
+ " be exported using the existing project.",
new Object[] {this.projectId, projectId});
}
}

@Nullable
synchronized OpenTelemetry getOpenTelemetry() {
return this.openTelemetry;
}

@VisibleForTesting
synchronized String getProjectId() {
return this.projectId;
}

@VisibleForTesting
synchronized void reset() {
if (this.openTelemetry instanceof OpenTelemetrySdk) {
((OpenTelemetrySdk) this.openTelemetry).getSdkMeterProvider().close();
}
this.openTelemetry = null;
this.projectId = null;
this.mismatchedProjectIdLogged = false;
}

// TODO: Remove when
// https://github.com/GoogleCloudPlatform/opentelemetry-operations-java/issues/421
// has been fixed.
Expand Down
63 changes: 23 additions & 40 deletions ...-cloud-spanner/src/main/java/com/google/cloud/spanner/SpannerCloudMonitoringExporter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,16 +42,15 @@
import io.opentelemetry.sdk.metrics.data.AggregationTemporality;
import io.opentelemetry.sdk.metrics.data.MetricData;
import io.opentelemetry.sdk.metrics.export.MetricExporter;
import io.opentelemetry.sdk.resources.Resource;
import java.io.IOException;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.Supplier;
import java.util.logging.Level;
import java.util.logging.Logger;
import java.util.stream.Collectors;
import javax.annotation.Nonnull;
import javax.annotation.Nullable;

Expand All @@ -70,16 +69,24 @@ class SpannerCloudMonitoringExporter implements MetricExporter {
// https://cloud.google.com/monitoring/quotas#custom_metrics_quotas.
private static final int EXPORT_BATCH_SIZE_LIMIT = 200;
private final AtomicBoolean spannerExportFailureLogged = new AtomicBoolean(false);
private final AtomicBoolean lastExportSkippedData = new AtomicBoolean(false);
private final MetricServiceClient client;
private final String spannerProjectId;
private final Supplier<String> spannerProjectIdSupplier;

static SpannerCloudMonitoringExporter create(
String projectId,
@Nullable Credentials credentials,
@Nullable String monitoringHost,
String universeDomain)
throws IOException {
return create(() -> projectId, credentials, monitoringHost, universeDomain);
}

static SpannerCloudMonitoringExporter create(
Supplier<String> projectIdSupplier,
@Nullable Credentials credentials,
@Nullable String monitoringHost,
String universeDomain)
throws IOException {
MetricServiceSettings.Builder settingsBuilder = MetricServiceSettings.newBuilder();
CredentialsProvider credentialsProvider;
if (credentials == null || credentials instanceof NoCredentials) {
Expand Down Expand Up @@ -114,13 +121,18 @@ static SpannerCloudMonitoringExporter create(
settingsBuilder.createServiceTimeSeriesSettings().setSimpleTimeoutNoRetriesDuration(timeout);

return new SpannerCloudMonitoringExporter(
projectId, MetricServiceClient.create(settingsBuilder.build()));
projectIdSupplier, MetricServiceClient.create(settingsBuilder.build()));
}

@VisibleForTesting
SpannerCloudMonitoringExporter(String projectId, MetricServiceClient client) {
this(() -> projectId, client);
}

@VisibleForTesting
SpannerCloudMonitoringExporter(Supplier<String> projectIdSupplier, MetricServiceClient client) {
this.client = client;
this.spannerProjectId = projectId;
this.spannerProjectIdSupplier = projectIdSupplier;
}

@Override
Expand All @@ -140,37 +152,20 @@ MetricServiceClient getMetricServiceClient() {

/** Export client built in metrics */
private CompletableResultCode exportSpannerClientMetrics(Collection<MetricData> collection) {
// Filter spanner metrics. Only include metrics that contain a valid project.
List<MetricData> spannerMetricData = collection.stream().collect(Collectors.toList());

// Log warnings for metrics that will be skipped.
boolean mustFilter = false;
if (spannerMetricData.stream()
.map(metricData -> metricData.getResource())
.anyMatch(this::shouldSkipPointDataDueToProjectId)) {
logger.log(
Level.WARNING, "Some metric data contain a different projectId. These will be skipped.");
mustFilter = true;
}

if (mustFilter) {
spannerMetricData =
spannerMetricData.stream()
.filter(this::shouldSkipMetricData)
.collect(Collectors.toList());
String spannerProjectId = spannerProjectIdSupplier.get();
if (Strings.isNullOrEmpty(spannerProjectId)) {
return CompletableResultCode.ofSuccess();
}
lastExportSkippedData.set(mustFilter);

// Skips exporting if there's none
if (spannerMetricData.isEmpty()) {
if (collection.isEmpty()) {
return CompletableResultCode.ofSuccess();
}

List<TimeSeries> spannerTimeSeries;
try {
spannerTimeSeries =
SpannerCloudMonitoringExporterUtils.convertToSpannerTimeSeries(
spannerMetricData, this.spannerProjectId);
collection, spannerProjectId);
} catch (Throwable e) {
logger.log(
Level.WARNING,
Expand Down Expand Up @@ -218,18 +213,6 @@ public void onSuccess(List<Empty> empty) {
return spannerExportCode;
}

private boolean shouldSkipMetricData(MetricData metricData) {
return shouldSkipPointDataDueToProjectId(metricData.getResource());
}

private boolean shouldSkipPointDataDueToProjectId(Resource resource) {
return !spannerProjectId.equals(SpannerCloudMonitoringExporterUtils.getProjectId(resource));
}

boolean lastExportSkippedData() {
return this.lastExportSkippedData.get();
}

private ApiFuture<List<Empty>> exportTimeSeriesInBatch(
ProjectName projectName, List<TimeSeries> timeSeries) {
List<ApiFuture<Empty>> batchResults = new ArrayList<>();
Expand Down
9 changes: 3 additions & 6 deletions ...d-spanner/src/main/java/com/google/cloud/spanner/SpannerCloudMonitoringExporterUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@
import io.opentelemetry.sdk.metrics.data.MetricDataType;
import io.opentelemetry.sdk.metrics.data.PointData;
import io.opentelemetry.sdk.metrics.data.SumData;
import io.opentelemetry.sdk.resources.Resource;
import java.util.ArrayList;
Comment thread
rahul2393 marked this conversation as resolved.
import java.util.Collection;
import java.util.List;
Comment thread
rahul2393 marked this conversation as resolved.
import java.util.logging.Level;
import java.util.logging.Logger;
Expand All @@ -75,12 +75,8 @@ class SpannerCloudMonitoringExporterUtils {

private SpannerCloudMonitoringExporterUtils() {}

static String getProjectId(Resource resource) {
return resource.getAttributes().get(PROJECT_ID_KEY);
}

static List<TimeSeries> convertToSpannerTimeSeries(
List<MetricData> collection, String projectId) {
Collection<MetricData> collection, String projectId) {
List<TimeSeries> allTimeSeries = new ArrayList<>();

for (MetricData metricData : collection) {
Expand All @@ -102,6 +98,7 @@ static List<TimeSeries> convertToSpannerTimeSeries(
monitoredResourceBuilder.putLabels(
key.getKey(), String.valueOf(resourceAttributes.get(key)));
}
monitoredResourceBuilder.putLabels(PROJECT_ID_KEY.getKey(), projectId);

metricData.getData().getPoints().stream()
.map(
Expand Down
2 changes: 2 additions & 0 deletions java-spanner/google-cloud-spanner/src/main/java/com/google/cloud/spanner/SpannerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,7 @@ public DatabaseClient getDatabaseClient(DatabaseId db) {
if (clientId == null) {
clientId = nextDatabaseClientId(db);
}
getOptions().initializeBuiltInMetrics(db);
MultiplexedSessionDatabaseClient multiplexedSessionDatabaseClient =
new MultiplexedSessionDatabaseClient(SpannerImpl.this.getSessionClient(db));
DatabaseClientImpl dbClient =
Expand Down Expand Up @@ -337,6 +338,7 @@ public BatchClient getBatchClient(DatabaseId db) {
if (this.dbBatchClients.containsKey(db)) {
return this.dbBatchClients.get(db);
}
getOptions().initializeBuiltInMetrics(db);
BatchClientImpl batchClient = new BatchClientImpl(getSessionClient(db));
this.dbBatchClients.put(db, batchClient);
return batchClient;
Expand Down
11 changes: 11 additions & 0 deletions java-spanner/google-cloud-spanner/src/main/java/com/google/cloud/spanner/SpannerOptions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2520,6 +2520,17 @@ public void enablegRPCMetrics(InstantiatingGrpcChannelProvider.Builder channelPr
}
}

void initializeBuiltInMetrics(DatabaseId databaseId) {
if (isEnableBuiltInMetrics() && !usesNoCredentials()) {
this.builtInMetricsProvider.setProjectIdIfAbsent(databaseId.getInstanceId().getProject());
this.builtInMetricsProvider.getOrCreateOpenTelemetry(
databaseId.getInstanceId().getProject(),
getCredentials(),
this.monitoringHost,
getUniverseDomain());
}
}

public ApiTracerFactory getApiTracerFactory(boolean isAdminClient, boolean isEmulatorEnabled) {
return createApiTracerFactory(isAdminClient, isEmulatorEnabled);
}
Expand Down
51 changes: 51 additions & 0 deletions ...anner/src/test/java/com/google/cloud/spanner/BuiltInOpenTelemetryMetricsProviderTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,31 @@
package com.google.cloud.spanner;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNull;
import static org.junit.Assert.assertTrue;

import com.google.auth.oauth2.AccessToken;
import com.google.auth.oauth2.OAuth2Credentials;
import java.util.Date;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;

@RunWith(JUnit4.class)
public class BuiltInOpenTelemetryMetricsProviderTest {

@Before
public void setUp() {
BuiltInMetricsProvider.INSTANCE.reset();
}

@After
public void tearDown() {
BuiltInMetricsProvider.INSTANCE.reset();
}

@Test
public void testGenerateClientHashWithSimpleUid() {
String clientUid = "testClient";
Expand Down Expand Up @@ -56,11 +72,46 @@ public void testGenerateClientHashWithSpecialCharacters() {
verifyHash(BuiltInMetricsProvider.generateClientHash(clientUid));
}

@Test
public void testApiTracerFactoryDoesNotSetBuiltInMetricsProject() {
SpannerOptions options = newTestOptions();

options.getApiTracerFactory(/* isAdminClient= */ false, /* isEmulatorEnabled= */ false);

assertNull(BuiltInMetricsProvider.INSTANCE.getProjectId());
}

@Test
public void testBuiltInOpenTelemetryDoesNotSetMetricsProject() {
SpannerOptions options = newTestOptions();

options.getBuiltInOpenTelemetry();

assertNull(BuiltInMetricsProvider.INSTANCE.getProjectId());
}

@Test
public void testInitializeBuiltInMetricsUsesDatabaseProject() {
SpannerOptions options = newTestOptions();

options.initializeBuiltInMetrics(DatabaseId.of("database-project", "i", "d"));

assertEquals("database-project", BuiltInMetricsProvider.INSTANCE.getProjectId());
}

private void verifyHash(String hash) {
// Check if the hash length is 6
assertEquals(hash.length(), 6);
// Check if the hash is in the range [000000, 0003ff]
long hashValue = Long.parseLong(hash, 16); // Convert hash from hex to decimal
assertTrue(hashValue >= 0 && hashValue <= 0x3FF);
}

private SpannerOptions newTestOptions() {
return SpannerOptions.newBuilder()
.setProjectId("host-project")
.setCredentials(
OAuth2Credentials.create(new AccessToken("test-token", new Date(Long.MAX_VALUE))))
.build();
}
}
Loading
Loading