Skip to content

CRITICAL - google-adk 2.2.0 breaks Agent Engine/Runtime observability. #6247

Description

@vishal-bulbule

Describe the Bug:

google-adk 2.2.0 breaks Vertex AI Agent Engine observability. Since v2.2.0, the Agent Engine dashboard (Sessions, Agent invocations, Model calls, Models panels) shows 0 for every deployment, even though spans are ingested correctly.

Root cause: get_gcp_resource() in src/google/adk/telemetry/google_cloud.py:252 emits the cloud resource identifier under the key cloud.resource.id (dots):

if cloud_resource_id is not None:
    resource_attributes["cloud.resource.id"] = cloud_resource_id   # should be "cloud.resource_id"

The OpenTelemetry semantic convention — and the key the dashboard's BigQuery query filters on — is cloud.resource_id (underscore):

>>> from opentelemetry.semconv.resource import ResourceAttributes
>>> ResourceAttributes.CLOUD_RESOURCE_ID
'cloud.resource_id'

The dashboard filters per deployment with ENDS_WITH(JSON_VALUE(resource.attributes,'$."cloud.resource_id"'), <agentEngineUri>). Because ADK writes the non-standard cloud.resource.id, this filter matches nothing → every panel reads 0. The value is correct (a GCP full resource name); only the attribute key is wrong.

Steps to Reproduce:

  1. Install google-adk==2.2.0.
  2. Reproduce directly in the library (no GCP required):
    python3 - <<'PY'
    import os
    os.environ["GOOGLE_CLOUD_AGENT_ENGINE_ID"] = "1234567890"
    os.environ["GOOGLE_CLOUD_LOCATION"] = "us-central1"
    from google.adk.telemetry.google_cloud import get_gcp_resource
    r = get_gcp_resource("my-project")
    print("cloud.resource_id :", r.attributes.get("cloud.resource_id"))
    print("cloud.resource.id :", r.attributes.get("cloud.resource.id"))
    PY
  3. (End-to-end) Deploy any agent to Agent Engine: adk deploy agent_engine --project=PROJECT --region=us-central1 <agent_dir>.
  4. Invoke the deployed agent through a session (Playground or stream_query), then open the deployment's Dashboard → Overview / Models in the console.
  5. Observe all panels show 0 / "no rows".

Expected Behavior:

get_gcp_resource() sets the OTel-standard cloud.resource_id attribute, so the dashboard's filter matches and the Sessions / Agent invocations / Model calls / Models panels populate.

Observed Behavior:

cloud.resource_id is None; the value is emitted under cloud.resource.id instead, so the dashboard matches 0 rows and every per-deployment panel shows 0.

Output of step 2:

cloud.resource_id : None
cloud.resource.id : //aiplatform.googleapis.com/projects/my-project/locations/us-central1/reasoningEngines/1234567890

Environment Details:

  • ADK Library Version (pip show google-adk): 2.2.0 (also present in 2.3.0 and main)
  • Desktop OS: macOS
  • Python Version (python -V): Python 3.12.12

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: gemini-2.5-flash

🟡 Optional Information

Regression:

Yes — this is a regression introduced in v2.2.0 (commit ffa057c1, 2026-06-04, "feat: Add --trigger_sources and ADK service options to cli_deploy_agent_engine"). Present in v2.3.0 and main.

Confirmed by A/B deploying the same agent code on the same project:

ADK version Resource key emitted in _Trace Dashboard Overview
2.1.0 cloud.resource_id (correct) Sessions 2, turns/session 1.5, invocations 3
2.2.0 cloud.resource.id (wrong) Sessions 0

Mechanism: in ≤2.1.0 the correct cloud.resource_id was produced by GoogleCloudResourceDetector. The v2.2.0 agent-engine branch in get_gcp_resource() returns early, skips GoogleCloudResourceDetector, and hardcodes the typo'd cloud.resource.id — so it both drops the correct key and adds a wrong one.

Logs:

Span resource keys in the Agent Engine _Trace store, per deployed engine (BigQuery over the _Trace linked dataset). Every 2.2.0/2.3.0 engine has 0 spans with the standard key; the 2.1.0 engine has the correct key:

+---------------------+--------------+------------+   engine (ADK ver)
| engine_id           | standard_key | dotted_key |
+---------------------+--------------+------------+
| 2361379891288473600 |           16 |          0 |   2.1.0  -> dashboard works
| 4220240637485645824 |            0 |         22 |   2.2.0  -> dashboard 0
+---------------------+--------------+------------+

Raw resource.attributes of a 2.2.0 span (note the dotted key):

{
  "cloud.platform": "gcp.agent_engine",
  "cloud.resource.id": "//aiplatform.googleapis.com/projects/.../locations/us-central1/reasoningEngines/...",
  "service.name": "...",
  "telemetry.sdk.name": "opentelemetry",
  "telemetry.sdk.version": "1.41.1"
}

Screenshots / Video: Click to Open

Demo: ADK Agent Engine observability broken in v2.2.0

Additional Context:

Suggested fix — use the OTel constant so the key can't drift from the standard:

from opentelemetry.semconv.resource import ResourceAttributes
resource_attributes[ResourceAttributes.CLOUD_RESOURCE_ID] = cloud_resource_id  # "cloud.resource_id"

(Minimal equivalent: resource_attributes["cloud.resource_id"] = cloud_resource_id.)

The existing tests/unittests/telemetry/test_google_cloud.py::test_get_gcp_resource only asserts gcp.project_id, so no test pins the wrong key; a case asserting cloud.resource_id is set when the agent-engine env vars are present would lock in the fix.

Minimal Reproduction Code:

import os
os.environ["GOOGLE_CLOUD_AGENT_ENGINE_ID"] = "1234567890"
os.environ["GOOGLE_CLOUD_LOCATION"] = "us-central1"

from google.adk.telemetry.google_cloud import get_gcp_resource
r = get_gcp_resource(project_id="my-project")

print("cloud.resource_id :", r.attributes.get("cloud.resource_id"))  # None  <-- dashboard expects this
print("cloud.resource.id :", r.attributes.get("cloud.resource.id"))  # value (wrong key)

How often has this issue occurred?:

  • Always (100%)

Metadata

Metadata

Assignees

Labels

request clarification[Status] The maintainer need clarification or more information from the authortracing[Component] This issue is related to OpenTelemetry tracing

Type

Fields

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions