feat(oam): migrate agentic OAM components from appmod-blueprints#18
Open
shapirov103 wants to merge 84 commits into
Open
feat(oam): migrate agentic OAM components from appmod-blueprints#18shapirov103 wants to merge 84 commits into
shapirov103 wants to merge 84 commits into
Conversation
Move the OAM artifacts that are specific to the agentic platform out of appmod-blueprints, leaving generic platform components (appmod-service, dp-service-account, dynamodb-*, rds-*, s3-*, ingress trait, platform-env) behind. KubeVela itself continues to be provisioned by the platform. - platform/oam/: CUE sources for agent, mcp-server, agentcore-memory ComponentDefinitions, plus generate.sh, README, DESIGN, and 11 examples covering minimal/simple/MCP/AgentCore-memory/Milvus/session-affinity flows. - gitops/addons/charts/oam-agent-components/: thin Helm chart packaging the three ComponentDefinition manifests so they can be applied as a standard addon once KubeVela is running. Chart only \u2014 no addon registration entry yet (waiting on platform-side KubeVela provisioning to land). - applications/strands-agent-base/: reference Strands agent app (Dockerfile with python:3.13-slim-bookworm + apt CVE patching, build/podman scripts, unit + integration tests, A2A and gateway integration fixtures).
The fleet-secret chart on each cluster reads enable_<addon> flags from
gitops/overlays/environments/<env>/enabled-addons.yaml and writes them as
labels on the ArgoCD cluster secret. ApplicationSets then match those
labels to deploy addons.
Previously:
hub: crossplane=true, kubevela=false → no KubeVela anywhere
dev: no crossplane, no crossplane_aws, no kubevela entries
prod: same as dev
Result was that crossplane-agentcore-spoke-{dev,prod} apps had been
retrying ~3400 times against spokes that had no Crossplane CRDs.
This change:
hub: flip kubevela to true
dev: add crossplane: true, crossplane_aws: true, kubevela: true
prod: same as dev
After fleet-secret reconciles, the cluster secret labels propagate and
Crossplane + AWS providers + KubeVela install via existing
ApplicationSets in the platform repo. The agentcore Compositions and
Provider that the spoke apps are trying to apply will then have CRDs to
land on.
Two related changes for the agentic platform:
1. Remove kagent operator (no longer needed)
- Drop kagent-crds, kagent, kagent-setup, kagent-monitoring from
bootstrap/default/addons.yaml
- Drop the same enable flags from environments/control-plane/addons.yaml
- Delete gitops/addons/charts/kagent-setup/, kagent-monitoring/, and
gitops/addons/default/addons/kagent/ values dir
- Rename namespaces previously colocated with kagent:
litellm: kagent → litellm
prometheus-operator-crds: kagent → monitoring
- Rename selector enable_kagent_monitoring → enable_monitoring on
prometheus-operator-crds
- Update README, Taskfile, scripts/bootstrap.sh to drop kagent
references
- Rewrite gitops/DEPLOYMENT.md to reflect the kagent-free deployment
- Clean up platform/oam/README.md (drop kagent-rollout-component
references, generic namespace examples, fix file list to match the
migrated content)
ArgoCD will prune the 12 kagent Applications (kagent-crds, kagent,
kagent-setup, kagent-monitoring × 3 clusters) on next sync.
2. Register oam-agent-components addon (KubeVela ComponentDefinitions
for agent, mcp-server, agentcore-memory)
- Add addon definition in bootstrap/default/addons.yaml at sync-wave 5
(after kubevela, which is at sync-wave 3 on the platform side)
- Selector enable_oam_components (no-op when global selector
enable_agent_platform is in effect)
- Enable in environments/control-plane/addons.yaml
Once KubeVela is installed (via the kubevela addon enabled in the
prior commit fba4afb), this addon registers the three agentic
ComponentDefinitions cluster-wide so OAM Applications can use them.
- Add deps: [agentic:ensure-kubeconfig] to bootstrap, status, destroy
- Pass --context {{.HUB_CLUSTER_NAME}} explicitly to every kubectl call
so the active kubeconfig context cannot leak through (e.g., from a
rebooted kind cluster on 127.0.0.1)
- Use --alias {{.HUB_CLUSTER_NAME}} on update-kubeconfig so the context
has a clean short name instead of the EKS ARN
- Add cluster-info validation to ensure-kubeconfig with a clear success
message; fails early with a clear error if creds/network are bad
The agent and mcp-server ComponentDefinitions in oam-agent-components
emit argoproj.io/v1alpha1 Rollout resources. KubeVela's mutating
admission webhook does pre-render validation and rejects them with:
no matches for kind "Rollout" in version "argoproj.io/v1alpha1"
Enabling argo_rollouts on all 3 overlays so the platform-side
ApplicationSet (registered in appmod-blueprints) deploys the Argo
Rollouts controller + CRDs to all clusters.
Validated by:
kubectl get applications.argoproj.io oam-agent-components-* -A
-> message: 'admission webhook denied request: no matches for kind
"Rollout"'
Only agentcore-memory ComponentDefinition registered; agent and
mcp-server were dropped due to admission failure.
The chart is a wave-5 declarative consumer in the addon-wave taxonomy documented in appmod-blueprints/gitops/addons/registry/README.md (see appmod-blueprints PR #751). Its ComponentDefinitions reference resource kinds delivered by lower-wave addons: wave 3: argo-rollouts (Rollout CRD), crossplane (Composition) wave 4: kubevela (ComponentDefinition CRD and webhook) Adding new ComponentDefinitions here does not change the chart's wave — only changes to the set of consumed addons do. The dependency on these addons is enforced by ArgoCD sync-wave ordering, not by Helm chart `dependencies:` which would bundle copies of the addons (each is its own ArgoCD Application). - Chart.yaml: rewritten description with explicit prerequisites - README.md: new file with role explanation, prerequisite table, lifecycle notes, and a pointer to the upstream wave taxonomy
…th, repoint Strands
- Add local umbrella chart gitops/addons/charts/bifrost wrapping upstream
maximhq/bifrost 2.1.16 and bundling, in one Application:
- Crossplane Role + PodIdentityAssociation granting bedrock:InvokeModel* to
the bifrost ServiceAccount (EKS Pod Identity, not IRSA)
- ExternalSecret for the encryption key + platform virtual key
- vendored subchart (Chart.lock + charts/*.tgz) for clean-state ArgoCD render
- Bedrock provider via IAM (no static keys); expose Claude Sonnet 4.5 as the
alias 'claude-sonnet' -> us.anthropic.claude-sonnet-4-5-20250929-v1:0
- Enable Bifrost governance (is_vk_mandatory) with a budget, rate-limit, and
platform virtual key; agents authenticate via the x-bf-vk header
- Switch bifrost addon to local path with per-cluster templated valuesObject
- Repoint Strands agent from LiteLLM to Strands OpenAIModel against Bifrost /v1
(OAM agent ComponentDefinition default + app config/deployment)
- Remove ExternalSecret template and externalSecret values block; customers do not need Secrets Manager - bootstrap.sh step 5 now creates the bifrost namespace + generates bifrost-secrets (BIFROST_ENCRYPTION_KEY + BIFROST_PLATFORM_VK) via openssl rand; prints the VK at the end so the operator can pass it to agents via LLM_GATEWAY_API_KEY / modelConfig.llmGatewayApiKey - Idempotent: skips if bifrost-secrets already exists - Remove secretManagerKey from addon valuesObject
…and spokes - Replace per-cluster bootstrap.sh Secret with ExternalSecret reading from account-level SM path 'open-agentic-platform/bifrost' (works on hub and all spoke clusters without per-cluster action) - Add Taskfile task agentic:bifrost-secrets: idempotent, auto-generates BIFROST_ENCRYPTION_KEY + BIFROST_PLATFORM_VK, creates the SM secret, prints the VK for operator use; runs automatically as dep of agentic:bootstrap - Remove bifrost step from bootstrap.sh (wrong layer — doesn't reach spokes) - No customer action required: task agentic:install handles everything
…test - Add bifrost: true to control-plane, dev, prod enabled-addons overlays so the addon actually deploys (was missing — Bifrost would never start) - Update OAM example files to Bifrost /v1 URL - Rewrite integration test for OpenAIModel + x-bf-vk (drop LiteLLMModel)
…ng design doc - Drop ExternalSecret template, externalSecret values blocks, envFrom, encryptionKeySecret — Bifrost runs without secrets until per-workload VK minting is implemented - Remove agentic:bifrost-secrets Taskfile task (no longer needed) - Add gitops/addons/charts/bifrost/DESIGN.md capturing per-workload VK minting architecture (KubeVela workflow step → management API → k8s Secret) - is_vk_mandatory remains false; TODO comment points to DESIGN.md
…ment not StatefulSet Config changes now roll out without delete-and-recreate. Postgres StatefulSet spec is stable. TODO: migrate to RDS with passwordCommand + Pod Identity.
Langfuse ExternalSecret was reading <aws_cluster_name>/keycloak-clients which resolves to spoke-dev/keycloak-clients — a path that doesn't exist. The keycloak config Job runs on the hub and writes all client secrets to <hub_cluster_name>/keycloak-clients (e.g. peeks-hub/keycloak-clients). Fix: use hub_cluster_name annotation (already set on every spoke cluster secret by the fleet-secret chart) so all langfuse instances — hub and spokes — read from the same Secrets Manager path.
…default-roles-platform
…stead The agentcore chart created a PodIdentityAssociation (eks.aws.upbound.io) on the spoke. Spokes have no EKS-API Crossplane provider (they never provision clusters), so the association could never reconcile -> agentcore provider had no credentials -> all AgentCore resources stuck OutOfSync. Gate the chart's Role + PodIdentityAssociation behind podIdentity.createViaChart (default false). The agentcore provider's Pod Identity is now provisioned by the hub's Crossplane via the pod-identities overlay, which targets the spoke cluster from the hub (where the working EKS provider lives).
…ver image + example
…attern in example
The client-provisioner ApplicationSet reads fleet-member values from the fleet repo (this install uses OAP as fleetRepo) and templates the chart path as platform-charts/<client_provisioner>-client-provisioner. Without the label the path rendered platform-charts/<no value>-client-provisioner and the generated Applications were stuck in Unknown/app-path-does-not-exist. Adds client_provisioner: keycloak so the path resolves to keycloak-client-provisioner (chart lives in the addons repo).
… gate Reverts the hub-overlay workaround. The agentcore provider's IAM Role + PodIdentityAssociation are created by the chart and reconciled locally by the spoke's credentialed iam/eks providers — consistent with bifrost and every other workload. No central per-workload identity registration.
…itecture - probabilistic_sampler at 100% (never drop agent spans) - memory_limiter with percentage-based limits (85%/10% spike) - batch processor (1024/500ms/2048) for efficient export - otlphttp/langfuse with 30s timeout + retry_on_failure (5s→30s, 5m max) - prometheus receiver scraping Bifrost at :8080/metrics - Two pipelines: traces (otlp→langfuse+xray), metrics (prometheus→xray) - Langfuse exporter conditional on endpoint being set (prevents crash) - Matches the target architecture diagram
…oesn't support metrics) awsxray exporter only supports traces. The metrics pipeline (Bifrost prometheus scrape) uses debug exporter until AMP remote write is configured.
…b cluster The hub cluster secret doesn't have hub_cluster_name annotation (set by Crossplane for spokes only). Use default 'hub' to prevent ApplicationSet template rendering failure.
…nnotation Go template .metadata.annotations.hub_cluster_name errors on missing key before default can evaluate. Use index function which returns empty string for missing keys, then pipe to default.
…ays on hub) The hub cluster secret lacks hub_cluster_name annotation and Go template index/default functions don't work with ArgoCD ApplicationSet's template engine for missing map keys. Since Langfuse always runs on hub, the SM path is always hub/langfuse-otel regardless of which cluster consumes it.
The OTel collector on spoke clusters reads Langfuse auth from SM via ExternalSecret. The hub Langfuse chart creates the source K8s Secret and a PushSecret that syncs it to hub/langfuse-otel in SM. Flow: Langfuse chart (hub) → K8s Secret → PushSecret → SM → ExternalSecret (spokes) → OTel collector env var
PushSecret stores values as SecretBinary. ExternalSecret needs decodingStrategy:Auto to handle both binary and string formats.
PushSecret stores as SecretBinary which corrupts the key when ExternalSecret reads it back. Use aws-cli Job to write as SecretString (proper JSON) so ExternalSecret can extract properties correctly.
The push-otel-keys Job needs AWS credentials to write to SM. Creates an IAM Role with secretsmanager:Create/Put/Describe/Get scoped to hub/langfuse-* secrets, and a PodIdentityAssociation binding langfuse-seed SA to that role.
…auth change The collector pod starts with optional LANGFUSE_OTEL_AUTH env var. If the secret didn't exist at startup, the env is empty and never reloads. Adding a checksum annotation ensures the Deployment spec changes when the secret config changes, triggering a rollout.
…INT is set The agent should emit traces via the collector (not just direct-to-Langfuse). When OTEL_EXPORTER_OTLP_ENDPOINT points to the local collector, initialize StrandsTelemetry without auth headers (collector handles auth to Langfuse).
…ackend Architecture: Agent → Collector → Langfuse (traces) + debug (metrics) X-Ray is not part of the target architecture.
1. Cost tracking: Seed CronJob registers claude-sonnet model pricing in Langfuse via API (idempotent). Bifrost OTEL spans will have canonical model names matching Langfuse built-in pricing. 2. AMP remote write: Collector supports prometheusremotewrite/amp exporter when amp.enabled=true and endpoint is set. Falls back to debug exporter when AMP is not configured. 3. Bifrost W3C traceparent: Enable Bifrost OTEL plugin pointing to the local OTel collector (http :4318). Bifrost emits child spans with gen_ai.* attributes (tokens, cost, model) that merge with agent root spans in the collector. Also: Fix Bifrost values.yaml hardcoded region (was us-west-2, now empty — set dynamically by ApplicationSet).
- Add sigv4auth extension for AWS authentication - Add prometheusremotewrite/amp exporter (conditional on amp.enabled) - Add Pod Identity (IAM Role + PodIdentityAssociation) for aps:RemoteWrite - Pass AMP endpoint from addons.yaml to all spoke collectors - Metrics pipeline: Bifrost prometheus scrape → batch → AMP remote write
…fig change The OTel collector reads config only at startup. Without a checksum annotation tracking config-affecting values, ArgoCD syncs update the ConfigMap but the pod continues using stale config (e.g. debug exporter instead of prometheusremotewrite/amp). This annotation forces a rolling restart whenever the Langfuse endpoint, Bifrost scrape target, or AMP settings change.
…ter name 1. AMP remote write endpoint: replaced hardcoded workspace ID (ws-1d91195d-cb39-470d-827a-9625b3574df4) with a templated annotation reference (amp_remote_write_endpoint). Each deployment must set this annotation on the cluster secret. When absent, AMP is gracefully disabled (falls back to debug exporter). 2. Langfuse pod-identity: replaced hardcoded clusterName 'hub' with a templated value from the ApplicationSet (aws_cluster_name annotation). This fixes the ResourceNotFoundException when EKS can't find a cluster named 'hub' in environments where the cluster has a different name. Both issues were reported by users running fresh installs where the cluster names and AMP workspace IDs differ from the original dev env.
The agent image uses opentelemetry-exporter-otlp-proto-http which only supports HTTP protocol. The CUE ComponentDefinition was injecting port 4317 (gRPC), causing the agent to hang during telemetry init and fail liveness probes. Both the agent SDK and Bifrost OTEL plugin use HTTP (:4318). Updated the CUE default and OBSERVABILITY.md to match.
…-east-1 The langfuse chart's pod-identity uses global.awsRegion for the PodIdentityAssociation region. It was defaulting to us-east-1 because the ApplicationSet valuesObject was not passing global.awsRegion. Clusters in us-west-2 (or any other region) would fail with: ResourceNotFoundException: No cluster found for name: hub Fix: - Add global.awsRegion to langfuse valuesObject (from aws_region annotation) - Change values.yaml default from 'us-east-1' to empty string (forces injection from ApplicationSet, never silently falls back to wrong region)
…ctorMatchLabels The appset-chart uses useSelectors: false with globalSelectors (enable_agent_platform), meaning the per-addon selector.matchExpressions is NOT used as a generator filter. All addons generate apps for every cluster with enable_agent_platform=true. This caused langfuse-spoke-dev and langfuse-spoke-prod ApplicationSet apps to be created, which fail because the pod-identity references a cluster name that doesn't match (langfuse is hub-only infrastructure). Fix: Add selectorMatchLabels.enable_langfuse=true to the langfuse addon. The appset-chart template renders selectorMatchLabels into the generator's matchLabels alongside globalSelectors. Since only the hub cluster secret has enable_langfuse=true, the ApplicationSet will only generate the langfuse-hub app.
The appset-chart only generates an ApplicationSet when enabled: true. With enabled: false, no ApplicationSet is created regardless of cluster labels. The per-addon selectorMatchLabels controls which clusters get the app (e.g. enable_langfuse for hub-only), but the ApplicationSet must exist first. Enabled: langfuse, otel-collector, bifrost, agent-gateway, gateway-api-crds, agentgateway, agentgateway-crds, oam-agent-components, crossplane-agentcore, prometheus-operator-crds. Kept disabled: litellm (replaced by Bifrost), jaeger (replaced by Langfuse).
litellm is replaced by Bifrost. jaeger is replaced by Langfuse. The litellm ApplicationSet was blocking the agent-platform-addons sync because EKS ArgoCD waits for ApplicationSet health checks which stall.
Allows per-addon matchLabels (e.g. enable_langfuse) to be merged into the cluster generator selector alongside globalSelectors. This restricts langfuse to hub-only without needing useSelectors: true.
- Changed all hardcoded 'us-east-1' defaults to 'us-west-2' in langfuse pod-identity, push-otel-secret, and otel-collector values. - Added 'langfuse: true' to control-plane enabled-addons.yaml so the fleet-secret chart sets enable_langfuse label on the hub cluster secret. Combined with selectorMatchLabels in the langfuse addon definition, this ensures the langfuse ApplicationSet only generates apps for clusters with the label (hub only).
The appmod-blueprints task install writes amp_endpoint_url annotation to the cluster secret (via Secrets Manager). Aligned the otel-collector addon to read from this annotation and append api/v1/remote_write. When the annotation is absent (not yet provisioned), the template renders empty and AMP is gracefully disabled (falls back to debug).
The collector template already appends api/v1/remote_write to the remoteWriteEndpoint value. The addons.yaml was also appending it, resulting in a double path that returns 404.
The CUE source was updated but the generated YAML in oam-agent-components/templates/agent.yaml was not regenerated.
The PlatformCluster claims are GitOps-managed: the fleet `clusters` ApplicationSet generates `clusters-<env>` Application(s) that own the claims with automated prune. Deleting a claim directly makes ArgoCD immediately re-sync and recreate it, spinning up a fresh spoke EKS + VPC while the old one is still deleting. Pause the ApplicationSet and disable self-heal on the generated Application(s) before deleting the claims so they stay deleted and Crossplane can drain the spokes cleanly.
fix(spokes:destroy): pause GitOps generator before deleting claims
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves the OAM artifacts that are specific to the agentic platform out of
appmod-blueprintsand into this repo, leaving generic platform components (appmod-service, dp-service-account, dynamodb-, rds-, s3-*, ingress trait, platform-env config) where they belong. KubeVela itself continues to be provisioned by the platform — this PR ships only the agentic ComponentDefinitions and supporting artifacts.What changed
platform/oam/agent,mcp-server,agentcore-memoryComponentDefinitions, plusgenerate.sh, README, DESIGN, and 11 examplesgitops/addons/charts/oam-agent-components/applications/strands-agent-base/Total: 50 files, +5920 lines, no deletions.
Migration scope rationale
This PR was deliberately scoped to agentic-only artifacts. Things explicitly left in
appmod-blueprints:appmod-service,dp-service-account,dynamodb-ack,dynamodb-table,rds-cluster,rds-cluster-mysql,s3-bucket,s3-bucket-ack,external-database-secret,platform-env(config)ingress,path-based-ingress,component-policygitops/addons/charts/kubevela/) — platform installs vela-coreThings deliberately not migrated even though they exist on the source branch:
mcp-gateway-operatorchart — out of scope per current iterationagentgatewaywrapper chart — duplicates this repo's existing split (agentgateway/controller +agent-gateway/resources +litellm/chart). Bedrock-backend and litellm pieces from the wrapper are redundant here.gitops/agentic/parallel hierarchy +agentic-addons.yamlApplicationSet — this repo uses a single unified addon hierarchyaddons.yamldiff — mostly version regressions (argocd 9.5.12→9.4.9, cert-manager v1.20.2→v1.19.4, crossplane 2.2.1→2.2.0, mlflow 1.8.1→1.7.2, etc.); this repo is ahead on all shared addon versionsOpen items (intentional follow-ups, not blockers)
Addon registration:
oam-agent-componentschart is not yet wired intogitops/addons/bootstrap/default/addons.yaml. Waiting on the platform-side KubeVela provisioning work infeature/agent-platformto land first. Suggested entry once ready:Doc references not yet updated:
platform/oam/README.mdstill referencesapplications/strands/(path renamed tostrands-agent-base/in this PR) and DESIGN.md mentions platform components likeappmod-service.cuethat aren't in this repo. These will be cleaned in a follow-up commit on this branch before merging.Source-branch cleanup: corresponding deletions in
appmod-blueprints(removing the migrated agentic files) will land in a separate PR there.Tested
git archive | tarfaithful copy verified — line counts match source for all 3 generated ComponentDefinition YAMLs (agent.yaml 296L, mcp-server.yaml 265L, agentcore-memory.yaml 85L).venv/__pycache__/ build artifacts leaked into the strands-agent-base copyhelm template gitops/addons/charts/oam-agent-components/smoke testhelm installagainst a cluster with KubeVela already running → ComponentDefinitions registered → example Application deploysSource provenance
Branch:
feature/oam-for-agentsinaws-samples/appmod-blueprintsSource HEAD at migration:
222fa7648c24cdf40f97fbf12fdef14d7e83e1f8