fix(langfuse): push OTEL keys via Sync hook so hub/langfuse-otel is created#27
Open
elamaran11 wants to merge 1 commit into
Open
fix(langfuse): push OTEL keys via Sync hook so hub/langfuse-otel is created#27elamaran11 wants to merge 1 commit into
elamaran11 wants to merge 1 commit into
Conversation
…lways created
The langfuse-push-otel-keys Job (writes Langfuse project API keys to Secrets
Manager at hub/langfuse-otel) was an argocd PostSync hook. PostSync only fires
after the whole Application reaches a completed, healthy sync — but langfuse-hub
routinely sits OutOfSync/Progressing (StatefulSets clickhouse/minio/postgres and
the keycloak ExternalSecret churn), so the hook never ran. Result: hub/langfuse-otel
never exists and every otel-collector ExternalSecret fails with SecretSyncedError,
leaving otel-collector-{hub,spoke-dev,spoke-prod} Degraded.
Switch the Job to a Sync hook with sync-wave 5 so it runs during the sync
operation (after the langfuse-seed SA + pod-identity in earlier waves) regardless
of whether other resources have converged. The IAM role (hub-langfuse-seed-role,
secretsmanager write on hub/langfuse-*) and pod identity already exist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
otel-collector-{hub,spoke-dev,spoke-prod} are stuck Degraded: their
langfuse-otel-authExternalSecret fails withSecretSyncedError: Secret does not existfor keyhub/langfuse-otel.Root cause: the
langfuse-push-otel-keysJob (writes the Langfuse project API keys to Secrets Manager athub/langfuse-otel) is an ArgoCD PostSync hook. PostSync only fires once the entire Application reaches a completed, healthy sync — butlangfuse-hubroutinely sitsOutOfSync/Progressing(the clickhouse/minio/postgres StatefulSets and the keycloak ExternalSecret churn). So the hook never runs,hub/langfuse-otelis never created, and every otel-collector ExternalSecret fails.The keys already exist in-cluster (
langfuse/langfuse-otel-keys) and the IAM role (hub-langfuse-seed-role, secretsmanager write onhub/langfuse-*) + pod identity for thelangfuse-seedSA are all in place — only the push never happens.Fix
Switch the push Job from a
PostSynchook to aSynchook withsync-wave: "5". A Sync hook runs during the sync operation itself (in wave order, after the seed SA + pod-identity in earlier waves), so the keys land in Secrets Manager regardless of whether the StatefulSets have converged.Testing
Rendered via
helm template— Job now carrieshook: Sync+sync-wave: "5". On the live hub the seed SA, pod identity, and IAM role are confirmed present; the k8slangfuse-otel-keyssecret already holds validpk-lf-.../sk-lf-...values.