secrets: support internal, name-keyed secrets alongside user secrets - #37996
Draft
jasonhernandez wants to merge 2 commits into
Draft
secrets: support internal, name-keyed secrets alongside user secrets#37996jasonhernandez wants to merge 2 commits into
jasonhernandez wants to merge 2 commits into
Conversation
Extend SecretsController and SecretsReader with internal secrets, keyed by name rather than by CatalogItemId. These hold system-generated credentials, such as the upcoming CTP transport keys, that have no corresponding catalog item. The two namespaces are disjoint in every backend: Kubernetes uses an "internal-" infix next to the existing "user-managed-" one, the process orchestrator prefixes secret files with "internal-", and AWS Secrets Manager gets an "internal-" infix after the deployment prefix. Internal secrets are excluded from list(), whose parsers only recognize user secret names. Names are validated against a conservative alphabet (lowercase alphanumerics and dashes) that is valid in all backends and cannot traverse paths. The process orchestrator's list() previously errored on any file in the secrets directory whose name did not parse as a CatalogItemId. It now skips such files, matching the documented behavior of the other backends. The caching reader deliberately does not cache internal secrets. They are read rarely, typically once at process startup, so caching would only extend the lifetime of key material in memory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Callers bootstrapping credentials must distinguish a secret that does not exist yet (generate and persist a new one) from a transient read failure (fail and retry). Conflating the two could cause a caller to regenerate and overwrite live key material. Return Option instead of an untyped error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
PR 2 of 3 toward encrypting the cluster transport protocol (see #37995 for the stack overview). The controller needs to distribute system-generated credentials (a per-environment CA, per-replica TLS keys) to clusterd processes. The right channel already exists — environmentd writes secrets through
SecretsControllerand every clusterd boots with aSecretsReader— but both traits are keyed byCatalogItemId, and transport credentials are not catalog items.This PR adds internal secrets: name-keyed secrets for system-generated credentials, alongside the id-keyed user secrets. It is independent of #37995 and reviewable standalone; the wiring PR depends on both.
Design
ensure_internal(name, contents)/delete_internal(name)onSecretsController,read_internal(name)onSecretsReader.{prefix}internal-{name}next to the existing{prefix}user-managed-{id}. Process orchestrator:internal-{name}files next to bare{id}files. AWS:{prefix}internal-{name}next to{prefix}{id}. Internal secrets are excluded fromlist(), whose parsers only recognize user secret names.validate_internal_secret_name): lowercase alphanumerics and dashes only, so names are valid in all backends and cannot traverse paths. Enforced at everyensure_internalentry point.Behavior changes
list()previously errored on any file in the secrets directory whose name did not parse as aCatalogItemId. It now skips such files, matching the documented behavior (and the Kubernetes backend's behavior) of ignoring unrecognized objects.ensure/delete/readbodies in the Kubernetes and AWS backends moved into name-keyed helpers shared with the internal variants; no functional change beyond the above.Testing
mz-secrets: unit tests for the in-memory controller round-trip (including namespace disjointness with a user secret and idempotent deletion) and for name validation accept/reject cases.mz-orchestrator-process: new integration test covering round-trip, overwrite, idempotent delete,list()skipping internal files instead of erroring, and rejection of path-traversal names.🤖 Generated with Claude Code