Skip to content

feat(Kind): add AddManifest resource for kubectl apply after cluster ready - #1481

Draft
tamirdresher wants to merge 1 commit into
CommunityToolkit:mainfrom
tamirdresher:feat/kind-with-manifest
Draft

feat(Kind): add AddManifest resource for kubectl apply after cluster ready#1481
tamirdresher wants to merge 1 commit into
CommunityToolkit:mainfrom
tamirdresher:feat/kind-with-manifest

Conversation

@tamirdresher

@tamirdresher tamirdresher commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Motivation

For dev-loop scenarios that run against a Kind cluster during F5 (operators, controllers, admission webhooks, or in-source contributions to an OSS Kubernetes project) the AppHost often needs to apply a set of CustomResourceDefinition, Namespace, ServiceAccount, ClusterRole, ClusterRoleBinding, or ConfigMap objects to the cluster after the cluster reaches Running but before dependent resources start.

Today CommunityToolkit.Aspire.Hosting.Kind handles the Helm side of this via cluster.AddHelmChart(...), but if you just have static YAML you have to hand-roll a lifecycle hook in the AppHost. This PR adds the natural equivalent of AddHelmChart for the kubectl apply case.

This adds runtime manifest apply (at F5 / aspire start). It is distinct from Aspire.Hosting.Kubernetes's PublishAsKubernetesService(r => r.AdditionalResources.Add(...)), which is build-time (baked into a generated Helm chart at aspire publish). Both have legitimate use cases; this PR fills the runtime gap for local Kind clusters.

What this adds

A new resource + extension API surface, all public APIs exported for Aspire tooling:

var cluster = builder.AddKindCluster("mycluster");

// Single file
cluster.AddManifest("crds", "./manifests/crds.yaml");

// Directory, recursive, into a namespace
cluster.AddManifest("platform", "./manifests/platform")
    .WithRecursive()
    .WithNamespace("platform");

// Kustomize overlay directory: detected at apply time and applied with kubectl apply -k
cluster.AddManifest("overlay", "./manifests/overlays/dev");

// Server-side apply with a stable field manager
cluster.AddManifest("operator", "./manifests/argocd/install.yaml")
    .WithServerSideApply(forceConflicts: true)
    .WithFieldManager("aspire-apphost");

// Inline content, applied through stdin without temp files
cluster.AddManifestFromContent("demo-ns", """
    apiVersion: v1
    kind: Namespace
    metadata:
      name: aspire-demo
    """);

// Downstream resources can wait for the apply to succeed
var crds = cluster.AddManifest("crds", "./manifests/crds.yaml");

builder.AddContainer("my-operator", "my-org/operator")
    .WithReference(cluster)
    .WaitFor(crds);

Design

Mirrors the existing Kind deployed-resource pattern and K3s manifest semantics while keeping Kind's host-side kubectl execution model:

Layer Helm equivalent New file
Resource KindHelmChartResource K8sManifestResource : KindDeployedResource
CLI orchestration HelmManager KubectlManager (wraps IProcessRunner, internal static argument builders for testability)
Extensions KindHelmChartResourceBuilderExtensions KindManifestResourceBuilderExtensions

Lifecycle callback matches AddHelmChart step-for-step:

  1. WaitForResourceAsync(parent.Name, Running)
  2. PublishAsync(BeforeResourceStartedEvent)
  3. PublishUpdateAsync(Starting)
  4. try { KubectlManager.ApplyAsync -> PublishUpdateAsync(Running) }
  5. catch { PublishUpdateAsync(FailedToStart) -> rethrow }

KubectlManager.ApplyAsync now probes kubectl cluster-info before apply so the Kind resource state and Kubernetes API reachability are both satisfied before manifests are applied.

New public API

Method Maps to
AddManifest(name, manifestPath) kubectl apply -f <path> --kubeconfig=<cluster kubeconfig> or kubectl apply -k <path> for Kustomize directories
AddManifestFromContent(name, content) kubectl apply -f - --kubeconfig=<cluster kubeconfig> with content on stdin
WithNamespace(ns) (inherited from KindDeployedResource) --namespace <ns> (does not create the namespace for manifests)
WithRecursive() --recursive for normal file/directory applies; ignored with a warning for Kustomize/stdin
WithServerSideApply(forceConflicts: false) --server-side (+ --force-conflicts)
WithFieldManager(name) --field-manager <name>
WithApplyTimeout(timeout) Bounded kubectl apply execution time (default 5 minutes)
WithCrdWaitTimeout(timeout) CRD Established wait timeout (default 5 minutes)
WithCrdWaitBehavior(behavior) Fail by default; BestEffort logs a warning and continues

Round 2 changes

After review against CommunityToolkit.Aspire.Hosting.K3s's AddK8sManifest pattern, this branch also:

  1. Renamed the resource type from KindManifestResource to K8sManifestResource and updated the dashboard resource type label to K8s Manifest / K8s Kustomize.
  2. Resolved relative AddManifest paths against builder.ApplicationBuilder.AppHostDirectory before passing them to kubectl.
  3. Auto-detected Kustomize directories and used kubectl apply -k; WithRecursive() is ignored with a warning for Kustomize.
  4. Parsed successful kubectl apply output for CRDs and waited for applied CRDs to reach Established.
  5. Added AddManifestFromContent(name, content), implemented with kubectl apply -f - and stdin support in the internal IProcessRunner abstraction.

Round 4 — squad review fixes

Addressed the 5-agent review findings:

  1. Removed the default manifest workload health check; manifest resource state now reflects apply success/failure without an empty selector.
  2. Added [AspireExport] to K8sManifestResource so the public API export claim is accurate.
  3. Made CRD waiting fail by default, with .WithCrdWaitTimeout(...) and .WithCrdWaitBehavior(CrdWaitBehavior.BestEffort) for opt-in best-effort behavior.
  4. Added a kubectl cluster-info API-reachability retry loop before kubectl apply.
  5. Added .WithApplyTimeout(...) and cancellation for hung kubectl apply calls.
  6. Moved Kustomize detection to apply time and made marker detection case-insensitive, including Kustomization / Kustomization.yml.
  7. Redacted --kubeconfig=<path> from debug command logs.
  8. Fixed the README operator keyword example by renaming it to operatorContainer.
  9. Expanded README coverage for kubectl prerequisites, namespace behavior, no URL fetch support, server-side apply conflict warnings, cluster-admin/security scope, path safety, stdin memory, and Kubernetes documentation links.
  10. Expanded the Kind example AppHost and manifest content to demonstrate recursive apply, SSA/field manager, namespace, best-effort CRD waiting, and downstream WaitFor(manifestResource).

Round 5 — verification-pass fixes

Addressed the final verification-pass findings:

  1. Exported CrdWaitBehavior for Aspire tooling.
  2. Redacted both --kubeconfig=<path> and --kubeconfig <path> debug-log forms.
  3. Changed cluster-info retries to use a real wall-clock budget with per-probe cancellation.
  4. Validated apply/CRD wait timeout bounds, rounded sub-second values up to 1s, and capped values above 1 hour.

Tests

xUnit v3 coverage in KindManifestTests.cs / AddKindClusterTests.cs now includes:

  • Resource creation, parent wiring, name/path behavior, and inline content registration
  • Null-guard tests for builder, name, manifestPath/content, and fieldManager
  • AppHost-relative path resolution and absolute-path preservation
  • Kustomize detection for lowercase and capitalized marker variants, including apply-time detection
  • Property-set tests for WithRecursive, WithServerSideApply(forceConflicts), WithFieldManager, WithApplyTimeout, WithCrdWaitTimeout, and WithCrdWaitBehavior
  • Default-value tests for recursive/server-side/force-conflicts, field manager, namespace, apply timeout, and CRD wait behavior
  • KubectlManager.CreateApplyArguments, CreateWaitArguments, and cluster-info retry behavior
  • Apply timeout cancellation
  • Default fail-fast and opt-in best-effort CRD wait behavior
  • kubeconfig path redaction in debug logs

Full CommunityToolkit.Aspire.Hosting.Kind.Tests project passes locally: 174 / 174.

Real-world usage

Upstreamed from https://github.com/tamirdresher/aspire-argocd-dev-loop, which spins up an argo-cd F5 inner loop against a Kind cluster and uses AddManifest to install the ArgoCD manifests before the argocd-server and repo-server projects start. Prior to this API the sample carried a hand-rolled OnBeforeStart hook with its own Process.Start("kubectl", ...).

Open questions for reviewers

No blockers remaining from prior review rounds.

Deferred: naming consistency between this integration's AddManifest and CommunityToolkit.Aspire.Hosting.K3s's AddK8sManifest. Both are [AspireExport]-ed public API in separate packages; unifying would be a breaking change for K3s consumers. Happy to align on either name if maintainers have a preference.

Follow-up

I have a separate small PR queued: WithPortMapping(hostPort, containerPort) as a fluent shorthand over the existing WithKindConfig(Action<KindConfigModel>) for exposing NodePort services on 127.0.0.1. Shipping it separately to keep this PR focused on the new resource type.

Docs

README section updated in src/CommunityToolkit.Aspire.Hosting.Kind/README.md: usage examples for single file, recursive directory, Kustomize auto-detect, inline content, server-side apply, CRD wait behavior, namespace behavior, security notes, and the API reference table entries.

Checklist

  • Design mirrors existing resource patterns (KindHelmChartResource and K3s K8sManifestResource semantics)
  • Public API export surface is covered where Aspire supports [AspireExport]
  • Tests pass locally (174 / 174)
  • README updated
  • Round 4 squad review fixes applied
  • Round 5 verification-pass fixes applied

Follow-on docs

learn.microsoft.com/dotnet/aspire/community-toolkit/hosting-kind lives in MicrosoftDocs/aspire-docs, outside this fork's tree, and will need a matching update for AddManifest. That can be handled as a separate PR after this merges.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.sh | bash -s -- 1481

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/CommunityToolkit/Aspire/main/eng/scripts/dogfood-pr.ps1) } 1481"

@tamirdresher
tamirdresher force-pushed the feat/kind-with-manifest branch 4 times, most recently from 5180420 to d708bd3 Compare July 27, 2026 06:06
…ready

Adds Kubernetes manifest resources for Kind clusters, aligned with the existing K3s manifest pattern while keeping Kind's host-side kubectl execution model.

Public API surface:

  cluster.AddManifest(name, manifestPath)
         .WithNamespace(ns)
         .WithRecursive()
         .WithServerSideApply(forceConflicts: false)
         .WithFieldManager("aspire-apphost")
         .WithApplyTimeout(TimeSpan.FromMinutes(5))
         .WithCrdWaitTimeout(TimeSpan.FromMinutes(5))
         .WithCrdWaitBehavior(CrdWaitBehavior.Fail)

  cluster.AddManifestFromContent(name, content)
         .WithNamespace(ns)
         .WithServerSideApply(forceConflicts: false)
         .WithFieldManager("aspire-apphost")

Key behavior:

  - K8sManifestResource extends KindDeployedResource(name, parent) and is exported for Aspire tooling.
  - AddManifest resolves relative paths against AppHostDirectory; Kustomize detection happens at apply time and supports case-insensitive marker variants.
  - AddManifestFromContent applies inline YAML via kubectl apply -f - using stdin, avoiding temporary files.
  - KubectlManager probes kubectl cluster-info before apply, bounds apply with ApplyTimeout, and fails CRD waits by default with an opt-in best-effort mode.
  - WithRecursive is honored for file/directory applies and ignored with a warning for Kustomize or stdin applies.
  - Manifest resources rely on Running/FailedToStart state rather than a default empty-selector workload health check.

Downstream resources can WaitFor the manifest resource so they only start once kubectl apply succeeds:

  var crds = cluster.AddManifest("crds", "./manifests/crds.yaml");

  builder.AddContainer("my-operator", "my-org/operator")
      .WithReference(cluster)
      .WaitFor(crds);

Manifests persist with the cluster - deleted with session clusters, retained with persistent clusters.

Tests cover resource registration, parent wiring, null guards, AppHost-relative path resolution, absolute-path preservation, Kustomize detection, stdin apply argument shape and input piping, recursive ignore behavior, cluster-info retry, apply timeout cancellation, CRD wait argument shape, and default/best-effort CRD wait behavior.

Closes: none (net new)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7c8c8109-2ae6-407b-8740-8c3018abe8f1
@tamirdresher
tamirdresher force-pushed the feat/kind-with-manifest branch from d708bd3 to 5c80dc4 Compare July 27, 2026 07:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant