feat(operator): allow pod scheduling on the proxy Deployment via resourceOverrides - #6183
Open
talshechanovitz wants to merge 3 commits into
Open
Conversation
…urceOverrides Adds nodeSelector, tolerations and affinity to resourceOverrides.proxyDeployment and applies them to the proxy pod spec, matching the scheduling control spec.podTemplateSpec already gives the MCP server pod. Without this the proxy is repelled by a dedicated pre-warmed pool's taint and lands on a cold node, so the server is warm but the proxy is not and the cold-start cost is paid anyway. The operator sets no proxy scheduling of its own, so the overrides are authoritative rather than merged. Refs stacklok#5879
…ling drift The helper had been inserted between deploymentForMCPServer's doc comment and the function, hijacking both the docstring and the nolint:gocyclo directive. deploymentNeedsUpdate now compares the proxy pod's scheduling fields against the shared proxyDeploymentScheduling helper (equality.Semantic, matching the ImagePullSecrets precedent) — without this, editing the overrides on an existing MCPServer silently changed nothing. Adds a drift test mirroring TestMCPServerDeploymentNeedsUpdate_ImagePullSecretsDrift.
MCPServer and MCPRemoteProxy share the ResourceOverrides type, so the three scheduling fields already appeared on both CRDs. Wiring only MCPServer left MCPRemoteProxy accepting them and silently ignoring them, so this honours them there as well (issue stacklok#5879, option 2). proxyDeploymentScheduling now takes ResourceOverrides directly and a shared proxySchedulingNeedsUpdate predicate serves both controllers' drift checks. MCPRemoteProxy's podTemplateSpec is applied to the proxy Deployment itself, so in that branch scheduling is compared against the rebuilt Deployment rather than the overrides alone — comparing against overrides reported false drift on a podTemplateSpec-supplied nodeSelector. MCPServer needs no equivalent: its podTemplateSpec becomes a runtime patch for the MCP pod and never touches the proxy PodSpec.
talshechanovitz
requested review from
ChrisJBurns,
JAORMX,
blkt,
jerm-dro,
jhrozek,
rdimitrov,
reyortiz3 and
tgrunnagle
as code owners
August 4, 2026 10:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5879.
What
Adds
nodeSelector,tolerationsandaffinitytoresourceOverrides.proxyDeploymentand applies them to the proxy pod, so the proxy can be steered onto the same nodes as the MCP server pod.Without it the proxy is repelled by a dedicated pre-warmed pool's taint, lands on a cold general node, and the ~40s node boot is paid anyway — the server is warm but the proxy isn't.
Diff shape: +335 hand-written, +7,827 generated. All the generated volume is
corev1.Affinity's schema expanding into four CRD files; regenerating on a clean tree produces no churn, so it's all attributable here.Scope — please sanity-check this call
@ChrisJBurns asked to keep this to
MCPServer. That turns out not to be cleanly possible:ProxyDeploymentOverridesis reached viaResourceOverrides, whichMCPRemoteProxyalso embeds (mcpremoteproxy_types.go:153), so the fields land on both CRDs regardless.This PR wires both, rather than leaving
MCPRemoteProxyaccepting fields it ignores.One wrinkle worth knowing before you weigh in — the two resources aren't symmetric:
MCPServerpodTemplateSpecis marshalled into a runtime patch for the MCP pod and never reaches the proxy PodSpecMCPRemoteProxypodTemplateSpecis applied to its proxy Deployment directlyAnd where both are set on
MCPRemoteProxy,podTemplateSpecwins silently (verified:nodeSelectorfrompodTemplateSpecoverwrites the override).So there's a case for splitting the type instead, giving
MCPServerits own overrides type so scheduling never appears onMCPRemoteProxy. I didn't, for one reason: it needs two new API types plus retyping ~24ResourceOverridesliterals across ~10 files, which makes the review considerably larger than the alternative you were trying to avoid. Happy to do it if you'd rather have the clean separation — just say and I'll push it.How
ProxyDeploymentOverrides(mcpserver_types.go)proxyDeploymentScheduling(*ResourceOverrides)— shared helper, so both controllers behave identically from one implementationdeploymentForMCPServeranddeploymentForMCPRemoteProxyproxySchedulingNeedsUpdate— shared drift predicate, used by bothdeploymentNeedsUpdatepathstask operator-generate+task operator-manifestsSimpler than I sketched on the issue: both operators build their proxy PodSpec directly, so no runtime or
applyPodTemplatePatchchange is needed.The overrides are authoritative, not merged — neither controller sets proxy scheduling of its own today (every
NodeSelector/Tolerations/Affinitymatch undercmd/thv-operatorisSessionAffinity, a Service field).Drift detection
Worth calling out because the first version of this PR got it wrong:
deploymentNeedsUpdatecompared no scheduling fields, so editing the overrides on an existing CR silently did nothing until an unrelated field changed. Create worked, update didn't.Both controllers now compare, following the
ImagePullSecretsprecedent (equality.Semantic.DeepEqual, so nil ≡ empty and an unset override isn't perpetual drift).The comparison has to differ per resource, which
TestMCPRemoteProxyPodTemplateSpecDriftDetectioncaught immediately: forMCPRemoteProxyscheduling may come frompodTemplateSpec, so it compares against the rebuilt Deployment; comparing against the overrides alone reported false drift — an infinite reconcile loop.MCPServerneeds no equivalent, since itspodTemplateSpecnever touches the proxy PodSpec.Tests
TestDeploymentForMCPServer_ProxyDeploymentScheduling— each field alone, all three together, and no overrides asserting scheduling stays unset so the default path is provably unchangedTestMCPServerDeploymentNeedsUpdate_ProxySchedulingDrift— spec-gains-value, spec-cleared-with-stale-deployment, nil-vs-empty-is-not-driftTestMCPRemoteProxySchedulingOverridesAndPodTemplateSpec— overrides applied and stable, clearing them detected, andpodTemplateSpec-supplied scheduling not mistaken for driftgo test ./cmd/thv-operator/...— 21 packages pass. Thetest-integration/...suites fail locally on missing envtest binaries (kubebuilder/bin/etcd): 23 failures both with this change and on a cleanmain, so environmental here. I couldn't rungolangci-lintlocally either — the installed build targets go1.23 and the repo is on 1.26 — so CI will be the first real lint pass;gofmtis clean.Follow-up
VirtualMCPServer, as agreed on the issue.