feat: embed Linux hotfix scripts in ANC

[Devinwong]

What this PR does / why we need it:

Moves selected Linux provisioning-hotfix scripts from CRP CustomData into patched aks-node-controller packages. This avoids making script size part of the long-term 64 KB CustomData constraint.

flowchart TD
    Script["Changed hotfixable script"] --> Assets["Generate embedded payload<br/>and retained CRP fallback"]
    Assets --> Pointer["Write version + scripts_version pointer"]
    Pointer --> Build["Build patched ANC with embedded payload"]
    Build --> Publish["Publish package to PMC"]
    Publish --> ABSvc["ABSvc emits existing pointer<br/>and CRP payload"]

    ABSvc --> CRP["VHD-baked ANC applies CRP scripts"]
    CRP --> Install["Attempt PMC install"]
    Install --> Result{"Install succeeds?"}

    Result -->|Yes| Patched["Patched ANC validates and<br/>applies embedded scripts"]
    Patched --> Provision["Run normal provisioning"]

    Result -->|No| Degraded["Degraded: patched ANC unavailable"]
    Degraded --> Legacy["Provision with CRP-applied scripts"]

    classDef owned fill:#ffedd5,stroke:#ea580c,stroke-width:3px,color:#111827
    classDef shared fill:#dbeafe,stroke:#2563eb,stroke-width:2px,color:#111827
    classDef fallback fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#111827
    classDef degraded fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#111827

    class Script,Assets,Pointer,Build,Patched owned
    class Publish,ABSvc,Install,Result,Provision shared
    class CRP,Legacy fallback
    class Degraded degraded

Loading

Orange: changed by this PR. Blue: existing/shared flow. Green: retained CRP fallback. Red: degraded hotfix outcome.

• Generates an ANC-owned manifest and embeds only selected hotfix scripts without importing the root AgentBaker parts module.

  • If there is an active hotfix, manifest.json will look like:

    {
      "schema_version": 1,
      "entries": [
        {
          "source": "cse_config.sh",
          "embedded_path": "payloads/cse_config.sh",
          "destination": "/opt/azure/containers/provision_configs.sh",
          "mode": "0744",
          "platforms": [
            "ubuntu"
          ]
        }
      ]
    }

• Applies validated payloads before normal provisioning using idempotent, replace-only, transactional writes. Missing destinations are skipped to preserve template gates.

• Retains CRP nodecustomdata.yml / scripts_version delivery during phase 1. Embedded bytes take precedence when applicable; failures remain fail-open to the existing scripts.

Pointer activation, PMC publication orchestration, faster ANC download with package SHA-256 verification, and CRP fallback retirement are separate work.

Validation: generator unit tests, full ANC tests, readonly build, golangci-lint, make generate, shellcheck, and generated-churn checks.

Which issue(s) this PR fixes:

N/A

[github-actions]

Windows Unit Test Results

  3 files   12 suites   52s ⏱️
389 tests 389 ✅ 0 💤 0 ❌
392 runs  392 ✅ 0 💤 0 ❌

Results for commit f4a3790.

♻️ This comment has been updated with latest results.

[aks-node-assistant]

🔍 AgentBaker Gate Detective — Build 175202979 · PR #9101

4 jobs failed — 1 new signature + 3 known flaky signatures (not caused by this PR).

---

🆕 build2604minimalgen2containerd — NEW SIGNATURE

Wiki: linux-vhd-prgate-ubuntu2604-minimal-testscan-bcc-diskusage-exit2

Field	Value
TL;DR	Packer build succeeded (SIG captured), but TestScan artifact collection fails; bcc-tools-installation.log + disk-usage.txt absent; script exits 2, 3 retries.
First failing step	Script failed exit code: 2 (3 retries at 1s/4s/9s — all fail)
Primary error	Path does not exist: /mnt/vss/_work/1/s/bcc-tools-installation.log
Likely root cause	TestScan artifact download step fails post-Packer-build for ubuntu2604-minimal. SIG version confirmed captured → Packer did NOT fail. Consistent across 3 retries → not transient.
Classification	VHD/Packer/post-scan artifact collection failure — not caused by this PR
Confidence	80%
Strongest alternative	linux-modules-extra-azure-lts-26.04 kernel package issue (prev cycle) now manifesting at TestScan scan-VM boot stage rather than Packer stage.
Recommended action	Node Lifecycle VHD team: investigate vhd_scanning_download_and_delete_blobs step for 2604minimal; check whether scan VM boots cleanly.
Cross-PR evidence	Same signature in build 175160408 (PR #9130, renovate acr-credential-provider) — both PRs unrelated to 2604 minimal.

---

✅ Known Flaky Signatures (pre-existing, not caused by this PR)

Job	Signature	Repair Item	Count
build2204gen2containerd	linux-vhd-prgate-cis-ubuntu2204-gen2-containerd-6131-pass-fail	#38829101 (New)	37 builds
build2204arm64gen2containerd	linux-vhd-prgate-cis-ubuntu2204-arm64-gen2-containerd-6131-pass-fail	#39004382 (New)	12 builds
build2404gen2containerd	linux-vhd-prgate-cis-ubuntu2404-gen2-containerd-6141-pass-fail	#38529622 (Closed — recurred post-fix)	29 builds

⚠️ CIS 6141 repair item #38529622 was Closed but failure recurred in this build — VHD team should review.

Posted by Clawpilot AgentBaker Linux Gate Detective Watcher.

[Devinwong]

Free feel to review thoroughly but not merge yet, because once it's merged and released, the script hotfix will completely switch to ANC (embedded) release. We want to make sure everything is done before switching. E.g: RP e2e test, per-hotfix document.