vm clone substitutes base layers from a different cached image (snapshot's cocoon.json digests ignored)

[tonicmuroq]

Summary

cocoon vm clone produces a runtime disk config that does not match the snapshot manifest's declared base-layer digests. It instead substitutes layers from a different cached image that happens to share the same repo:tag "stem" path. The substitution silently downgrades the cloned VM's lowerdirs from the snapshot's declared N layers to whatever shorter-tag image happens to be cached locally, masking files that exist only in the upper base layers.

Related to but distinct from #37/#38: in our case the base image is present locally at the correct digest, but cocoon still constructs the runtime config from a different image's layer set.

Environment

• Cocoon cluster: cocoonset-gke, vk-cocoon on cocoonset-node-2
• vm-service env: testing
• Hot snapshot: epoch.simular.cloud/simular/ubuntu-hot-testing:v1
• Snapshot baseimage annotation: epoch.simular.cloud/simular/ubuntu:24.04-xface

Evidence

1. Snapshot manifest on epoch declares the correct layer set

cocoon.json blob from the hot-snapshot manifest references 11 base layers:

cocoon-layer0:  b40150c1c2717d... (29 MB)
cocoon-layer1:  3a6844925eb6c8... (444 MB)
cocoon-layer2:  e91c01bdde5a23...
cocoon-layer3:  f930570bc53388...
cocoon-layer4:  e88e723e1e6992...
cocoon-layer5:  f230ba36f6926f...
cocoon-layer6:  65be6c6a51e92d...
cocoon-layer7:  d8e7381f1b6d5d...
cocoon-layer8:  01f018bd59be27...
cocoon-layer9:  a91b26a4acfbe4...
cocoon-layer10: 4f81e1bad188e0...

The snapshot's vm.config layer agrees — 12 disks (11 base + cow.raw).

2. The referenced base manifest at epoch matches

crane manifest epoch.simular.cloud/simular/ubuntu:24.04-xface returns digest sha256:fff6f7a6786e... with exactly those 11 layers.

3. All 11 base-layer .erofs blobs are present on the cloning node

$ ls /var/lib/cocoon/oci/blobs/
b40150c1c2717d...erofs  3a6844925eb6c8...erofs  e91c01bdde5a23...erofs
f930570bc53388...erofs  e88e723e1e6992...erofs  f230ba36f6926f...erofs
65be6c6a51e92d...erofs  d8e7381f1b6d5d...erofs  01f018bd59be27...erofs
a91b26a4acfbe4...erofs  4f81e1bad188e0...erofs

4. The cloned VM's runtime config uses different layers

/var/lib/cocoon/run/cloudhypervisor/<ULID>/config.json of the live cloned VM lists only 6 base disks:

cocoon-layer0: b40150c1c2717d...erofs   (matches new base layer 0)
cocoon-layer1: 3a6844925eb6c8...erofs   (matches new base layer 1)
cocoon-layer2: 51189d853427f2...erofs   ← NOT in new base manifest
cocoon-layer3: 268433ca369440...erofs   ← NOT in new base manifest
cocoon-layer4: bab9e12b5c15d0...erofs   ← NOT in new base manifest
cocoon-layer5: ff10e322f9ef77...erofs   ← NOT in new base manifest

Inside the VM mount confirms a 6-lowerdir overlay (vs. the 11-lowerdir overlay produced by cocoon vm run epoch.simular.cloud/simular/ubuntu:24.04-xface directly on the same node).

5. The four "alien" digests are from a different cached image

$ crane manifest epoch.simular.cloud/simular/ubuntu:24.04-xface-20260513 | jq '.layers[].digest'
sha256:b40150c1c2717d...
sha256:3a6844925eb6c8...
sha256:51189d853427f2...   ← match
sha256:268433ca369440...   ← match
sha256:bab9e12b5c15d0...   ← match
sha256:ff10e322f9ef77...   ← match

simular/ubuntu:24.04-xface-20260513 is yesterday's date-pinned alias of the same image stream. It shares prefix layers (b40150, 3a6844) with today's :24.04-xface and diverges thereafter (6 layers total vs. 11 today). On disk the node has both :24.04-xface and :24.04-xface-20260513 cached.

6. Result

The cloned VM is missing every file added in the upper 5 layers of the snapshot's declared base — in our case the agent watchdog assets (/usr/local/bin/simular-agent-watchdog.sh, /etc/cron.d/simular-pro-agent, /etc/xdg/autostart/simular-pro-agent.desktop, /var/lib/systemd/linger/root). They were present in the bake VM at snapshot time but never end up in cow.raw because they live in the (read-only) base, and at clone time the wrong base layer set is mounted.

Hypothesis

cocoon vm clone does not faithfully reproduce the layer set declared in the snapshot's cocoon.json. It looks like clone resolves base layers by consulting the local image DB at the snapshot's baseimage tag, and the local DB may return a different image whose :tag happened to be cached. With short stems like :24.04-xface colliding with :24.04-xface-20260513, the wrong layer set is selected.

Expected

cocoon vm clone should treat the snapshot's cocoon.json storage_configs[*].path digests as authoritative. If a declared .erofs blob is missing locally, auto-pull (or fail loudly) instead of substituting from an arbitrary local image entry that shares prefix layers.

Repro

1. Build base image A and push as :tag-A (e.g. 6 layers).
2. Pull :tag-A on a cocoon node.
3. Rebuild the image (more steps) and push as :tag (e.g. 11 layers). Both :tag and :tag-A exist in epoch; the node has both in cache.
4. Bake a hot snapshot from :tag and push it.
5. cocoon vm clone <hot-snapshot> on the same node.

Observed: cloned VM mounts 6 lowerdirs from :tag-A, not 11 from :tag. Files added by the upper 5 layers of :tag are gone.

Workaround for our pipeline: write all snapshot-critical assets via the install/warm script so they end up in cow.raw instead of base layers — but this defeats the purpose of layered base images.

[tonicmuroq]

Update — cache cleanup doesn't fix this; the snapshot pin is honored by epoch but ignored at clone time.

Cleaned both cocoon nodes (cocoon image rm for :24.04-xface-20260513 and :24.04-xface plus @sha256:67fef1b5dc4d; deleted the 4 orphan .erofs blobs). Re-ran clone. Same outcome.

What I now have hard evidence for: the snapshot's config artifact (application/vnd.cocoonstack.snapshot.config.v1+json) on epoch correctly pins today's base:

{
  "image": "epoch.simular.cloud/simular/ubuntu:24.04-xface",
  "imageDigest": "sha256:fff6f7a6786e8ffce99060ee3421e3c501a73ae72b3f540c83d51ccff727fcc7",
  "imageType": "oci",
  "imageBlobIds": [
    "01f018bd59be27d7d813d992d8a8fd858c6dbb1b49d3fe015c31187e7b26ff3a",
    "3a6844925eb6c8e04951ba3100c1bda7fd9467a1a6e81e6270665824a1cb4afb",
    "4f81e1bad188e08ba7b5bc460fe08daa22556e7ba5f27f5217ee394d99bb9930",
    "65be6c6a51e92d35d204a2bcf3484bd4bef0003ca69ceb729b1883119414cb31",
    "a91b26a4acfbe4b58fdca98ca8a4ab71e7e3592138c2797e4c6d485217145bff",
    "b40150c1c2717d324cdb17278c8efdfa4dfcd2ffe083e976f0bcedf31115f081",
    "d8e7381f1b6d5d1b77efdba194c1422de497e37dc57005715a7c7055a8ed6054",
    "e88e723e1e6992f3ddd3a7f495d0105aa8b1715fef8567b4ecf06c180284241a",
    "e91c01bdde5a23262a704dbe3573cd421e3819aa25a2beeff987fce11280ddd2",
    "f230ba36f6926f57a6ba28695d1ea7bfd2aea3fb4e5538249c6c2b318e625bf5",
    "f930570bc53388b8c17f6cc996d25fd1505e11b982f16e21f4b9c5dc70f76c18"
  ]
}

This is today's base (sha256:fff6f7a6786e, 11 layers, the one that does contain the missing files). The hot snapshot is correctly anchored to it.

But on a node where today's base is not locally cached, cocoon vm clone --pull of the snapshot does not pull sha256:fff6f7a6786e. The cocoon image ls output right after the clone shows it pulled @sha256:67fef1b5dc4d (yesterday's digest — different blobs, only 6 layers):

oci  epoch.simular.cloud/simular/ubuntu@sha256:67fef1b5dc4d...  1.273GB  2026-05-14 11:49:34  ← pulled by the clone

The resulting runtime config.json uses yesterday's blobs (51189d, 268433, bab9e1, ff10e3 from the 6-layer image) for cocoon-layer2..5, not today's blobs from the snapshot's imageBlobIds. Files in the upper layers of today's base are masked off.

So the clone is not just ignoring vm.config / cocoon.json layer paths — it's ignoring the snapshot config's imageDigest pin and resolving the base from somewhere else entirely. My best guess based on what gets pulled: it's resolving from the snapshot's internal saved state (the cocoon snapshot save time view of the base), and at bake time, after cocoon image pull --force, the immediately-following cocoon vm run somehow still used yesterday's digest. Whatever cocoon writes into the snapshot data file at cocoon snapshot save time wins over the manifest's imageDigest at clone time.

That makes this a bake-side AND clone-side bug:

• Bake-side: after image pull --force the next vm run should consume the freshly-pulled digest, not yesterday's.
• Clone-side: the OCI snapshot manifest's imageDigest should be authoritative — if the snapshot's saved state disagrees, the saved state is stale and should be rejected, not silently used.

Workaround for our pipeline: move all snapshot-critical assets into cow.raw so the base substitution can't strip them. This sidesteps the bug but isn't a real fix.

[CMGS]

现场调查（cocoonset-node-2 / 34.143.214.12）+ 代码审计结论

花了一些时间把 OSS 代码 5 个关口 + 测试机上的残留物证 + epoch 当下 manifest 全部对了一遍。结论：cocoon 代码层没有 "substitution" 路径；issue 报告的现象来自 epoch 上 :24.04-xface tag 曾经短暂漂移到 6-layer manifest，导致 snapshot 自身就是 6-layer 烤出来的，clone 是 verbatim 复刻。

1. 代码层（5 个关口审完，没有任何一条路径能让 snapshot 声明 N 层、实际跑 M 层）

关口	文件位置	行为	能否 substitute
EnsureImage	cmd/core/helpers.go:223	--pull 时按 cmp.Or(ImageDigest, Image) 调 b.Pull()，不返回 storage_configs	❌
ResolveImage	cmd/core/helpers.go:196	从 image DB 读 layer 列表 —— clone 路径不调用，只 createVM 用	❌
cloneAfterExtract	hypervisor/cloudhypervisor/clone.go:33-51	storageConfigs := meta.StorageConfigs（snapshot cocoon.json verbatim）	❌
patchCHConfig	hypervisor/cloudhypervisor/patch.go:34-38	strict count check：len(opts.storageConfigs) != diskCount → error；声明 11 实际 6 会先报错	❌
ImageBlobIDs / GC	hypervisor/clone.go:28 + gc.go	只参与 GC pinning（UsedBlobIDs），不参与路径解析	❌

.erofs blob 全是 content-addressed（hash = 文件名），ValidFile(blobPath) 之后 skip 重写，不可能内容漂移。

2. 现场物证（XEOU75LF32BMGTUIFEIFO5M577 = 当前 v1 snapshot 在节点上的本地副本）

• 11 个正确 layer .erofs 文件全在 /var/lib/cocoon/oci/blobs/
• 4 个 "alien" layer .erofs 也都在，mtime 是 2026-05-14 11:49:22-34 —— 正好对应 vk-cocoon 触发的那次 cocoon vm clone --pull 的 force-pull 时刻
• OCI image DB（/var/lib/cocoon/oci/db/images.json）里两份 image 并存：
  • simular/ubuntu:24.04-xface → fff6f7a6...（11-layer，created_at: 2026-05-14T11:58:59Z）
  • simular/ubuntu@sha256:67fef1b5... → 6-layer（created_at: 2026-05-14T11:49:34Z），tag 已被新版本抢走，降级为 digest-only key 残留
• XEOU snapshot DB record：ImageDigest = sha256:fff6f7a6...，ImageBlobIDs 11 个 hash 全对
• XEOU 的 config.json 和 cocoon.json 都是 12 disks（11 layer + 1 cow），count 一致

3. epoch 当前 ubuntu-hot-testing:v1 工件

manifest created_at: 2026-05-14T12:03:57Z   ← vk-cocoon 12:02 重推之后
cocoon.json blob: 12 storage_configs (11 layers + 1 cow)，全是正确 hash
baseimage annotation: epoch.simular.cloud/simular/ubuntu:24.04-xface  ← 仅 tag，无 digest pin

4. 时间线还原（vk-cocoon journalctl on 34.143.214.12）

T0 之前  epoch :24.04-xface tag 短暂指向 67fef1...（6-layer manifest）
T0       某 VM 用此版本启动并 snapshot 成 ubuntu-hot-testing:v1
11:30:33 tonicmuroq `cocoon image pull ubuntu-hot-testing:v1` → 本地 snapshot cocoon.json 自带 6 layers
11:45:37 issue #46 创建（runtime 6-disk vs epoch 当下 :24.04-xface 11-layer 对不上）
11:49:22 vk-cocoon `cocoon vm clone --pull --on-demand ubuntu-hot-testing:v1`
11:49:34 force-pull → OCI DB 写入 67fef1...(6-layer)，4 个 alien .erofs 落盘
11:58:59 `cocoon image pull --force :24.04-xface` → fff6f7a6...(11-layer) 覆盖 tag
12:02:10 `cocoon snapshot export` → epoch 上的 v1 manifest 被刷新为 11-layer 版
12:13:22 本地 XEOU snapshot 完成（现状）

5. 根因 ≠ "cocoon 替换 base"，而是 epoch tag drift

issue 描述里的现象其实是：

• snapshot 自身就是 6-layer 烤的（因为烤的时刻 :24.04-xface tag 在 epoch 上指向 6-layer manifest）
• clone 忠实复刻了 snapshot 声明的 6 个 layer → runtime config.json 6 disks
• reporter 看到的是「snapshot 当时的 6-layer」 vs 「epoch 现状（已修复后）的 11-layer」 —— 这两份本来就不是同一个版本
• 现场残留的 4 个 alien .erofs 文件不是 substitution 的产物，是 vk-cocoon 在 11:49 那次 --pull 时按本地缺失的 digest 跑 b.Pull() 拿回来的

6. 建议的加固（即使不是 cocoon 代码 bug，仍可降低复发）

1. cocoon snapshot save 时持久化 base image digest，annotation 直接写成 image@sha256:... 而非 :tag —— 让 snapshot 自我闭环、不依赖 tag 漂移
2. cocoon snapshot inspect 暴露 baseimage digest，让用户可以 verify "我的 snapshot 用的是哪个 base 版本"
3. vk-cocoon push snapshot 时 annotation 写 digest-pinned ref —— 同 (1)，但在 vk-cocoon 侧也加
4. epoch registry 端：同 tag 不允许 push 不同 manifest（或要求 tag@sha256 形式覆盖），防 tag drift —— 这条最一劳永逸

7. 复现/验证步骤（任何人都可以跑）

# 在 cocoonset-node-2 (34.143.214.12) 上
# 看 OCI image DB 里两份并存的 ubuntu image：
python3 -c "
import json
d = json.load(open('/var/lib/cocoon/oci/db/images.json'))
for n, i in d['Images'].items():
    if 'ubuntu' in n and 'xface' in n or '67fef1' in n:
        print(n, '->', len(i['layers']), 'layers,', i['created_at'])
"

# 看 4 个 alien .erofs 的 mtime（11:49:22-34 = force-pull 时刻）：
ls -la /var/lib/cocoon/oci/blobs/{51189d,268433,bab9e1,ff10e3}*.erofs

# 看当前 XEOU snapshot 的 11+1 disks 跟 cocoon.json 一致：
python3 -c "
import json
c = json.load(open('/var/lib/cocoon/snapshot/localfile/XEOU75LF32BMGTUIFEIFO5M577/config.json'))
m = json.load(open('/var/lib/cocoon/snapshot/localfile/XEOU75LF32BMGTUIFEIFO5M577/cocoon.json'))
print('config.json disks:', len(c['disks']))
print('cocoon.json storage_configs:', len(m['storage_configs']))
"