Skip to content

Add OCI image support #191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
henrybear327 wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

Add OCI image support #191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c218af2
Extract elfuse_launch and add guest launch flags
henrybear327 Jul 8, 2026
11f9b14
Emulate procfs and devices for container guests
henrybear327 Jul 8, 2026
7a50f4c
Add elfuse-container CLI for OCI images
henrybear327 Jul 9, 2026
4684f2b
Add macOS sparsebundle rootfs with COW clones
henrybear327 Jul 9, 2026
79d4376
Add OCI lifecycle commands and reachability GC
henrybear327 Jul 9, 2026
50ebdc1
Add OCI layout conformance checks and Linux CI
henrybear327 Jul 9, 2026
f8b865c
Document OCI image support
henrybear327 Jul 9, 2026
2bf9787
Expand elfuse-container test coverage
henrybear327 Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 198 additions & 0 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -551,6 +551,15 @@ jobs:

ls -l "$ROSETTA"

- name: Set up Go
# Only the release leg runs the OCI run smoke below, which needs the Go
# toolchain to build build/elfuse-container.
if: ${{ matrix.run_matrix }}
uses: actions/setup-go@v6
with:
go-version-file: go.mod
cache: true

- name: Build elfuse
# make does not track EXTRA_CFLAGS changes, so an object built for one
# sanitizer must not be reused for another. Checkout already wipes
Expand Down Expand Up @@ -580,6 +589,21 @@ jobs:
run: |
make EXTRA_CFLAGS="$EXTRA_CFLAGS" ${{ matrix.check_target }}

- name: OCI run smoke (pull -> sparsebundle -> COW clone -> HVF boot)
# The only leg that exercises the full default `run` path end to end:
# pull an image, provision the case-sensitive sparsebundle, COW-clone it,
# boot the guest under HVF, and propagate its exit status. Release leg
# only (sanitizer legs skip the fixture/qemu-heavy paths).
if: ${{ matrix.run_matrix }}
env:
ELFUSE_OCI_STORE: ${{ runner.temp }}/oci-store
run: |
set -euo pipefail
make build/elfuse-container
out="$(build/elfuse-container run alpine:3 /bin/echo elfuse-container-ci-ok)"
printf 'guest said: %s\n' "$out"
printf '%s\n' "$out" | grep -q elfuse-container-ci-ok

- name: Test matrix
if: ${{ matrix.run_matrix }}
run: |
Expand Down Expand Up @@ -611,3 +635,177 @@ jobs:
else
echo "No externals/test-fixtures to save"
fi

# OCI image-layout conformance + cross-tool interop on Linux. elfuse-container
# is pure Go (no Hypervisor.framework), so pull/inspect/unpack and the
# conformance tests run in hosted CI; only `run` needs HVF and is excluded.
# The on-disk store is the contract: it must be a valid OCI image-layout that
# crane/skopeo/umoci can read and that agrees with registry truth.
oci-conformance:
name: OCI conformance + interop (Linux)
runs-on: ubuntu-24.04
timeout-minutes: 15
env:
# Pinned to elfuse-container's go-containerregistry version so the crane
# CLI reads layouts with the same schema handling it writes with.
GGCR_VERSION: v0.21.7
# umoci release tag for the interop gate; built from a checkout below.
UMOCI_VERSION: v0.6.0
steps:
- name: Checkout
uses: actions/checkout@v7

- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: go.mod
cache: true

- name: Install jq + skopeo
# skopeo reads our layout via the oci: transport. CI treats it as part
# of the conformance gate; local runs may omit it and get a skipped
# interop section from scripts/oci-interop.sh.
run: |
set -euo pipefail
sudo apt-get update
sudo apt-get install -y jq skopeo

- name: Install crane + umoci from source
# crane (registry-truth comparison) and umoci (layout parse) are Go
# tools; install crane at elfuse-container's ggcr version where applicable.
run: |
set -euo pipefail
go install github.com/google/go-containerregistry/cmd/crane@${GGCR_VERSION}
# `go install pkg@version` refuses umoci: its go.mod carries replace
# directives. Build from a pinned checkout instead, where replace
# directives apply; a read-only `umoci list --layout` conformance
# check needs nothing newer.
git clone --quiet --depth 1 --branch "$UMOCI_VERSION" \
https://github.com/opencontainers/umoci.git "$RUNNER_TEMP/umoci"
(cd "$RUNNER_TEMP/umoci" && \
go build -o "$(go env GOPATH)/bin/umoci" ./cmd/umoci)
echo "$(go env GOPATH)/bin" >>"$GITHUB_PATH"

- name: Build elfuse-container
# Pure Go target; does not require the C toolchain or HVF.
run: make build/elfuse-container

- name: Go fmt + vet (Linux and darwin cross-check)
# gofmt + vet gate. vet also runs under GOOS=darwin so the sparsebundle
# files (csrun.go, sparsebundle.go, cache_darwin.go) that never build on
# this Linux runner are still compile- and vet-checked here.
run: |
set -euo pipefail
out="$(gofmt -l cmd/elfuse-container)"
if [ -n "$out" ]; then
echo "::error::gofmt needed on:"; echo "$out"; exit 1
fi
go vet ./cmd/elfuse-container/
GOOS=darwin GOARCH=arm64 go build -o /dev/null ./cmd/elfuse-container/
GOOS=darwin GOARCH=arm64 go vet ./cmd/elfuse-container/

- name: CLI lifecycle smoke (pull/list/inspect/unpack/rmi/prune)
# Exercises the built binary through the same user-facing flow that the
# Go unit tests model in-process. `run` itself remains covered by Go
# orchestration tests here and by macOS/HVF runtime jobs.
run: |
set -euo pipefail
bin="build/elfuse-container"
store="$(mktemp -d)"
err="$store/rmi.err"

"$bin" version
"$bin" pull --store "$store" alpine:3
"$bin" inspect --store "$store" --json alpine:3 \
| jq -e '(.os == "linux") and (.architecture == "arm64")' >/dev/null
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
"$bin" unpack --store "$store" alpine:3

json="$("$bin" images --store "$store" --json)"
full_digest="$(printf '%s\n' "$json" | jq -er '.[0].digest')"
cache="$store/rootfs/sha256/${full_digest#sha256:}"
test -e "$cache/bin/sh"

table="$("$bin" list --store "$store")"
printf '%s\n' "$table"
short_digest="$(printf '%s\n' "$table" | awk 'NR == 2 {print $2}')"
test -n "$short_digest"

if "$bin" rmi --store "$store" alpine:3 2>"$err"; then
echo "::error::rmi by ref succeeded despite an unpacked cache without --force"
exit 1
fi
grep -q 'cache' "$err"

"$bin" rmi --store "$store" --force alpine:3
test -z "$("$bin" list --store "$store")"

"$bin" pull --store "$store" alpine:3
json="$("$bin" images --store "$store" --json)"
full_digest="$(printf '%s\n' "$json" | jq -er '.[0].digest')"
manifest_path="$store/blobs/sha256/${full_digest#sha256:}"
layer_hex="$(jq -er '.layers[0].digest' "$manifest_path" | sed 's/^sha256://')"
stale_tmp="$store/blobs/sha256/${layer_hex}1072211852"
printf 'stale temp blob' >"$stale_tmp"

table="$("$bin" list --store "$store")"
printf '%s\n' "$table"
short_digest="$(printf '%s\n' "$table" | awk 'NR == 2 {print $2}')"
"$bin" rmi --store "$store" "$short_digest"
test ! -e "$stale_tmp"
test -z "$("$bin" list --store "$store")"

valid_orphan="$(printf 'ci-prune-orphan' | sha256sum | awk '{print $1}')"
malformed_orphan="$store/blobs/sha256/${valid_orphan}9999"
printf 'orphan blob' >"$store/blobs/sha256/$valid_orphan"
printf 'malformed orphan blob' >"$malformed_orphan"
"$bin" prune --store "$store"
test ! -e "$store/blobs/sha256/$valid_orphan"
test ! -e "$malformed_orphan"
"$bin" prune --store "$store" --cache

- name: Go unit + conformance tests (with network pull round-trip)
# ELFUSE_OCI_NETTEST enables the pull round-trip that re-opens the store
# with crane's independent layout reader and asserts digest agreement.
env:
ELFUSE_OCI_NETTEST: "1"
run: go test -race ./cmd/elfuse-container/

- name: Cross-tool interop (crane + skopeo + umoci)
# Pulls fixtures, then asserts the on-disk layout is spec-shaped and
# that available tools read it and agree with registry truth.
run: scripts/oci-interop.sh

# Darwin elfuse-container build + tests on a hosted macOS runner. The default `run`
# path (csrun.go, sparsebundle.go, cache_darwin.go) only compiles on darwin, so
# the Linux job above can only cross-vet it -- this job actually builds and runs
# it. Hosted runners provide hdiutil + case-sensitive APFS (so the real
# sparsebundle round-trip runs) even though they lack Hypervisor.framework; the
# HVF-backed guest boot is covered by the self-hosted runtime-macos job.
oci-macos:
name: OCI container CLI (macOS Apple Silicon)
runs-on: macos-15
timeout-minutes: 15
steps:
- name: Checkout
uses: actions/checkout@v7

- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: go.mod
cache: true

- name: Build elfuse-container
run: make build/elfuse-container

- name: Go unit tests (darwin native)
# Runs the whole suite on darwin, exercising the sparsebundle/clone seams
# in csrun/sparsebundle/cache_darwin that the Linux job cannot compile.
run: go test ./cmd/elfuse-container/

- name: Sparsebundle round-trip (hdiutil + case-sensitive APFS)
# ELFUSE_OCI_DARWIN_CS un-skips the real hdiutil create/attach/detach +
# case-sensitive APFS sweep; hosted runners have hdiutil and APFS.
env:
ELFUSE_OCI_DARWIN_CS: "1"
run: go test -run TestDarwinCSSweep ./cmd/elfuse-container/
52 changes: 51 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ SRCS := \
core/vdso.c \
core/shim-globals.c \
core/bootstrap.c \
core/launch.c \
core/rosetta.c \
core/sysroot.c \
runtime/thread.c \
Expand Down Expand Up @@ -100,7 +101,7 @@ endef
.PHONY: all elfuse
.PHONY: gen-syscall-dispatch check-syscall-dispatch

all: elfuse
all: elfuse elfuse-container

## Regenerate build/dispatch.h from src/syscall/dispatch.tbl
gen-syscall-dispatch:
Expand All @@ -125,6 +126,55 @@ elfuse: $(ELFUSE_BIN)
$(ELFUSE_BIN): $(OBJS) | $(BUILD_DIR)
$(call link-and-sign,$@,$(OBJS))

# OCI container CLI (Go). Pure Go, no HVF entitlement or codesigning required,
# so it also builds under Linux for spec-conformance / interop CI. The version
# is stamped from the same VERSION string the C binary uses.
CONTAINER_BIN := $(BUILD_DIR)/elfuse-container
CONTAINER_SRCS := $(shell find cmd/elfuse-container -type f -name '*.go' 2>/dev/null)

.PHONY: elfuse-container
elfuse-container: $(CONTAINER_BIN)

# OCI image-layout conformance + cross-tool interop. Pulls fixtures into a
# throwaway store and asserts the on-disk layout is spec-shaped and readable by
# crane/skopeo/umoci (whichever are installed locally; all are required in CI).
# Pure Go + jq; no HVF, runs on Linux. Requires network to pull fixtures.
.PHONY: oci-interop
oci-interop: $(CONTAINER_BIN)
$(Q)scripts/oci-interop.sh

# Go unit tests for the OCI container CLI (offline). Set ELFUSE_OCI_NETTEST=1
# to also exercise the network pull round-trip conformance test.
.PHONY: oci-test
oci-test:
$(Q)$(GO) test ./cmd/elfuse-container/

# gofmt + go vet gate for the OCI container CLI. `go vet` is run for both GOOS
# values so the darwin-only sparsebundle files are checked from Linux CI and the
# non-darwin stubs are checked from a macOS host. oci-lint bundles both so a
# local run matches the CI gate.
.PHONY: oci-vet oci-fmt-check oci-lint
oci-vet:

@cubic-dev-ai cubic-dev-ai Bot Jul 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The oci-vet darwin cross-vet run doesn't set GOARCH=arm64, so on an amd64 Linux host it vets the darwin code for darwin/amd64 instead of the actual target darwin/arm64. CI already pins GOARCH=arm64 for this vet. It's unlikely to cause a real build issue (the darwin-specific files are //go:build darwin without an arch constraint), but adding GOARCH=arm64 makes the local make oci-vet gate match CI more closely and avoids any future arch-specific vet diagnostics from slipping through.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At Makefile, line 157:

<comment>The `oci-vet` darwin cross-vet run doesn't set `GOARCH=arm64`, so on an amd64 Linux host it vets the darwin code for `darwin/amd64` instead of the actual target `darwin/arm64`. CI already pins `GOARCH=arm64` for this vet. It's unlikely to cause a real build issue (the darwin-specific files are `//go:build darwin` without an arch constraint), but adding `GOARCH=arm64` makes the local `make oci-vet` gate match CI more closely and avoids any future arch-specific vet diagnostics from slipping through.</comment>

<file context>
@@ -125,6 +126,55 @@ elfuse: $(ELFUSE_BIN)
+# non-darwin stubs are checked from a macOS host. oci-lint bundles both so a
+# local run matches the CI gate.
+.PHONY: oci-vet oci-fmt-check oci-lint
+oci-vet:
+	$(Q)$(GO) vet ./cmd/elfuse-container/
+	$(Q)GOOS=darwin $(GO) vet ./cmd/elfuse-container/
</file context>

$(Q)$(GO) vet ./cmd/elfuse-container/
$(Q)GOOS=darwin $(GO) vet ./cmd/elfuse-container/
$(Q)GOOS=linux $(GO) vet ./cmd/elfuse-container/

oci-fmt-check:
$(Q)out="$$(gofmt -l cmd/elfuse-container)"; \
if [ -n "$$out" ]; then \
echo "gofmt needs to run on:"; echo "$$out"; exit 1; \
fi

oci-lint: oci-fmt-check oci-vet

# rm -f first: `go build -o` follows an existing symlink at the output path,
# so a stale build/elfuse-container symlink would clobber build/elfuse.
$(CONTAINER_BIN): go.mod $(CONTAINER_SRCS) | $(BUILD_DIR)
@echo " GO $@"
$(Q)rm -f $@
$(Q)cd $(CURDIR) && $(GO) build -ldflags "-X main.version=$(VERSION)" \
-o $@ ./cmd/elfuse-container

# Native test binaries (macOS, Hypervisor.framework)

## Build the multi-vCPU HVF validation test (native macOS binary)
Expand Down
33 changes: 31 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ boot-time overhead those tools impose.
- Xcode Command Line Tools, `clang`, `codesign`, and GNU `make`
- GNU `objcopy` or `llvm-objcopy`
- Hypervisor entitlement: `com.apple.security.hypervisor`
- Go, for building the `elfuse-container` OCI CLI

To build only (`make elfuse`) without running tests, just the
Xcode Command Line Tools and `objcopy` (`brew install binutils`) suffice.
Expand Down Expand Up @@ -101,18 +102,43 @@ state.
The build signs `build/elfuse` before use. Override the signing identity with
`SIGN_IDENTITY="Developer ID ..."` when needed.

## OCI Images

OCI images are handled by `elfuse-container`, a Go companion binary that owns
the whole image lifecycle and invokes `elfuse` purely as the runtime. This
uses OCI image packaging only; it is not a Docker-compatible container runtime
and does not add namespaces, cgroups, port mapping, or a daemon.

```sh
make elfuse elfuse-container

build/elfuse-container pull alpine:3
build/elfuse-container run alpine:3 /bin/sh -c 'echo hello from elfuse'
```

Images are stored under `$ELFUSE_OCI_STORE`, or `~/.local/share/elfuse/oci`
by default. On macOS, `run` uses a case-sensitive APFS sparsebundle and a
per-run copy-on-write rootfs clone so normal APFS case folding does not
corrupt Linux filenames.

See [docs/usage.md](docs/usage.md#oci-images) for commands and flags, and
[docs/oci-design.md](docs/oci-design.md) for the implementation model.

## Documentation

- [docs/usage.md](docs/usage.md): command-line options, x86_64 via
Rosetta, dynamic linking via `--sysroot`, and attaching `gdb` /
`lldb` to the built-in stub.
Rosetta, dynamic linking via `--sysroot`, OCI images, and attaching
`gdb` / `lldb` to the built-in stub.
- [docs/testing.md](docs/testing.md): build prerequisites, the
`make check` flow, the QEMU and Rosetta cross-check matrices, and
fixture handling.
- [docs/internals.md](docs/internals.md): canonical technical
reference -- runtime lifecycle, HVF constraints, EL1 shim and HVC
protocol, page-table splitting, syscall translation tables, threads
/ futex, fork / clone IPC, signals, ptrace, and the GDB stub.
- [docs/oci-design.md](docs/oci-design.md): how elfuse-container, the image
store, layer unpacker, sparsebundle run path, and lifecycle commands
fit into elfuse.

## Build And Validation

Expand All @@ -123,6 +149,7 @@ make elfuse # build and codesign build/elfuse
make check # quick unit suite + BusyBox applet smoke
make test-gdbstub # debugger integration
make test-matrix # cross-check elfuse against QEMU on the same corpus
make oci-test # elfuse-container unit and conformance tests
make lint # clang-tidy
```

Expand All @@ -141,6 +168,8 @@ do.
- Linux kernel features that have no user-space-syscall analog:
namespaces, cgroups, kernel modules, eBPF, `io_uring`, KVM, perf
events.
- Docker-compatible container runtime features such as port mapping,
detached containers, `docker exec`, image build/push, and daemon APIs.
- Intel Macs. Apple Silicon only (M1 and later).
- Hosting a VM from inside a guest. The guest cannot use HVF or KVM.
- One guest process tree per `elfuse` host process. HVF allows one VM
Expand Down
Loading
Loading