CI/CD performance benchmarking

[jserv]

Motivation

Performance is one of the primary goals of elfuse. While functional correctness is verified through automated testing, continuous performance evaluation is equally important to detect regressions and quantify improvements over time.

This proposal introduces a lightweight benchmark suite that can be executed automatically in CI/CD and provides a consistent basis for comparing elfuse against native Linux and OrbStack.

Objectives:

• Detect performance regressions in pull requests.
• Provide reproducible measurements across releases.
• Keep CI execution time short.
• Focus on workloads representative of Linux application execution.

Benchmark Strategy

Two complementary tiers:

1. Tier 1: Microbenchmarks using lmbench to isolate low-level operating system overheads.
2. Tier 2: Application Benchmarks using widely adopted command-line tools to measure real-world developer workflows.

Tier 1: lmbench Microbenchmarks

Rather than adopting large suites such as SPEC CPU or Phoronix Test Suite, this proposal selects three focused lmbench benchmarks representing the most critical execution paths of elfuse.

Benchmark	Purpose	Relevance
lat_syscall	Measure system call latency	Evaluates the overhead of the Linux ABI and syscall forwarding path.
lat_proc exec	Measure process startup latency	Evaluates ELF loading, dynamic linking, memory mapping, and process initialization.
lat_fs	Measure filesystem metadata latency	Evaluates pathname translation and filesystem-intensive workloads.

lat_syscall

Measures the latency of fundamental system calls. Recommended operations: null, read, write, stat, open. These represent the basic cost of entering the Linux execution environment and are exercised by nearly every Linux application.

lat_proc exec

Measures end-to-end process startup latency via exec(). For elfuse, it covers nearly the entire execution path: ELF parsing, program loading, dynamic linker initialization, memory mapping, initial process setup, and process termination. Primary indicator of Linux application startup performance.

lat_fs

Measures filesystem metadata operations (file creation, deletion, directory traversal). Metadata-intensive workloads are common in Git, Make, CMake, package managers, and shell utilities, so lat_fs approximates many software development workloads.

Tier 2: Application Benchmarks

Microbenchmarks explain where time is spent; developers ultimately care about application performance. The suite includes representative command-line tools exercising common Linux development workflows.

Tool	Primary workload	Primary subsystem
python3 -c pass	Process startup	ELF loader, dynamic linker, interpreter initialization
git status	Filesystem metadata	open(), stat(), readdir()
rg (ripgrep)	Large-scale source search	mmap(), read(), page cache
zstd	Compression	CPU computation and filesystem I/O
make (small project)	Software build	Process creation, filesystem access, compiler invocation

These were selected because they are representative of daily Linux developer workflows, small enough to execute in CI, stable and reproducible, and portable across native Linux, OrbStack, and elfuse.

CI/CD Integration

The complete suite should run for every pull request:

Build elfuse
        │
Run lmbench benchmarks
        │
Run application benchmarks
        │
Collect benchmark results
        │
Generate JSON report
        │
Compare with baseline
        │
Upload artifacts

Expected execution time remains only a few minutes.

Result Format

Benchmark results exported as structured JSON to simplify regression analysis:

{
  "lmbench": {
    "lat_syscall": { "null": 0.23, "open": 1.98, "stat": 2.45 },
    "lat_proc": { "exec": 705 },
    "lat_fs": { "create": 47, "delete": 34 }
  },
  "applications": {
    "python_startup_ms": 24.1,
    "git_status_ms": 132.7,
    "ripgrep_ms": 218.4,
    "zstd_ms": 645.8,
    "make_ms": 1038.2
  }
}

Comparison Targets

Compare three execution environments: Native Linux (baseline), OrbStack, and elfuse. Each benchmark reports both absolute time and the ratio relative to native Linux.

Benchmark	Native Linux	OrbStack	elfuse
lat_proc exec	520 µs	730 µs	680 µs
lat_syscall(open)	1.4 µs	1.8 µs	2.0 µs
lat_fs(create)	41 µs	46 µs	44 µs
python3 -c pass	21 ms	26 ms	24 ms
git status	118 ms	126 ms	121 ms
rg	205 ms	214 ms	208 ms
zstd	630 ms	651 ms	642 ms
make	1.02 s	1.08 s	1.04 s

(Numbers above are illustrative placeholders, not measured results.)

Implementation Plan

Concrete, self-contained tasks. Each is independently mergeable and grounded in the current tree (tests/bench-*, tests/test-matrix.sh, tests/fetch-fixtures.sh, .github/workflows/main.yml).

Constraint: where benchmarks can run

elfuse requires the Hypervisor.framework entitlement, so any benchmark that executes a guest through elfuse only runs on the existing [self-hosted, macOS, arm64] runner (the runtime-macos job in main.yml), not on GitHub-hosted ubuntu-24.04 / macos-15. Consequence: native-Linux and OrbStack baselines cannot be produced inside GitHub-hosted CI. They are captured manually on a reference host and checked in as static baseline JSON; CI measures only the elfuse-aarch64 (and, where available, qemu-aarch64) columns and compares against the checked-in baseline.

Task 1: Vendor/fetch benchmark guest binaries

• Add lmbench aarch64-linux-musl binaries (lat_syscall, lat_proc, lat_fs) to the fixtures flow in tests/fetch-fixtures.sh (same Alpine-package pattern already used for busybox/coreutils). Pin the package version and record the SHA.
• Add Tier-2 guest binaries as static aarch64-musl fixtures where possible: git, ripgrep (rg), zstd, make, and a Python interpreter. Where a static build is impractical, document the dynamic + --sysroot invocation instead.
• Land a fixed source tree for the git status / rg / make targets (a small committed corpus under tests/bench-corpus/) so measurements are reproducible run to run.
• Verify each fixture runs under elfuse manually: ./elfuse <fixture> --help (or equivalent) exits 0.

Task 2: Benchmark runner + JSON emitter

• Add tests/bench-suite.sh that runs the Tier-1 and Tier-2 workloads and writes one JSON document matching the schema in "Result Format". Reuse tests/lib/test-runner.sh (run/run_check) so every invocation inherits the existing timeout $TEST_TIMEOUT wrapping.
• Drive each workload N iterations (default N=10), discard warmup, and report median (not mean) to reduce noise on a shared runner. Emit iteration count and raw samples alongside the median.
• Parameterize the execution environment via an arg/env (BENCH_ENV=elfuse-aarch64|native|orbstack) so the same script produces every column. Reuse detect_x86_64_host_class / the mode plumbing from tests/test-matrix.sh rather than duplicating host detection.
• Add a make bench target (and a CI-scoped make bench-ci) that builds the harness and runs tests/bench-suite.sh, writing build/bench-results.json.

Task 3: Baseline + regression comparison

• Add scripts/bench-compare.py: takes build/bench-results.json and a checked-in tests/bench-baseline.json, prints a per-metric ratio table, and exits non-zero when any metric regresses beyond a threshold (default: +20%, configurable via env). Reuse the CLI/reporting style of scripts/check-syscall-coverage.py.
• Commit an initial tests/bench-baseline.json captured on the self-hosted runner (elfuse column) plus manually captured native-Linux and OrbStack columns on a documented reference host. Record host class / CPU / date in the file so stale baselines are visible.
• Document the baseline-refresh procedure (when and how to regenerate) in tests/README or the suite script header.

Task 4: CI/CD integration

• Add a benchmark leg to the runtime-macos matrix in .github/workflows/main.yml (self-hosted arm64), gated the same way as the existing legs (github.repository == 'sysprog21/elfuse'), running make bench-ci then scripts/bench-compare.py.
• Upload build/bench-results.json via actions/upload-artifact@v7 (mirror the existing infer-* artifact step) with a short retention.
• Make regression a soft signal first: run bench-compare.py in report-only mode (|| true or a continue-on-error step) for an initial settling period, then flip to hard-fail once the runner's variance is characterized. State the flip criterion in the workflow comment.
• Keep the leg off the sanitizer legs and out of GitHub-hosted jobs; confirm total added wall-clock stays within the existing runtime-macos timeout-minutes budget.

Acceptance criteria

• make bench produces valid JSON matching the documented schema on a dev machine.
• scripts/bench-compare.py correctly flags an injected 2x regression and passes on baseline.
• The CI leg runs green on a PR, uploads the artifact, and its added runtime is within a few minutes.
• Native / OrbStack baseline capture is documented and reproducible.