ci: add self-hosted Linux runner (Docker) and opt-in smoke test - #263
Open
Yaraslaut wants to merge 24 commits into
Open
ci: add self-hosted Linux runner (Docker) and opt-in smoke test#263Yaraslaut wants to merge 24 commits into
Yaraslaut wants to merge 24 commits into
Conversation
Yaraslaut
force-pushed
the
ci/self-hosted-linux-runner
branch
from
August 24, 2026 12:14
dc48cd3 to
1be5c6b
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…rkflow Registers a Dockerized Linux runner (Ubuntu 24.04 base, labels self-hosted/Linux/X64/morph-docker) against this repo and adds self-hosted-smoke.yml as a proving-ground job on it, separate from ci.yml's existing GitHub-hosted jobs. The runner image intentionally does not prebake gcc/clang/sccache -- ci.yml's jobs already bootstrap their own toolchain per run, and this mirrors that rather than duplicating it in a second place. The smoke-test workflow excludes pull_request from forks (checked via pull_request.head.repo.full_name == github.repository): a self-hosted runner executes arbitrary job code on a real machine, so only push, workflow_dispatch, and same-repo PRs are treated as trusted triggers.
Covers building the image, minting a registration token, running the container long-lived, deregistering cleanly, the env vars entrypoint.sh reads, and the pull_request trust-boundary requirement any new workflow targeting this runner needs to mirror.
Adds a probe-self-hosted job that checks the runners API for an online, non-busy morph-docker runner and outputs which runs-on: label set to use; linux-compilers consumes that output instead of a hardcoded ubuntu-24.04. Falls back to ubuntu-24.04 whenever: no RUNNER_STATUS_TOKEN secret is available (e.g. a forked-repo PR, which never gets repo secrets), the runners API call fails, or no morph-docker runner is currently online and idle. GITHUB_TOKEN can't call the runners API itself -- it's a repo-admin operation regardless of the permissions: block -- so this needs RUNNER_STATUS_TOKEN, a fine-grained PAT scoped to this repo with Administration: Read-only, added as a repo secret (not done by this commit). Other Linux jobs (sanitizers, coverage, valgrind, Qt, ladder, clang-tidy) stay on ubuntu-24.04 for now -- linux-compilers is the first real workload on the self-hosted runner, chosen for being the cheapest/ fastest legs to validate the fallback plumbing against.
Records that two runners are now registered (this machine + Hetzner morph-hetzner-hel1), why running more than one matters for the 4-leg linux-compilers matrix, and documents the RUNNER_STATUS_TOKEN secret ci.yml's probe-self-hosted job needs -- what scope to grant it, where to add it, and what happens (safe fallback) while it's absent.
ci.yml's Linux jobs untar sccache straight into /usr/local/bin with no sudo, matching how GitHub-hosted ubuntu-24.04 runners are set up (that path is writable by the runner user there). Stock ubuntu:24.04 leaves it root:root 755, so the same step failed here with 'Permission denied' -- confirmed via the first live linux-compilers run against this runner, all 4 matrix legs failing identically at the sccache install step.
The container image only had curl/git/jq/sudo/wget/tar-class base tools; GitHub-hosted ubuntu-24.04 runner images additionally bundle ~500 packages ci.yml's jobs rely on without an explicit apt-get step. Confirmed missing: cmake (linux-compilers failed 'command not found' on 3 of 4 legs the moment /usr/local/bin's permission bug -- cde7548 -- stopped masking it as a second failure first). Added the rest (python3, perl, binutils, build-essential) from an audit of every Linux job's run: steps against what they invoke but never install: clang-tidy and the coverage leg's aggregate script need python3; deprecation-lint and test-type-name-lint's scripts need perl; ladder-sanitizers' instrumentation check needs binutils' nm; build-essential gives a base gcc/g++/make (ci.yml still installs its own pinned gcc-15/clang on top of this, unaffected).
…o self-hosted Both jobs now depend on probe-self-hosted the same way linux-compilers does. clang-coverage is split out of linux-sanitizers into its own linux-coverage job pinned to ubuntu-24.04, rather than moved along with the other 3 legs -- it additionally builds the full ladder stack (ODBC/SQLite/yaml-cpp/zip) and Qt and uploads to Codecov, meaningfully heavier than plain asan/tsan/ubsan, and keeping it on GitHub-hosted avoids adding that weight to the self-hosted rotation before the lighter jobs have proven out. linux-all-features' clang/gcc matrix needed no such split -- both legs run the same steps unconditionally, unlike linux-sanitizers' old if: matrix.preset == 'clang-coverage' branches threaded through Configure/Build/Test.
…first clang (installed per-job from apt.llvm.org) uses libstdc++'s headers by default. Stock ubuntu:24.04 ships libstdc++ 13.x, which predates C++23 <print> (needs GCC 14+); morph's headers include it unconditionally, so a clang-only leg with no gcc install step of its own (clang-release, clang-asan, ...) failed with "fatal error: 'print' file not found" -- confirmed live on linux-compilers' clang-release leg. It had been masked on earlier runs by container state leftover from a same-run gcc leg installing libstdc++-15-dev as gcc-15's dependency first; that's execution-order luck, not a guarantee, so it's baked into the image instead. Same ubuntu-toolchain-r/test PPA ci.yml's own "Install GCC 15" step already uses, so this tracks that step's version.
Ubuntu 24.04's apt package is CMake 3.28.3. GitHub-hosted ubuntu-24.04 ships a newer CMake from Kitware's own release archive rather than the distro package -- confirmed the gap is not cosmetic: linux-all-features' configure step failed on this image with "Impossible to link target ... because the link item ... has already occurred with the feature 'WHOLE_ARCHIVE', which is not allowed", a false positive in $<LINK_LIBRARY:WHOLE_ARCHIVE,...> duplicate-feature detection that CMake was still patching in the 3.28-3.30 range. The identical configure passes cleanly on GitHub-hosted ubuntu-24.04 and on this same repo checkout once pinned to Kitware's 3.31.12 release, confirming this was a CMake-version gap, not a real target-graph conflict in cmake/morph_add_rung.cmake.
…est workflow Only the maintainer's Windows machine registration remains. Notes that adding a second host back is what restores per-leg parallelism for a matrix job, without needing any other change. Also removes self-hosted-smoke.yml: linux-compilers, linux-sanitizers, and linux-all-features now exercise the self-hosted runner directly on every real CI run, making the standalone smoke test redundant.
Morph's vendored copy was 1123 lines to fastcached's current 302 -- significant drift. Picks up FASTCACHE_AUTO_INSTALL (fetches a prebuilt fastcache-cc from the fastcached project's GitHub Releases when no compiler-cache launcher is already on PATH) and FASTCACHE_AUTO_START, plus the FASTCACHE_SOURCE_DIR/FASTCACHE_BINARY_DIR env var rename from the older FASTCACHE_SRCROOT/FASTCACHE_BUILDTREE. CMakeLists.txt's own launcher detection (MORPH_COMPILER_CACHE_IS_FASTCACHE) matches on the chosen launcher's path substring, not on either env var name, so it needed no change.
linux-compilers, linux-sanitizers, and linux-all-features drop their explicit -DCMAKE_C_COMPILER_LAUNCHER=sccache / -DCMAKE_CXX_COMPILER_LAUNCHER flags and instead set FASTCACHE_ADDR=host.docker.internal:6674 and FASTCACHE_AUTO_INSTALL=ON as job-level env, conditional on probe-self-hosted having actually chosen the self-hosted path. CompileCache.cmake bails out the instant either launcher variable is already set, so it never even probed for fastcache-cc before this -- these jobs were only ever going to use sccache regardless of what answered at FASTCACHE_ADDR. The condition matters, not just the address: host.docker.internal does not resolve on a GitHub-hosted VM (it is not a Docker container), and an address that is merely unreachable rather than refusing the connection stalls fastcache-cc's TCP connect for up to ~2m30s per matrix leg (CompileCache.cmake's own measured worst case) instead of failing fast. Left empty/OFF on the GitHub-hosted fallback, the jobs keep using their already-installed sccache exactly as before -- CompileCache.cmake's own launcher preference (fastcache-cc, then sccache, then ccache) does the rest with no other change needed. The fastcached daemon itself is not part of this repository: it is the maintainer's existing Windows service (D:\caching), rebound from 127.0.0.1 to 0.0.0.0 so host.docker.internal can reach it from inside a container -- see the self-hosted-runner README for the full picture.
Covers: sizing --cpus=N per container against host logical processors, why multiple runners matter for a matrix job's parallelism, the pinned CMake/libstdc++ fixes, the corrected trust-boundary explanation (fork PRs are kept off self-hosted via GitHub's own secret-withholding guarantee on probe-self-hosted's RUNNER_STATUS_TOKEN read, not a job-level skip -- which would have dropped Linux coverage for fork PRs entirely rather than just routing them to GitHub-hosted), and the new Compiler cache: fastcache-cc section explaining host.docker.internal, the daemon's bind requirement, and the sccache fallback.
Confirmed live: FASTCACHE_ADDR landed in CMakeCache.txt correctly (that one IS read from the environment, per CompileCache.cmake's own _fc_addr_env), but FASTCACHE_AUTO_INSTALL stayed OFF despite the job env setting it to 'ON' -- because CompileCache.cmake declares it with plain option(), which CMake populates only from a -D flag or an existing cache entry, never from the process environment. Replaces the FASTCACHE_AUTO_INSTALL env var with MORPH_FASTCACHE_AUTO_INSTALL_FLAG, a precomputed -DFASTCACHE_AUTO_INSTALL=ON (or empty) string spliced onto each self-hosted job's cmake invocation directly, in linux-compilers, linux-sanitizers, and linux-all-features.
CompileCache.cmake's FASTCACHE_AUTO_INSTALL only fetches fastcache-cc
when NONE of fastcache-cc/sccache/ccache are already on PATH ("only
when there is nothing else to use") -- confirmed live: even with the
-DFASTCACHE_AUTO_INSTALL=ON fix, every self-hosted Configure step still
logged '[cache] Enabling sccache', because the unconditional 'Install
sccache' step earlier in the same job had already put sccache on PATH,
permanently satisfying the guard's AND NOT SCCACHE as false before
auto-install ever got a chance to run.
Guards Restore sccache / Install sccache / sccache stats / Save sccache
in linux-compilers, linux-sanitizers, and linux-all-features behind
!contains(needs.probe-self-hosted.outputs.runs_on, 'self-hosted') --
sccache now only appears on the GitHub-hosted fallback, where it's
needed, leaving the self-hosted path free for fastcache-cc's
auto-install to actually run.
Third fastcache-cc auto-install blocker found this session: even with FASTCACHE_AUTO_INSTALL=ON correctly reaching CMake and ci.yml's Install sccache step skipped on the self-hosted path, CMakeCache.txt still showed SCCACHE:FILEPATH=/usr/local/bin/sccache found -- left over from an earlier job's run on this same persistent container, which unlike a GitHub-hosted VM is not thrown away between jobs. find_program(SCCACHE) doesn't care which job put a binary there, so a leftover sccache from job N permanently blocks fastcache-cc auto-install on job N+1's Configure regardless of whether N+1 itself ever installs sccache. entrypoint.sh now removes it unconditionally on every registration, so each container starts genuinely clean of it rather than carrying forward whatever a previous job happened to leave behind.
…y and vcpkg caching
Three independent changes:
1. ladder-tests ("Application ladder") and ladder-sanitizers
("Application ladder / ASan+UBSan") now route through probe-self-hosted
the same way linux-compilers/linux-sanitizers/linux-all-features do,
with the matching FASTCACHE_ADDR/MORPH_FASTCACHE_AUTO_INSTALL_FLAG env
and sccache steps skipped on the self-hosted path.
2. clang-tidy-diff no longer builds the project. clang-tidy-diff.py reads
compile_commands.json (a configure-time artifact,
CMAKE_EXPORT_COMPILE_COMMANDS=ON in the base preset) and runs clang-tidy
itself per translation unit -- it never needed the project actually
compiled or linked. pinned_facts.cmake's generated header is likewise a
configure_file() (configure-time), not a build-time add_custom_command,
so it's already on disk too. Dropping the Build step also removes this
job's only reason to touch sccache at all, so those steps are gone
with it rather than merely guarded.
3. Windows vcpkg binary caching is fixed, not just reconfigured: vcpkg's
'x-gha' backend has been REMOVED upstream, confirmed via a warning
already present in this repo's own CI logs ("The 'x-gha' binary
caching backend has been removed") -- every Windows job has been
silently rebuilding every vcpkg dependency from source, with the
'Export vcpkg GHA binary cache vars' step doing nothing useful.
Replaced with the officially documented successor: a NuGet-based
provider backed by this repo's own GitHub Packages feed
(VCPKG_BINARY_SOURCES: clear;nuget,github,readwrite), authenticated via
GITHUB_TOKEN with a new packages: write permission scoped to just the
windows job.
Yaraslaut
force-pushed
the
ci/self-hosted-linux-runner
branch
2 times, most recently
from
August 24, 2026 17:22
5d06b98 to
09d96bc
Compare
#265 (b863bc6) added VERIFY_INTERFACE_HEADER_SETS's public-header audit and, on master's own next CI run, immediately caught a header it introduced no coverage for: include/morph/forms/instance_constraints.hpp was never added to any target's FILE_SET, so master's CI has been failing at Configure on every job that builds the morph target since that merge (confirmed: master's own CI run right after #265 failed identically). Verified the fix directly by copying the corrected CMakeLists.txt into a live self-hosted runner and reconfiguring -- 'Configuring done', no FILE_SET error.
Confirmed live: restarting Docker Desktop re-runs entrypoint.sh against
the same container filesystem (these are long-lived containers, not
recreated per start), with .runner/.credentials from the previous
registration still on disk and a RUNNER_TOKEN baked in at the original
docker run that's now past its ~1h TTL. config.sh --replace refused to
reconfigure over that leftover state ('Cannot configure the runner
because it is already configured'), which failed under set -e, fired
the cleanup trap, and failed *that* too (the old registration was
already gone server-side, so removing it 404's) -- restart-looping
forever under --restart unless-stopped. All 4 runner containers hit
this simultaneously on this session's Docker Desktop restart, silently
killing two in-flight Application ladder jobs.
Removes .runner/.credentials/.credentials_rsaparams unconditionally
before calling config.sh, rather than relying on --replace to get past
state it apparently doesn't actually replace.
Lightweight's CMakeLists.txt does find_package(SQLite3) as a system CONFIG package the moment MORPH_BUILD_LADDER=ON pulls it in. GitHub-hosted ubuntu-24.04 ships libsqlite3-dev preinstalled; ladder-tests never installed it itself, unlike ladder-sanitizers and linux-compilers/linux-sanitizers, which already do. Confirmed live: 'Could NOT find SQLite3 (missing: SQLite3_INCLUDE_DIR SQLite3_LIBRARY)' on the self-hosted runner, which has no such preinstalled package to paper over the gap.
Sets up a fresh Linux cloud VM (a Hetzner box, or any bare Ubuntu/Debian) with both a fastcached compile-cache daemon and N self-hosted runner containers, all sized dynamically from the machine's actual nproc/ /proc/meminfo at run time -- no hardcoded worker count or per-worker CPU/RAM, so the same script works on any box size. Sizing rule: fastcached gets a fixed 2 GiB RAM / 10 GiB disk / 1 CPU (does not scale with machine size, per its own tuning); an OS/Docker reserve holds back max(1 GiB, 10% of total RAM); whatever CPUs remain are split into 2-CPU workers, each getting an even share of whatever RAM is left. Refuses to proceed rather than start something undersized -- verified by hand that a 2-CPU/4GiB box is genuinely too small under this scheme (needs at least 3 CPUs: 1 fastcached + 2 for one worker). Verified the resulting plan against a real 8-CPU/15GiB Hetzner box: 3 workers x 2 CPU/4 GiB, fastcached at 1 CPU/2 GiB/10 GiB-disk, 1 GiB OS headroom -- uses the box's resources exactly with no waste.
Confirmed live on the Hetzner box: the daemon started and served requests fine, but Docker reported it "(unhealthy)" forever -- fastcached's own Dockerfile HEALTHCHECK is `fastcached --healthcheck`, which needs --metrics (and a reachable --metrics-bind) to actually answer. The bootstrap script's docker run overrides the image's whole default CMD (which does pass --metrics) with its own arg list, silently dropping it.
7 runners online across two hosts now (4 Windows + 3 Hetzner), each host running its own fastcached rather than sharing one -- clarifies that host.docker.internal:6674 is the same literal address on every host (ci.yml hardcodes it), so every host needs its own daemon reachable there, and documents how that resolves differently on Docker Desktop (automatic) vs plain Linux Docker Engine (needs --add-host=host.docker.internal:host-gateway, which bootstrap-cloud-node.sh already adds -- confirmed REACHABLE from a Hetzner worker to its local fastcached).
Rerun from scratch on the fixed self-hosted runner fleet (all 4 Windows containers now have a real --memory=6g cap, closing the OOM-kill that failed 4 jobs on the previous run) rather than reusing a run with already-failed jobs baked into its history.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.github/self-hosted-runner/— Dockerfile + entrypoint for a repo-level self-hosted Linux runner (Ubuntu 24.04 base, labelsself-hosted, Linux, X64, morph-docker). Currently running locally, registered against this repo and online..github/workflows/self-hosted-smoke.yml— a small opt-in job (gcc-debug configure/build/test) proving the runner works end-to-end. Kept separate fromci.ymlrather than modifying the existing GitHub-hosted jobs.Trust boundary
A self-hosted runner executes arbitrary job code on a real machine, so the smoke-test workflow is deliberately narrower than
ci.yml'spull_request:trigger:pushtomain/masterworkflow_dispatchpull_request, but gated withif: github.event.pull_request.head.repo.full_name == github.repository— same-repo branches only, forked PRs never run on this runner.Notes
ci.yml's existing pattern of bootstrapping the toolchain via apt/apt.llvm.org each run, so there's one place (not two) to keep the toolchain versions in sync.docker runtime (short-lived, ~1hr TTL); nothing secret is committed.ci.yml's existing jobs are untouched — this is purely additive/opt-in.🤖 Generated with Claude Code