sec: hash-pin every dependency install and add coverage-guided fuzzing

[cdeust]

Closes all 22 open code-scanning alerts: 21 Pinned-Dependencies + 1 Fuzzing. There were no Dependabot alerts and no known-CVE vulnerabilities open — every finding was OpenSSF Scorecard.

Refs #203, and goes past it: that issue covers the three Dockerfiles (11 alerts). The other 10 pinning alerts (ci.yml ×7, release.yml ×2, scripts/setup.sh) and the Fuzzing alert had no issue at all, and are fixed here rather than filed for later.

The core point

An exact version is not a pin. ruff==0.15.20 and torch==2.11.0 both counted as unpinned, and Scorecard is right: a version names a release, not the bytes the index serves for it today. Only a hash pins the artifact.

pip install --require-hashes is all-or-nothing — one hash means every requirement including transitive ones needs one — so it needs a resolved lock. uv.lock becomes the single source of truth.

Mechanism	What it does
scripts/generate_pip_constraints.py	Exports one hashed file per call site into requirements/. Refuses an export that is empty or carries an unhashed requirement. --check is a blocking Lint step, so a lock change that isn't re-exported fails there, not at install time.
[dependency-groups]	CI's own tools (ruff, pyright, build, hatchling, atheris) are locked instead of restated as bare version strings across two workflow files.
[[tool.uv.index]] + [tool.uv.sources]	Binds torch to the PyTorch CPU index on Linux.

The torch finding was worse than "unpinned"

The containers passed --index-url https://download.pytorch.org/whl/cpu at the call site. So uv.lock recorded PyPI's torch and its PyPI hashes, while the image installed a different artifact from a different index. No source of truth could produce a hash for what was actually installed — which is why --require-hashes wasn't reachable before.

The lock now carries torch 2.13.0+cpu with 22 hashes, and resolution drops 18 nvidia/cuda packages plus triton. torch is named in a container dependency-group only so the source can bind to it — PEP 735 groups aren't published, so nothing changes for anyone installing hypermnesia-mcp from PyPI.

Two findings couldn't be pinned — they had to stop being what they were

• curl https://deb.nodesource.com/setup_22.x | bash — an unreviewed remote script executed as root at build time, and there is no hash to check a pipe against. Replaced by what that script does: fetch the signing key, register the signed apt source, install the signed package. curl now feeds gpg --dearmor, which executes nothing.
• npm install -g @anthropic-ai/claude-code — unversioned, so the image tracked whatever the registry served that minute. Now npm ci against a committed lockfile, which is also the only form Scorecard accepts (an exact version on the command line is not enough) and which records a sha512 integrity hash per transitive package.

Accepted forms, from the checker's source

I read isUnpinnedPipInstall rather than inferring it. A bare pip install --no-deps . is unpinned — . is neither a flag nor a .whl, so it sets hasAdditionalArgs; isPinnedEditableSource is consulted only for -e. Hence: --require-hashes, or -e <local> --no-deps, or a .whl path. The root image therefore builds a wheel — an editable install there would leave a .pth pointing at /build, which the runtime stage never copies, and the venv would arrive broken.

Fuzzing — and what it found

Two harnesses over pure parsers that read untrusted text (§13.1 D2): the hand-rolled YAML frontmatter parser and the wiki source-path canonicaliser. ClusterFuzzLite runs a 120s batch on PRs (blocking) and a longer scheduled run (non-blocking — a fuzzer left running eventually finds something, and holding the merge queue hostage to an unrelated input makes the check ignored within a week).

Writing the path harness found a live bug. normalize_source_path stripped ./ in a loop, then / exactly once. Removing the slashes can expose a ./ the loop already walked past:

.//./x   ->  ./x     # still carries the prefix the function exists to remove
/./z     ->  ./z

The result was not idempotent, and extract_document_paths dedupes on it — so one document reachable by two spellings counted as two. Fixed by iterating to a fixed point. The four reproducers are committed as corpus inputs and fail on the pre-fix code (verified by stashing the fix and re-running).

fuzz/replay_corpus.py replays every corpus input with no atheris, so the properties run in the ordinary pytest suite on every platform. Atheris publishes manylinux x86_64 wheels for cpython 3.12–3.14 and nothing else — a property only one CI job can run is one that rots.

Also fixed (§14, seen en route)

• docker/Dockerfile could not build at all — it copied /usr/local/lib/python3.12/site-packages against a python:3.14 base, a path absent from both stages. Invisible because no CI job built this image. Now installs into a version-free venv (the rule the root Dockerfile already documents).
• scripts/setup.sh reported success over any failure — the install ended in 2>/dev/null, which is exactly where a resolution failure, hash mismatch and network error appear, and it printed "Python packages installed" regardless. Exit status is now checked.
• .gitattributes marks fuzz/corpus/** binary — EOL normalisation would have rewritten the CRLF seed on checkout and deleted the case it exists to cover.

---

Completion Ledger (§13.2)

Path enumeration

Path	Evidence
named_index_url — found	test_every_local_version_pin_names_a_serving_index
named_index_url — absent → ExportError	test_undeclared_index_is_refused
ConstraintSet.command() — extras / groups / only-groups	test_export_asserts_the_lock_matches_pyproject
header() — with and without index directive	test_every_local_version_pin_names_a_serving_index
render() — uv missing → ExportError	test_export_failure_exits_two_not_one
render() — uv non-zero exit → ExportError	test_export_failure_exits_two_not_one
render() — empty export refused	test_empty_export_is_refused
render() — unhashed requirement refused	test_unhashed_export_is_refused
render() — local pin absent from lock	test_local_pin_absent_from_the_lock_is_refused
write() — changed / unchanged	test_check_passes_on_the_committed_tree
stale() — missing file	test_check_fails_when_a_file_is_missing
stale() — content differs	test_check_fails_on_a_mutated_file
stale() — current → None	test_check_passes_on_the_committed_tree
main() — --check clean → 0 / drift → 1 / cannot run → 2	test_check_passes_on_the_committed_tree, test_check_fails_on_a_mutated_file, test_export_failure_exits_two_not_one
Drift message names the fix command	test_drift_message_names_the_regeneration_command
Every set has a file; no file orphaned; names unique	test_every_set_has_a_committed_file, test_no_committed_file_is_orphaned, test_filenames_are_unique
Every requirement version-pinned and hashed	test_every_requirement_is_version_pinned_and_hashed
No editable/directory requirements leak into a hashed file	test_no_editable_or_directory_requirements
normalize_source_path — canonical / ./ / / / backslash / blank / separators-only	TestNormalizeSourcePath (6 tests)
normalize_source_path — fixed-point regressions	TestNormalizeSourcePathReachesAFixedPoint (4 + idempotence)
normalize_source_path — .. left intact (negative)	test_parent_traversal_is_left_intact
extract_document_paths — two spellings collapse	test_two_spellings_of_one_document_collapse_to_one
Harness discovery non-empty (a fuzz setup finding nothing passes everything)	test_at_least_one_harness_exists
Every harness holds on its whole corpus; empty corpus refused	test_harness_holds_on_its_whole_corpus
YAML harness — type, dict-ness, lowercased keys, substring body	fuzz_yaml_frontmatter.consume over 10 corpus inputs
Path harness — no leading /, no leading ./, non-empty, idempotent	fuzz_source_path.consume over 10 corpus inputs

§13.1 checklist

A. Correctness & behavior

• A1 Full suite green locally: 6383 passed, 97 subtests passed in 200.81s. --require-hashes install proven end-to-end (ruff 0.15.20 installed from requirements/lint.txt, exit 0).
• A2 Edge cases: missing/mutated/empty/unhashed export, absent index declaration, uv absent, uv non-zero, blank & separator-only paths, CRLF, lone surrogates, unicode, 200-char dash runs — each mapped above.
• A3 Every failure arm asserted on observable effect, including the exit-code distinction between "could not run" (2) and "found drift" (1) — a missing uv must not read as a clean gate.
• A4 Trust boundary is the index; --require-hashes is the validation. Adversarial case tested directly: every hash corrupted → pip exits 1 with "THESE PACKAGES DO NOT MATCH THE HASHES". A single corrupted hash is correctly not enough (pip accepts if any listed hash matches) — the first attempt at this test was invalid for that reason and was redone.
• A5 Invariants: --check never writes; generation idempotent; normalize_source_path idempotent (the property the bug violated).
• A6 Idempotent by construction — write() no-ops when content matches.

B. Concurrency — N/A: single-shot synchronous scripts, no shared mutable state, no async. CI jobs are independent.

C. Resources & performance

• C1 Loops are over a fixed 13-set registry, a fixed corpus, and file lines. No unbounded growth.
• C2 No handles held; corpus read via read_bytes. Image size falls — 18 nvidia/cuda packages + triton removed from every container.
• C3 PR fuzz batch capped at 120s/harness/sanitizer so it doesn't become the slowest required check.

D. Security — this is the point of the change

• D1 No SQL/shell built from data. subprocess.run uses a fixed argv, no shell, no user input (# noqa: S603 with that rationale).
• D2 Untrusted input is the whole subject: the two fuzz targets read LLM-written and user-supplied documents.
• D3 No secrets touched; permissions: read-all on the fuzz workflow. Net effect: one unreviewed remote script no longer executes as root at build time, and every installed artifact is hash-verified. The one --extra-index-url added is documented as safe only because the file is hash-pinned — an artifact from either index that isn't the locked one fails the hash check before unpacking.

E. Interfaces & compatibility

• E1 Additive: new script, new requirements/, new groups. No public API change; PEP 735 groups are not published, so PyPI consumers see identical metadata.
• E2 Downstream consumers identified by name and verified: ci.yml (all 7 sites), release.yml (2), scripts/setup.sh, Dockerfile, docker/Dockerfile, .devcontainer/Dockerfile, .clusterfuzzlite/build.sh. extract_document_paths and tests_py/handlers/test_wiki_page_sources.py re-verified against the changed normalisation (637 passed).
• E3 N/A: no persisted-data format change.
• E4 .gitattributes handles the cross-platform EOL case explicitly. The Windows CI leg installs ci-sqlite-min.txt, which resolves without a tree-sitter toolchain by design.

F. Observability & operations

• F1 Every failure names the file and the remedy (run scripts/generate_pip_constraints.py); asserted by test_drift_message_names_the_regeneration_command. Success prints requirements OK (13 checked). setup.sh no longer swallows pip's stderr.
• F2 No silent degraded mode: an unhashed or empty export raises rather than writing a weaker file.

G. Tests

• G1 Ledger above maps every diff path.
• G2 Both bugs carry regressions that fail on pre-fix code — verified for the path bug by git stash-ing the fix and re-running the replayer (it failed on repro-backslash-mixed, then passed once restored).
• G3 Deterministic and isolated: temp dirs, no network in unit tests, no sleeps, no ordering dependence.
• G4 Negative assertions present: test_no_editable_or_directory_requirements, test_no_committed_file_is_orphaned, test_parent_traversal_is_left_intact, corrupted-hash rejection.
• G5 Full suite quoted in A1; ruff check . → All checks passed!; ruff format --check . → 1064 files already formatted; check_doc_claims.py and generate_pip_constraints.py --check both green.

H. Code quality & delivery

• H1 SOLID/layering/sizes: the constraint table is separated from the engine that runs uv, so the mapping is testable without uv on PATH and both files stay under the cap. Adding a set is one registry entry; adding a harness is one file (discovered, not listed). The two lint waivers are per-site with stated reasons (S603 fixed argv; PLC0415 atheris has no wheel for macOS/aarch64).
• H2 No dead code, no debug leftovers. --upgrade pip build removed from the root image: it was itself an unpinned install and build was never invoked there.
• H3 Matches neighbouring script conventions (check_doc_claims.py, generate_repo_badges.py — same fail-closed --check idiom).
• H4 CHANGELOG has Security/Fixed entries; suite size synced 6348 → 6383 across 11 sites in 5 files.
• H5 Two commits: the change, then CI wiring + count sync + docs.
• H6 CI green on the exact pushed tree — pending; will not be merged before it is.
• H7 Boy-scout (§14) — every defect seen in touched material is fixed here, none deferred: the unbuildable docker/Dockerfile, setup.sh's swallowed stderr and drifted package list, the non-idempotent path canonicaliser, the CRLF-corruptible corpus, and hatchling being fetched unpinned during build isolation. No unfixed, un-issued seen defect remains.

What I could not verify locally, and why

Stated plainly rather than implied green:

1. The three Docker images are not built on this machine. The volume has <1 GiB free (100% full) and the local Docker daemon's snapshotter returns input/output error on its own overlayfs. Building them needs many GB. Rather than defer sec: hash-pin the pip/npm/download-then-run steps in the three Dockerfiles (Scorecard Pinned-Dependencies) #203's "built at least once as evidence" criterion, I added CI build jobs for docker/Dockerfile and .devcontainer/Dockerfile — which is stronger than a one-off local build and closes the "no CI builds either, so regressions are invisible" gap that issue's own note raises. The build evidence will be those jobs on this PR.
2. The fuzzers themselves have not run here. Atheris has no macOS/aarch64 wheel. The harness properties are verified locally via the corpus replayer (20 inputs, in the pytest suite); the coverage-guided campaign runs in CI.
3. Scorecard has not been re-run. sec: hash-pin the pip/npm/download-then-run steps in the three Dockerfiles (Scorecard Pinned-Dependencies) #203 asks for a Pinned-Dependencies field diff. That needs the scanner against the pushed branch — the scorecard.yml workflow, or a manual run — and I cannot produce it locally without Docker. This is the one acceptance criterion of sec: hash-pin the pip/npm/download-then-run steps in the three Dockerfiles (Scorecard Pinned-Dependencies) #203 still outstanding, and it is a measurement, not an implementation gap.

Unrelated, pre-existing, and still open

Reported, not silently absorbed: every CI job emits Node.js 20 is deprecated for actions/checkout, actions/cache, actions/upload-artifact and docker/build-push-action. All are SHA-pinned, so fixing it means deliberate version bumps across ~6 action refs — a different change from this one, and outside its blast radius. Say the word and I'll do it in a follow-up PR or fold it in here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BJt9KN23rUNQzwoJv4jRWw