Binary cache Bloom filter

[edolstra]

Motivation

This adds the ability for binary caches to expose a Bloom filter that lets clients quickly decide whether a store path definitely does not exist in the binary cache.

nix serve now generates a Bloom filter, which it advertises to clients using a new field in nix-cache-info:

$ curl http://localhost:8080/nix-cache-info
...
BloomFilter: /bloom-filter

You can also generate the Bloom filter manually, e.g.

$ nix path-info --all --store file:///tmp/binary-cache | nix store generate-bloom-filter --from-file /dev/stdin --false-positive-rate 0.0001 > bloom-filter
Wrote bloom filter (4815 bytes) for 1996 store paths (0.000100 false positive rate).

Clients cache the Bloom filter according to narinfo-cache-negative-ttl. They revalidate the cached filter using its ETag, so revalidation should be quick if the Bloom filter doesn't change often. Thus, Bloom filters are primarily useful for binary caches that are not very big and don't change very often (like https://install.determinate.systems).

Context

Summary by CodeRabbit

• New Features

  • Added a CLI command to generate Bloom filters, a server /bloom-filter endpoint, and client-side use of published Bloom filters with conditional fetches to short‑circuit absent store‑path queries.

• Documentation

  • New Binary Cache Bloom Filter Format doc and updated nix-cache-info examples.

• Tests

  • Functional tests covering generation, serving, reuse, revalidation, and client-side rule‑out behavior of Bloom filters.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds Bloom filter publication and consumption for binary caches: format, server endpoint and ETag handling, CLI generation, disk persistence, client conditional fetching and negative-lookup short-circuits, documentation, and functional tests.

Changes

Bloom Filter Support

Layer / File(s)	Summary
Bloom filter contracts and interface extensions src/libstore/include/nix/store/bloom-filter.hh, src/libstore/include/nix/store/binary-cache-store.hh, src/libstore/include/nix/store/nar-info-disk-cache.hh, src/libstore/include/nix/store/http-binary-cache-store.hh, src/libstore/include/nix/store/meson.build	Introduce ConditionalGetResult, BloomState and bloomFilterUrl in BinaryCacheStore, CacheInfo/new NarInfoDiskCache virtual methods, and buildBloomFilter() / forEachBloomBitPosition() declarations; add header to Meson.
Bloom filter construction library & CLI src/libstore/bloom-filter.cc, src/nix/generate-bloom-filter.cc, src/libstore/meson.build, src/nix/meson.build	Implement buildBloomFilter() (header + bit array serialization, Kirsch–Mitzenmacher double-hashing) and add store generate-bloom-filter CLI with --from-file and --false-positive-rate; add build wiring.
Disk cache bloom filter persistence src/libstore/nar-info-disk-cache.cc, src/libstore-tests/nar-info-disk-cache.cc	Extend SQLite schema with BloomFilters table and bloomFilterUrl in BinaryCaches; refactor Cache to use CacheInfo; implement probeBloomFilter, upsertBloomFilter, touchBloomFilter, and getBloomFilterETag with sqlite blob reads and v2 DB filename. Update tests to the struct-based createCache.
Binary cache store bloom filter integration src/libstore/binary-cache-store.cc	isDefinitelyMissing() computes bloom bit positions, consults disk cache, conditionally fetches filter via getFileConditional, validates payload, upserts bits to disk, transitions in-process state, and disables bloom probing on failures. Short-circuit isValidPathUncached() and queryPathInfoUncached() for definite-missing results.
HTTP cache conditional GET support src/libstore/include/nix/store/http-binary-cache-store.hh, src/libstore/http-binary-cache-store.cc	Add getFileConditional(path, expectedETag) to perform ETag-aware downloads and map responses to ConditionalGetResult; propagate bloomFilterUrl into cache creation.
Nix serve bloom filter endpoint src/nix/serve.cc	Add --false-positive-rate option, advertise BloomFilter: /bloom-filter in /nix-cache-info, serve /bloom-filter generating a filter from all valid store paths, compute an SHA-512 ETag, and honor If-None-Match → 304.
Bloom filter format and protocol documentation doc/manual/source/SUMMARY.md.in, doc/manual/source/protocols/binary-cache-bloom-filter.md, doc/manual/source/protocols/nix-cache-info.md	Document binary format (magic/version/k/m header + bit array), client membership-test (Nix32 decode + double-hashing), server parameter formulas, and nix-cache-info BloomFilter field.
Functional tests tests/functional/binary-cache.sh	Add test covering server generation/publication, client disk-cache reuse, conditional GET 304 handling on refresh, and update startNixServe to log nix serve output.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• DeterminateSystems/nix-src#428: Modifies src/nix/serve.cc server endpoints; related to adding/adjusting nix serve behavior used by this PR.

Suggested reviewers

• cole-h

Poem

🐰 I nibble bits and set the bloom,
Tiny flags that banish gloom,
Twice I hash, then hop and write,
Paths rejected save the night,
Cache and server in a quiet boom!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title "Binary cache Bloom filter" accurately describes the main change: adding Bloom filter support for binary caches. It is concise, specific, and captures the primary feature addition.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch eelcodolstra/nix-423

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

🚀 Deployed on https://6a29743d435e381d1fcfd34c--determinate-nix.netlify.app

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (1)

src/libstore/nar-info-disk-cache.cc (1)

467-474: 💤 Low value

Potential integer overflow in blob offset cast for extremely large filters.

The cast (int)(pos / 8) on line 469 could overflow if mBits exceeds ~17 billion bits (2GB blob). While practically unlikely given bloom filter sizes are typically much smaller, SQLite's sqlite3_blob_read accepts int offset, which limits addressable blob size to ~2GB.

This is acceptable for the intended use case of "relatively small and infrequently changed" caches per the PR objectives.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/libstore/nar-info-disk-cache.cc` around lines 467 - 474, The code casts
(pos / 8) to int when calling sqlite3_blob_read which can overflow for extremely
large bloom filters; before calling sqlite3_blob_read (in the loop over
bitPositions inside nar-info-disk-cache.cc) check that (pos / 8) <= INT_MAX (or
otherwise fits in an int) and if not, raise/throw a clear SQLiteError (or return
false) so you never pass a truncated negative offset to sqlite3_blob_read;
reference the loop over bitPositions and the sqlite3_blob_read call when adding
the bounds check.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@doc/manual/source/protocols/binary-cache-bloom-filter.md`:
- Around line 38-40: Add explicit language identifiers (e.g., ```text) to the
unlabeled fenced code blocks that contain the formulas so MD040 doesn't trigger;
specifically update the fence that wraps "pos = ((h1 + i * h2) mod 2^64) mod m"
and the fence that wraps the block starting with "m = ceil(-n * ln(p) / (ln
2)^2),  rounded up to a multiple of 8" to use a language label such as text (or
math) instead of an unlabeled ``` fence.

In `@doc/manual/source/protocols/nix-cache-info.md`:
- Around line 46-49: The fenced code blocks containing the example protocol
snippets (the blocks that include lines like "BloomFilter: /bloom-filter",
"BloomFilter: https://filters.example.com/cache-abc.bloom", and the block
containing "StoreDir: /nix/store", "WantMassQuery: 1", etc.) need explicit
language tags to satisfy MD040; update both code fences to use a language token
such as "text" (i.e., change ``` to ```text) so the blocks are marked as plain
text in the markdown.

In `@src/libstore/bloom-filter.cc`:
- Around line 16-20: Validate and sanitize falsePositiveRate before computing
Bloom parameters: in the function that computes mBits/k (where
falsePositiveRate, mBits, k, and n are used), check that falsePositiveRate is
strictly between 0 and 1 and n > 0; if not, either clamp falsePositiveRate into
a safe open interval (e.g. min = 1e-9, max = 1 - 1e-9) or return/throw an error,
then compute mF/mBits/k as before and ensure mBits is at least 8 (or >0) to
avoid later modulo by zero; update the code around the existing mF, mBits, k
calculations to perform this validation/sanitization before using
falsePositiveRate.

In `@src/libstore/include/nix/store/binary-cache-store.hh`:
- Around line 88-94: The header comment on BinaryCacheStore's bloom-filter field
overstates support for absolute URLs; update the comment (near the field
documented and references in init() and probe initialization) to state that the
BloomFilter URL is either an absolute path on the same cache root or a path
relative to the cache root and that external HTTP/HTTPS absolute URLs are not
supported by the probe initialization, or alternatively implement full
absolute-URL fetch support in the probe initialization code path (adjusting
init() and any probe init logic accordingly); prefer the simpler change: change
the comment to explicitly say "path relative to the cache root; external
http/https absolute URLs are not supported" and reference init()/probe
initialization behavior.

---

Nitpick comments:
In `@src/libstore/nar-info-disk-cache.cc`:
- Around line 467-474: The code casts (pos / 8) to int when calling
sqlite3_blob_read which can overflow for extremely large bloom filters; before
calling sqlite3_blob_read (in the loop over bitPositions inside
nar-info-disk-cache.cc) check that (pos / 8) <= INT_MAX (or otherwise fits in an
int) and if not, raise/throw a clear SQLiteError (or return false) so you never
pass a truncated negative offset to sqlite3_blob_read; reference the loop over
bitPositions and the sqlite3_blob_read call when adding the bounds check.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d30c0226-025b-4e6d-a546-03a368724fcd

📥 Commits

Reviewing files that changed from the base of the PR and between fe36456 and 14aba1f.

📒 Files selected for processing (18)

• doc/manual/source/SUMMARY.md.in
• doc/manual/source/protocols/binary-cache-bloom-filter.md
• doc/manual/source/protocols/nix-cache-info.md
• src/libstore-tests/nar-info-disk-cache.cc
• src/libstore/binary-cache-store.cc
• src/libstore/bloom-filter.cc
• src/libstore/http-binary-cache-store.cc
• src/libstore/include/nix/store/binary-cache-store.hh
• src/libstore/include/nix/store/bloom-filter.hh
• src/libstore/include/nix/store/http-binary-cache-store.hh
• src/libstore/include/nix/store/meson.build
• src/libstore/include/nix/store/nar-info-disk-cache.hh
• src/libstore/meson.build
• src/libstore/nar-info-disk-cache.cc
• src/nix/generate-bloom-filter.cc
• src/nix/meson.build
• src/nix/serve.cc
• tests/functional/binary-cache.sh

[coderabbitai]

🧹 Nitpick comments (1)

src/libstore/nar-info-disk-cache.cc (1)

443-452: 💤 Low value

Potential integer overflow in blob offset calculation for very large filters.

The pos / 8 value is cast to int on line 448, but pos is uint64_t and can theoretically exceed INT_MAX * 8 (~17GB). For realistic Bloom filter sizes this is unlikely to be an issue, but the cast could silently truncate and cause incorrect reads for pathologically large filters.

Consider using an explicit size check or documenting the implicit limit:

+            // Bloom filters larger than ~2GB are not supported due to
+            // sqlite3_blob_read offset limitations.
             forEachBloomBitPosition(path, params->k, params->mBits, [&](uint64_t pos) {
                 if (!allSet)
                     return;
                 unsigned char byte = 0;
-                if (sqlite3_blob_read(blob, &byte, 1, (int) (bloomFilterHeaderLen + pos / 8)) != SQLITE_OK)
+                auto offset = bloomFilterHeaderLen + pos / 8;
+                if (offset > INT_MAX)
+                    SQLiteError::throw_(state->db, "bloom-filter blob too large");
+                if (sqlite3_blob_read(blob, &byte, 1, static_cast<int>(offset)) != SQLITE_OK)
                     SQLiteError::throw_(state->db, "reading bloom-filter blob");

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/libstore/nar-info-disk-cache.cc` around lines 443 - 452, The loop in
forEachBloomBitPosition reads a byte with sqlite3_blob_read using
(int)(bloomFilterHeaderLen + pos / 8) which can overflow when pos (uint64_t)
produces an offset > INT_MAX; add an explicit bounds check before the
sqlite3_blob_read call: compute uint64_t offset = bloomFilterHeaderLen + pos /
8, verify offset <= INT_MAX (or otherwise handle/throw via SQLiteError::throw_),
then call sqlite3_blob_read with a safe cast to int; update the code around
forEachBloomBitPosition/pos/sqlite3_blob_read to use this checked offset so
reads cannot be truncated silently.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/libstore/nar-info-disk-cache.cc`:
- Around line 443-452: The loop in forEachBloomBitPosition reads a byte with
sqlite3_blob_read using (int)(bloomFilterHeaderLen + pos / 8) which can overflow
when pos (uint64_t) produces an offset > INT_MAX; add an explicit bounds check
before the sqlite3_blob_read call: compute uint64_t offset =
bloomFilterHeaderLen + pos / 8, verify offset <= INT_MAX (or otherwise
handle/throw via SQLiteError::throw_), then call sqlite3_blob_read with a safe
cast to int; update the code around
forEachBloomBitPosition/pos/sqlite3_blob_read to use this checked offset so
reads cannot be truncated silently.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 61c25c05-7634-41aa-87b6-e9658dd8496b

📥 Commits

Reviewing files that changed from the base of the PR and between 0dd8185 and c107b50.

📒 Files selected for processing (8)

• src/libstore/binary-cache-store.cc
• src/libstore/bloom-filter.cc
• src/libstore/include/nix/store/binary-cache-store.hh
• src/libstore/include/nix/store/bloom-filter.hh
• src/libstore/include/nix/store/nar-info-disk-cache.hh
• src/libstore/nar-info-disk-cache.cc
• src/nix/generate-bloom-filter.cc
• tests/functional/binary-cache.sh

💤 Files with no reviewable changes (1)

• src/nix/generate-bloom-filter.cc

🚧 Files skipped from review as they are similar to previous changes (3)

• tests/functional/binary-cache.sh
• src/libstore/include/nix/store/binary-cache-store.hh
• src/libstore/binary-cache-store.cc