fix: rebuild index when mode adds capabilities

[astandrik]

Summary

Fixes #1273
Related: #563

• keep the caller-requested discovery/reporting mode separate from the stronger effective coverage already stored in index_coverage_meta
• compare downgrade runs against a fresh semantic manifest at the stored effective scope
• rebuild at that effective scope when a stored-only input changes or disappears, without widening the caller-visible excluded/ignored report
• export explicit and existing persistent artifacts exactly once, after the replacement database is atomically published

What this PR completes

Current main already stores the canonical mode in index_coverage_meta and forces a rebuild when a stronger request needs capabilities missing from the stored generation. This PR does not reimplement that upgrade route. It completes the remaining downgrade half of #1273 and fixes artifact publication ordering in the same pipeline path.

Previously, preserving stronger stored coverage promoted the pipeline's single mutable mode before discovery. A full -> fast request therefore widened back to full discovery/reporting. Conversely, comparing only the requested subset could leave changed or deleted full-only inputs stale.

The pipeline now keeps requested_mode and effective mode separate. It reports exclusions from requested discovery, but when stored coverage is stronger it compares the stored generation with a fresh manifest built at that effective scope. An exact match remains an incremental no-op. Any stored-only change or deletion forces a complete effective-scope rebuild, after which the requested exclusion/ignored report is restored.

Artifact export is centralized at the outer publication boundary. No route exports from the staging database; explicit persistence and refresh of an existing artifact happen only after the final database rename.

Mode contract

Stored coverage	Requested mode	Result
weaker	stronger	existing current-main route rebuilds and records the stronger coverage
same	same	existing exact no-op / repair / rebuild routing
stronger	weaker, effective manifest unchanged	requested-scope reporting, stronger capabilities retained, incremental no-op
stronger	weaker, effective-only input changed or deleted	effective-scope full rebuild, requested-scope reporting retained

Cancellation or a staging/publication failure before the final rename preserves the previous database and artifact. An explicitly requested artifact-export failure is returned after the database has been published; automatic refresh of an existing artifact remains best-effort.

The public MCP API, SQLite schema, and artifact format are unchanged. index_coverage_meta remains the sole mode representation. This PR intentionally does not add duplicate Project-node metadata, special normalization of externally modified/corrupt mode values, or generic SQLite read-error hardening; those are separate artifact-integrity/recovery concerns rather than normal #1273 mode transitions.

Regression coverage

• an unchanged fast -> moderate run forces a full rebuild, creates SIMILAR_TO, records index_mode=moderate, and refreshes the artifact; a later fast request retains those capabilities
• full -> fast reports tools/ as excluded while retaining FULL stored coverage
• changed and deleted full-only inputs force an effective-FULL rebuild
• an unchanged downgrade takes the incremental no-op route
• cancellation preserves both the previous database and existing artifact; retry publishes the new generation
• explicit full, implicit forced-full, explicit no-op, and implicit no-op paths each export exactly once after the real rename
• explicit artifact-export failures still propagate after database publication

Verification

Exact published head: 3a4ad3a2 on base 43531195.

• exact current-main affected matrix: 343 passed (pipeline, pipeline_semantic_manifest_repro, artifact, and upstream mem)
• focused UBSan matrix on the byte-identical candidate: 42 passed
• post-rewrite compatibility audit: seven representation-independent scenarios from the previous implementation passed on the current product code (32 passed in a disposable focused fixture)
• fresh isolated fast -> moderate -> fast capability contract: 1 passed
• production build, formatting, NOLINT policy, git diff --check, diff-only clang-format, static security audit, cppcheck, and CI lint: passed
• local macOS ASan remained blocked inside sanitizer runtime initialization before any test; no product failure was observed

Exact-head GitHub CI is still running; clean-runner sanitizer, platform, and analyzer jobs remain authoritative.

[Copilot]

Pull request overview

This PR fixes incremental routing decisions across index mode transitions by persisting the effective index capability level in Project metadata (properties.index_mode), forcing a full rebuild when the requested mode requires capabilities not present in the stored index, and preserving stronger stored capabilities during downgrades (while still honoring the requested mode for discovery/exclusions). It also ensures persistent artifacts are refreshed after reindexing and adds regression coverage for mode upgrades/downgrades and malformed/legacy metadata handling.

Changes:

• Persist effective capability mode in the Project node (properties.index_mode) and use exact JSON string comparisons when reading it.
• Route mode upgrades to full rebuilds; keep stored effective mode for incremental changed-file re-extraction on downgrades.
• Refresh existing artifacts after reindexing (FAST quality unless persistence is explicitly requested) and add end-to-end tests for these transitions.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/test_pipeline.c	Adds regression tests covering upgrade/downgrade routing, capability preservation, malformed metadata, and artifact refresh behavior.
src/store/store.h	Clarifies that index_mode in coverage metadata is the requested discovery mode; effective capability mode is stored on the Project node.
src/pipeline/pipeline.c	Persists Project index_mode, reads stored mode safely via yyjson, routes upgrades to rebuilds, and adjusts post-publish artifact export behavior.
src/pipeline/pipeline_internal.h	Exposes cbm_pipeline_mode_name for consistent internal serialization and updates incremental API contract docs/signature.
src/pipeline/pipeline_incremental.c	Accepts an effective_mode for changed-file re-extraction, preserves macro extraction capability on downgrades, and propagates persistence failures when required.
src/mcp/mcp.c	Updates MCP tool schema description to document artifact refresh behavior when an artifact already exists.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[DeusData]

Thank you for the contribution and for covering both index mode upgrades and downgrades, including malformed metadata. This is now triaged as a high-priority graph correctness bug for 0.9.1-rc. Our community PR queue is currently quite full, so it may take a little time before we can complete the review and, if approved, merge it. We are doing our best to support community contributions and will return with code-grounded feedback as capacity opens.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/pipeline/pipeline.c:1527

• Assigning the stored mode to p->mode here also changes discovery, because cbm_pipeline_run_staged later passes p->mode to cbm_discover_ex2. A full → fast downgrade therefore still scans full-only directories/files instead of using the requested discovery scope, contradicting the stated requested/effective-mode separation and defeating the fast-mode workaround for large repositories. Keep the requested discovery mode separate and use the stored mode only for effective extraction/post-passes and persisted capability metadata.

        p->mode = stored_mode;

src/pipeline/pipeline_incremental.c:2426

• This exports the artifact during the inner staged run, but the successful return is then marked incremental and export_after_publish exports it again (BEST quality for explicit persistence, FAST for an existing artifact). Thus every exact no-op that needs an artifact performs compression/VACUUM twice, with the first export occurring before final publication. Let the existing post-publication path perform the single export.

            return cbm_pipeline_refresh_artifact(p, db_path);

[DeusData]

Merged (415a64a) — thank you for what was the technically strongest external PR in this batch. The export-exactly-once-after-rename fault-injection tests are the reason this could land as one unit: they prove the publish-boundary design under cancel/retry/delete, which is precisely the evidence a ~400-line rework of the pipeline's most critical routing needs. The requested/effective mode split completes #1273's downgrade half cleanly. We chose your centralized export over our own per-route refresh design — #1163 is being closed in its favor with credit.