|
| 1 | +# Feature System v2 — Stage 2: feature-activated optional dependencies (Design) |
| 2 | + |
| 3 | +Date: 2026-06-29 |
| 4 | +Status: **S2a implemented**; S2b (feature unification) next. |
| 5 | +Builds on: `.agents/docs/2026-06-29-feature-capability-model-design.md` |
| 6 | +Scope: `src/manifest.cppm` (parse), `src/build/prepare.cppm` (worklist resolution |
| 7 | ++ feature activation), `src/pm/dep_spec.cppm` (DepSpec reuse). |
| 8 | + |
| 9 | +## Implementation status |
| 10 | + |
| 11 | +- **S2a — DONE.** `Manifest.featureDeps` (`map<feature, map<depKey, DependencySpec>>`). |
| 12 | + Parsed from the TOML `[feature-deps.<name>]` section and from a Lua descriptor |
| 13 | + feature's nested `deps = { ["name"] = "ver" }`; Lua feature `implies` is now |
| 14 | + parsed too (was TOML-only). In `prepare_build`, two local lambdas |
| 15 | + `activateFeatures` / `mergeActiveFeatureDeps` merge a manifest's active |
| 16 | + feature-deps into its `dependencies` map — for the root before the worklist is |
| 17 | + seeded, and for each dependency right after its manifest loads (before its |
| 18 | + children are pushed). The existing worklist BFS then fetches/version-merges |
| 19 | + them, and Stage-3 capability binding finds a feature-pulled provider in the |
| 20 | + graph. Optional-by-default falls out for free (a dep declared only under a |
| 21 | + feature is never seen by the worklist unless the feature is active). |
| 22 | + Tests: `e2e/82_feature_optional_deps.sh`, `Manifest.FeatureDepsTomlSection`, |
| 23 | + `SynthesizeFromXpkgLua.FeatureDepsAndImplies`. |
| 24 | + |
| 25 | + > Implementation note: `activateFeatures`/`mergeActiveFeatureDeps` MUST be |
| 26 | + > local lambdas, not file-scope functions. As exported (inline) functions in |
| 27 | + > this module-interface unit their `std::map` instantiations leak into the |
| 28 | + > emitted BMI and trip a GCC-16 modules bug — *another* TU importing `std` |
| 29 | + > then fails with `fatal error: failed to load pendings for __normal_iterator`. |
| 30 | + > Keeping them local confines the instantiations to the implementation. |
| 31 | +
|
| 32 | +- **S2b — feature unification: NEXT.** Union feature requests per resolved |
| 33 | + package identity across the graph (today the first requester's features win). |
| 34 | + Needed for correct diamond behavior with feature-deps; called out separately |
| 35 | + below because it is the one genuine resolver-semantics change. |
| 36 | + |
| 37 | +--- |
| 38 | + |
| 39 | +## 1. Problem |
| 40 | + |
| 41 | +A feature cannot pull a dependency today. `[features]` entries parse `implies`, |
| 42 | +`defines`, `requires`, `provides` — the `deps` key is explicitly reserved |
| 43 | +("`requires`/`provides`/`deps` keys are reserved for later stages", |
| 44 | +`manifest.cppm`). So: |
| 45 | + |
| 46 | +- `requires = ["blas"]` (Stage 3) only **binds** a provider that is already in |
| 47 | + the dependency graph; it does not bring one in. |
| 48 | +- There is no way to say "*activating feature X pulls dependency Y*", which is |
| 49 | + the natural way to express optional backends (e.g. Eigen's `use_blas` wanting |
| 50 | + an external OpenBLAS), optional codecs, GPU backends, etc. |
| 51 | + |
| 52 | +This is **Stage 2** of the capability model. Implementing it makes the |
| 53 | +provider/consumer story self-contained: a single feature can pull a provider |
| 54 | +**and** bind the capability to it. |
| 55 | + |
| 56 | +## 2. Goals |
| 57 | + |
| 58 | +1. `features.<name>.deps` — a dependency listed under a feature is pulled **only |
| 59 | + when that feature is active**. A dependency under `[dependencies]` is always |
| 60 | + pulled (unchanged). Optionality is expressed by *where* the dep is declared, |
| 61 | + not a separate `optional = true` flag (simpler than Cargo). |
| 62 | +2. Works for the **root package** (`mcpp build --features X` pulls X's deps) and |
| 63 | + **transitively** (a dependency's active feature pulls that dependency's deps). |
| 64 | +3. **Composes with Stage 3**: a feature can `deps` a provider package **and** |
| 65 | + `requires` the capability it provides; the resolver then binds the just-pulled |
| 66 | + provider. This is the headline (`backend-openblas` below). |
| 67 | +4. **Feature unification** (Stage 2b): when the same package is reached with |
| 68 | + different feature sets from different consumers, the sets are **unioned**, so |
| 69 | + all feature-deps are pulled and all feature effects apply (Cargo's additive |
| 70 | + model). Replaces today's first-requester-wins behavior. |
| 71 | +5. No regression: packages with no feature-deps resolve exactly as today. |
| 72 | + |
| 73 | +## 3. Key insight — it rides the existing worklist BFS |
| 74 | + |
| 75 | +Dependency resolution in `prepare_build` is already a breadth-first worklist |
| 76 | +(`prepare.cppm:849` `WorkItem`, `:1547` seed from the root manifest, `:1560` |
| 77 | +`while (!worklist.empty())`). Each `WorkItem` carries the `DependencySpec` |
| 78 | +(including its requested `features`), the requester, and the consumer slot. |
| 79 | +Transitive deps are discovered by pushing a loaded manifest's `[dependencies]` |
| 80 | +onto the same worklist. |
| 81 | + |
| 82 | +So feature-deps need **no new resolution phase** and **no re-entrancy** (the |
| 83 | +earlier worry). They are simply *more deps pushed onto the worklist* at the |
| 84 | +moment a package's manifest is loaded and its active feature set is known: |
| 85 | + |
| 86 | +``` |
| 87 | +seed worklist: |
| 88 | + root [dependencies] (existing) |
| 89 | + + root active-feature deps (default ∪ --features) (NEW) |
| 90 | +
|
| 91 | +per worklist item (a dep whose manifest just loaded): |
| 92 | + push its [dependencies] (existing) |
| 93 | + + push its active-feature deps (NEW) |
| 94 | + active = item.spec.features ∪ dep.default ∪ implied(expanded) |
| 95 | +``` |
| 96 | + |
| 97 | +The BFS then fetches/resolves/version-merges the feature-deps exactly like any |
| 98 | +other dep. Stage 3 capability binding (already implemented, `prepare.cppm` |
| 99 | +`capProviders`/`capRequires`) runs after resolution and now finds the |
| 100 | +feature-pulled provider in `packages`, binding the capability to it — no Stage 3 |
| 101 | +change required. |
| 102 | + |
| 103 | +## 4. Data model |
| 104 | + |
| 105 | +Add to `Manifest` (next to `featuresMap` / `featureRequires`): |
| 106 | + |
| 107 | +```cpp |
| 108 | +// feature name → dependencies activated by that feature. A dep that appears |
| 109 | +// ONLY here (not in [dependencies]) is optional: resolved only when the |
| 110 | +// feature is active. Each entry is a full DependencySpec (version/path/git + |
| 111 | +// its own features/backend), so a feature-dep can itself request features. |
| 112 | +std::map<std::string, std::map<std::string, DependencySpec>> featureDeps; |
| 113 | +``` |
| 114 | + |
| 115 | +A `map<depKey, DependencySpec>` per feature mirrors `Manifest::dependencies`, so |
| 116 | +the same parse/merge/fetch code applies unchanged. |
| 117 | + |
| 118 | +## 5. Syntax |
| 119 | + |
| 120 | +### Lua descriptor (index packages — the primary surface) |
| 121 | + |
| 122 | +```lua |
| 123 | +features = { |
| 124 | + -- consumer capability switch (Stage 1+3, already supported) |
| 125 | + ["use_blas"] = { defines = { "EIGEN_USE_BLAS" }, requires = { "blas" } }, |
| 126 | + |
| 127 | + -- Stage 2: a backend convenience feature pulls a provider AND turns on the |
| 128 | + -- consumer switch. `deps` mirrors the top-level `deps` table shape. |
| 129 | + ["backend-openblas"] = { |
| 130 | + implies = { "use_blas" }, |
| 131 | + deps = { ["compat.openblas"] = "0.3.x" }, |
| 132 | + }, |
| 133 | +} |
| 134 | +``` |
| 135 | + |
| 136 | +### TOML project manifest (a project's own features) |
| 137 | + |
| 138 | +```toml |
| 139 | +[features] |
| 140 | +use_blas = { defines = ["EIGEN_USE_BLAS"], requires = ["blas"] } |
| 141 | +backend-openblas = { implies = ["use_blas"] } |
| 142 | + |
| 143 | +# Nested dep tables don't fit cleanly in a feature inline-table, so feature deps |
| 144 | +# get their own section, keyed by feature name. (Parser: read [feature-deps.*] |
| 145 | +# into Manifest.featureDeps with the existing dependency loader.) |
| 146 | +[feature-deps.backend-openblas] |
| 147 | +compat.openblas = "0.3.x" |
| 148 | +``` |
| 149 | + |
| 150 | +Rationale: the Lua surface accepts a nested `deps = { ... }` inside the feature |
| 151 | +table (the descriptor parser already walks nested tables). The TOML surface uses |
| 152 | +a dedicated `[feature-deps.<name>]` section because TOML inline tables nested |
| 153 | +inside a feature inline-table are awkward and the existing dependency loader |
| 154 | +(`load_deps`) can be pointed at `[feature-deps.<name>]` verbatim. |
| 155 | + |
| 156 | +## 6. Resolution flow (where it changes in `prepare_build`) |
| 157 | + |
| 158 | +``` |
| 159 | +1. toolchain → workspace (existing) |
| 160 | +2. Compute ROOT active features early (NEW, small) |
| 161 | + active_root = expand(default ∪ --features) |
| 162 | +3. Seed worklist: |
| 163 | + root [dependencies] (existing, :1547) |
| 164 | + + for f in active_root: root.featureDeps[f] (NEW) |
| 165 | +4. Worklist BFS (existing, :1560): |
| 166 | + for each item: fetch + load manifest (existing) |
| 167 | + compute the dep's active features: (NEW) |
| 168 | + active = expand(item.spec.features ∪ dep.default ∪ implied) |
| 169 | + push dep [dependencies] (existing) |
| 170 | + + push dep.featureDeps[active] (NEW) |
| 171 | + SemVer-merge / dedupe by identity (existing) |
| 172 | +5. Feature activation (defines/sources/MCPP_FEATURE_, :2049) (existing) |
| 173 | +6. Capability binding (0/1/many over in-graph providers) (existing, Stage 3) |
| 174 | + — now finds feature-pulled providers |
| 175 | +7. modgraph → plan → lockfile (existing) |
| 176 | +``` |
| 177 | + |
| 178 | +`expand(...)` is the existing `activate()` closure (`prepare.cppm:2064`) factored |
| 179 | +out so it can run during the worklist, not only at step 5. |
| 180 | + |
| 181 | +## 7. Feature unification (Stage 2b) |
| 182 | + |
| 183 | +Today, when a dependency is requested by more than one consumer, only the first |
| 184 | +requester's `features` are applied (`prepare.cppm` dep loop uses the first match |
| 185 | +then `break`). With feature-deps this is incorrect: consumer A may request |
| 186 | +`compat.eigen[backend-openblas]` while consumer B requests `compat.eigen[use_lapacke]`; |
| 187 | +both feature-deps must be pulled. |
| 188 | + |
| 189 | +Fix: accumulate the **union** of feature requests per resolved package identity |
| 190 | +across the whole worklist, and: |
| 191 | +- seed feature-deps for the union (so all optional deps are pulled), and |
| 192 | +- at step 5, activate the union (so all defines/sources/capabilities apply). |
| 193 | + |
| 194 | +The worklist already dedupes packages by identity and merges versions; unifying |
| 195 | +the feature set is the analogous merge on the feature axis. This is the one piece |
| 196 | +that is genuinely a resolver-semantics change (everything else is additive), so |
| 197 | +it is called out as its own sub-stage with its own tests. |
| 198 | + |
| 199 | +## 8. Worked example — OpenBLAS + Eigen (the headline) |
| 200 | + |
| 201 | +```lua |
| 202 | +-- compat.openblas (a real provider package) |
| 203 | +package = { |
| 204 | + name = "compat.openblas", |
| 205 | + provides = { "blas", "lapack" }, -- Stage 3 capability |
| 206 | + mcpp = { /* build that exposes -lopenblas, headers */ }, |
| 207 | +} |
| 208 | +``` |
| 209 | + |
| 210 | +```lua |
| 211 | +-- compat.eigen |
| 212 | +features = { |
| 213 | + eigen_blas = { sources = {"*/blas/*.cpp","*/blas/f2c/*.c"}, provides = {"blas"} }, |
| 214 | + use_blas = { defines = {"EIGEN_USE_BLAS"}, requires = {"blas"} }, |
| 215 | + use_lapacke = { defines = {"EIGEN_USE_LAPACKE"}, requires = {"lapack"} }, |
| 216 | + mpl2only = { defines = {"EIGEN_MPL2_ONLY"} }, |
| 217 | + -- Stage 2: one-liner backend that PULLS the provider and turns on the switch. |
| 218 | + ["backend-openblas"] = { |
| 219 | + implies = { "use_blas", "use_lapacke" }, |
| 220 | + deps = { ["compat.openblas"] = "0.3.x" }, |
| 221 | + }, |
| 222 | +} |
| 223 | +``` |
| 224 | + |
| 225 | +Consumer's `mcpp.toml`: |
| 226 | + |
| 227 | +```toml |
| 228 | +[dependencies] |
| 229 | +compat.eigen = { version = "5.0.1", features = ["backend-openblas"] } |
| 230 | +``` |
| 231 | + |
| 232 | +Resolution walk-through: |
| 233 | + |
| 234 | +1. Worklist seeds `compat.eigen` (with `features=["backend-openblas"]`). |
| 235 | +2. `compat.eigen` manifest loads. Active features expand: |
| 236 | + `backend-openblas` → `implies` → `use_blas`, `use_lapacke`. |
| 237 | +3. `backend-openblas.deps` → **push `compat.openblas@0.3.x`** onto the worklist. |
| 238 | +4. Worklist resolves `compat.openblas` → it `provides = ["blas","lapack"]`. |
| 239 | +5. Feature activation: `use_blas`/`use_lapacke` contribute `-DEIGEN_USE_BLAS` |
| 240 | + / `-DEIGEN_USE_LAPACKE` and `requires = ["blas"]` / `["lapack"]`. |
| 241 | +6. Capability binding (Stage 3): `blas`/`lapack` each have exactly one provider |
| 242 | + in the graph (compat.openblas) → bound. Its `-lopenblas` link/include flow to |
| 243 | + the consumer via usage requirements. |
| 244 | + |
| 245 | +Result: a single `features = ["backend-openblas"]` pulls OpenBLAS, defines the |
| 246 | +Eigen macros, binds the capability, and links the library — the full |
| 247 | +provider/consumer loop with no manual `[dependencies]` entry and no |
| 248 | +`[capabilities]` pin (one provider ⇒ unambiguous). |
| 249 | + |
| 250 | +Note the mutual-exclusion rule still holds: `backend-openblas` must NOT also |
| 251 | +imply `eigen_blas` (compiling Eigen's own BLAS while defining `EIGEN_USE_BLAS` |
| 252 | +is self-contradictory — see the v2 design doc). `backend-openblas` is the |
| 253 | +external-provider path; `eigen_blas` is the self-provider path; pick one. |
| 254 | + |
| 255 | +## 9. Edge cases |
| 256 | + |
| 257 | +- **Version conflict**: a feature-dep colliding with a top-level dep on a |
| 258 | + different version is handled by the existing SemVer merge across the worklist |
| 259 | + (no new logic). |
| 260 | +- **Optional-only dep absent when feature off**: a package referenced *only* in |
| 261 | + `featureDeps` and never activated is never fetched (the worklist never sees |
| 262 | + it) — the desired optional behavior, for free. |
| 263 | +- **Transitive feature-deps**: a feature-dep can itself carry `features=[...]`, |
| 264 | + whose feature-deps are pushed when that package is processed — natural BFS |
| 265 | + recursion. |
| 266 | +- **Cycles**: the worklist's existing identity seen-set breaks cycles. |
| 267 | +- **Dev-deps**: feature-deps are normal (non-dev) deps; they are not propagated |
| 268 | + through `[dev-dependencies]` rules. |
| 269 | +- **`--strict`**: requesting an undeclared feature already errors under strict; |
| 270 | + a feature-dep that fails to resolve surfaces the existing fetch error. |
| 271 | + |
| 272 | +## 10. Staging |
| 273 | + |
| 274 | +| Stage | Content | Resolver change | Unlocks | |
| 275 | +|---|---|---|---| |
| 276 | +| **S2a** | Parse `featureDeps`; seed root + push per-dep feature-deps onto the worklist; factor `activate()` for reuse. | Additive (push more onto the existing worklist). | `--features X` and dep `features=[X]` pull X's deps; composes with Stage 3 to auto-bind a pulled provider. | |
| 277 | +| **S2b** | Feature **unification**: union feature requests per package identity across the graph; activate + seed feature-deps for the union. | Semantics change (union vs first-wins). | Correct diamond behavior; multiple consumers' features all apply. | |
| 278 | + |
| 279 | +S2a alone already delivers the OpenBLAS+Eigen example (single consumer, single |
| 280 | +requester). S2b hardens multi-consumer graphs. |
| 281 | + |
| 282 | +## 11. Testing |
| 283 | + |
| 284 | +- **Parse** (`test_manifest`): `featureDeps` from the Lua descriptor and from |
| 285 | + `[feature-deps.<name>]`; a feature with no deps yields no entry. |
| 286 | +- **S2a e2e**: a root feature pulls a path-dep only when active (and not when |
| 287 | + inactive — assert the dep is absent from the build/lockfile); a dep's feature |
| 288 | + pulls a transitive path-dep. |
| 289 | +- **Composition e2e**: a `backend-*` feature that `deps` a provider + `requires` |
| 290 | + its capability resolves and binds with no explicit dependency/pin (the |
| 291 | + OpenBLAS+Eigen shape, using small synthetic provider/consumer packages like the |
| 292 | + existing `81_capability_binding.sh`). |
| 293 | +- **S2b e2e**: two consumers request the same package with different features; |
| 294 | + both feature-deps are pulled and both defines applied. |
| 295 | + |
| 296 | +## 12. Deliberately deferred |
| 297 | + |
| 298 | +- **Mutually-exclusive feature groups / `conflicts`** (e.g. forbidding |
| 299 | + `eigen_blas` + `use_blas`): documented in the recipe for now; a declarative |
| 300 | + `conflicts` is a separate, later addition (the single-valued capability slot |
| 301 | + already covers the backend case). |
| 302 | +- **`optional = true` on top-level deps + same-named auto-feature** (Cargo's |
| 303 | + other style): the `featureDeps` table covers the same need more directly; |
| 304 | + revisit only if a real case wants a top-level dep gated by an unrelated |
| 305 | + feature name. |
| 306 | +- **Weak features (`dep?/feat`)**: not needed until a concrete case appears. |
0 commit comments