From e3852cc629ea49d4b1c1f73502a48005af6c9824 Mon Sep 17 00:00:00 2001 From: Roger-luo Date: Thu, 16 Jul 2026 23:47:52 -0400 Subject: [PATCH 01/10] docs(design): document traits 2 refactor --- docs/design/tableau-data-structure.md | 329 ++++++++++++ .../traits-2-configuration-and-hashing.md | 508 ++++++++++++++++++ docs/design/word-data-structures.md | 357 ++++++++++++ 3 files changed, 1194 insertions(+) create mode 100644 docs/design/tableau-data-structure.md create mode 100644 docs/design/traits-2-configuration-and-hashing.md create mode 100644 docs/design/word-data-structures.md diff --git a/docs/design/tableau-data-structure.md b/docs/design/tableau-data-structure.md new file mode 100644 index 000000000..7b283359f --- /dev/null +++ b/docs/design/tableau-data-structure.md @@ -0,0 +1,329 @@ +# Contiguous tableau data structure + +Status: design sketch + +## Purpose + +The tableau should be a specialized bit-matrix data structure. It should not +be represented as `Vec>`, and its internal rows should not +affect the shared PPVM trait system. + +This design separates: + +- the logical stabilizer-tableau model; +- its physical, contiguous memory layout; +- orientation changes used to accelerate different operations; and +- structural hashing used when a tableau is a classical-mixture key. + +Gate, noise, measurement, reset, and `Indexable` traits observe tableau +behavior. They do not expose matrix blocks, row values, strides, alignment, or +orientation. + +## Goals + +- Store the X/Z tableau bits in contiguous, aligned, bit-packed memory. +- Make one- and two-qubit column operations efficient. +- Permit temporary transposition for row-oriented elimination and collapse. +- Keep phases and optional loss state independently addressable and hashable. +- Hash the logical tableau independently of its current physical orientation. +- Avoid parameterizing the trait system by a row type or Pauli-word storage. +- Leave room for SIMD-width and padding changes without changing public + behavioral traits. + +## Non-goals + +- Reusing the standalone `PauliWord` representation for tableau rows. +- Exposing borrowed tableau rows as the primary public interface. +- Standardizing a general-purpose matrix trait in `ppvm-traits-2`. +- Maintaining both row-major and column-major copies before benchmarks show + that the memory and synchronization cost is worthwhile. +- Deciding in this document whether PPVM should store a forward or inverse + tableau. Orientation and inversion are independent design choices. + +## Logical model + +For `n` qubits, a stabilizer/destabilizer tableau has `2n` generators. Each +generator contains an X bit and a Z bit for every qubit, plus a phase: + +```text + qubit + 0 1 2 ... n-1 +generator 0 x/z x/z x/z +generator 1 x/z x/z x/z + ... +generator 2n-1 x/z x/z x/z + +phase one value per generator +``` + +The logical state consists of: + +- an X matrix of shape `(2n, n)`; +- a Z matrix of shape `(2n, n)`; +- a phase plane of length `2n`; and +- for lossy variants, a loss plane of length `n` or another explicitly defined + logical shape. + +This model does not imply a physical row object. + +## Physical storage + +The first prototype should use one aligned contiguous allocation for the bit +planes, divided by computed offsets: + +```rust +pub struct TableauData { + blocks: AlignedVec, + x_offset: usize, + z_offset: usize, + phase_offset: usize, + loss_offset: Option, + major_stride: usize, + n_qubits: usize, + orientation: Orientation, +} +``` + +`Block` is an internal implementation choice such as `u64` or a SIMD-width +block. It is not an associated type of a public tableau trait. Offsets and +strides account for alignment and padding, while logical accessors enforce the +actual `(2n, n)` dimensions. + +Keeping all planes in one allocation improves cloning and locality and avoids +one allocation per generator. It also lets a mixture branch copy a single +contiguous region. If benchmarks favor separate aligned allocations for X and +Z, that change remains internal to `TableauData`. + +Padding must either be kept zero or excluded from equality and hashing. Zeroed +padding is preferable because it permits bulk comparison and hashing of +canonical ranges. + +## Column-major orientation + +The default orientation should make the generator dimension contiguous for a +fixed qubit. In other words, the X and Z planes are column-major with respect +to the logical `(generator, qubit)` matrix: + +```text +qubit 0: X bits for generators 0..2n, then padding +qubit 1: X bits for generators 0..2n, then padding +... + +qubit 0: Z bits for generators 0..2n, then padding +qubit 1: Z bits for generators 0..2n, then padding +... +``` + +This layout makes a selected qubit column contiguous. Single-qubit gates can +load the X and Z columns, update them with bitwise operations, and update the +affected phase bits. Two-qubit gates operate on two pairs of contiguous +columns. Measurement can scan the selected anticommutation column without +stepping across separately allocated row objects. + +Stim uses aligned SIMD bit tables and documents its tableau layout as +column-major, with output-observable iteration following contiguous memory. It +also provides an explicit quadrant transpose and a transposition guard for +operations that need the opposite orientation: + +- [Stim `Tableau`](https://github.com/quantumlib/Stim/blob/main/src/stim/stabilizers/tableau.h) +- [Stim `TableauSimulator`](https://github.com/quantumlib/Stim/blob/main/src/stim/simulators/tableau_simulator.h) +- [Stim `simd_bit_table`](https://github.com/quantumlib/Stim/blob/main/src/stim/mem/simd_bit_table.h) + +Column-major storage is only one part of Stim's performance strategy. PPVM +should benchmark its own gate, measurement, and sampling workloads instead of +assuming the same total performance from layout alone. + +## Temporary transposition + +Row multiplication and elimination need long generator rows and therefore +prefer the opposite orientation. The initial design should transpose the X/Z +quadrants temporarily instead of storing two permanent copies: + +```rust +pub enum Orientation { + ColumnMajor, + RowMajor, +} + +pub struct TransposedTableau<'a> { + tableau: &'a mut Tableau, +} +``` + +Creating the guard transposes the bit matrices and marks the physical +orientation. Dropping it restores the canonical column-major orientation: + +```rust +impl Drop for TransposedTableau<'_> { + fn drop(&mut self) { + self.tableau.restore_column_major(); + } +} +``` + +Operations that require row-major access receive the guard, making the +orientation precondition explicit. Public methods return with the tableau in +canonical orientation. Panic safety must be preserved by the guard's `Drop` +implementation. + +Transposition is a physical reordering of the same logical bits. It does not +invalidate structural hashes or change equality. Hashing should occur only +through logical access or while the tableau is in its public canonical state. + +Maintaining both orientations with dirty flags remains a future alternative +for workloads that switch frequently enough to amortize the doubled storage. + +## Tableau API boundary + +The public API exposes logical operations instead of storage slices: + +```rust +impl Tableau { + pub fn n_qubits(&self) -> usize; + + pub fn h(&mut self, qubit: usize); + pub fn cnot(&mut self, control: usize, target: usize); + pub fn measure_z(&mut self, qubit: usize) -> bool; + + pub fn x_bit(&self, generator: usize, qubit: usize) -> bool; + pub fn z_bit(&self, generator: usize, qubit: usize) -> bool; + pub fn phase(&self, generator: usize) -> Phase; +} +``` + +Logical bit accessors are useful for tests, serialization, debugging, and +interoperability. They are not intended as the hot gate implementation path. +Bulk import/export methods may use a canonical serialized representation +without exposing the in-memory layout. + +There should be no `stabilizers_mut() -> &mut [Row]` or equivalent escape hatch +that bypasses hash invalidation and orientation invariants. Specialized +internal row operations operate through `TableauData` or a transposition guard. + +## Gate access patterns + +The implementation should categorize mutations by physical access and hash +effect: + +| Operation | Preferred access | X/Z changed | Phase changed | +| --- | --- | --- | --- | +| Pauli `X`, `Y`, `Z` | column | no | possibly | +| `H`, `S` | one column pair | yes | possibly | +| `CNOT`, `CZ` | two column pairs | yes | possibly | +| Find measurement pivot | column | no | no | +| Row multiplication | transposed row | yes | possibly | +| Collapse/elimination | column scan + transposed rows | yes | yes | +| Physical transpose | bulk matrix | logically no | no | + +This table describes logical mutations. A gate may determine that no phase bit +changed and preserve the phase cache in that special case, but conservative +component invalidation is correct for the first implementation. + +## Structural hashing + +A tableau used in a classical mixture is an `Indexable` key. It owns its own +hasher and cache representations; neither is inherited from `PauliWord`. + +The structural hash is composed from independent logical components: + +```text +tableau hash = combine(xz hash, phase hash[, loss hash]) +``` + +The X and Z planes share an `xz_hash` cache because most Clifford mutations +update them together. The phase plane has a separate cache so Pauli +conjugations and sign changes do not force a matrix rehash. Loss should use a +third cache if it is independently mutable and part of key identity. + +```rust +pub struct Tableau { + data: TableauData, + xz_hash: XzHashCache, + phase_hash: PhaseHashCache, + loss_hash: Option, + rng: SmallRng, +} +``` + +The cache types above are concrete choices made by the tableau author. The +shared `Indexable::HashCache` associated type may name a small aggregate or +another representation chosen by the implementation. + +Equality and hashing include logical qubit count, generator order, all logical +X/Z bits, phases, and loss state when present. They exclude: + +- RNG state; +- allocation capacity; +- alignment padding; +- cache values and validity flags; and +- current physical orientation. + +The component invalidation rules are: + +| Mutation | X/Z cache | Phase cache | Loss cache | +| --- | --- | --- | --- | +| Pauli `X`, `Y`, `Z` | preserve | invalidate if changed | preserve | +| Direct phase change | preserve | invalidate | preserve | +| `H`, `S`, `CNOT`, `CZ` | invalidate | invalidate if changed | preserve | +| Row multiplication | invalidate | invalidate if changed | preserve | +| Mark or clear loss | preserve | preserve | invalidate | +| Physical transpose | preserve | preserve | preserve | +| RNG update | preserve | preserve | preserve | + +The current `ppvm-tableau-sum` split between `word_fingerprint` and +`phase_loss_hash` is evidence that component hashing matters. The new tableau +owns these components directly and separates loss as well when the logical +model permits it. + +## Cloning and mixture use + +Classical-mixture branching can clone tableaus frequently. Contiguous backing +storage makes cloning a bulk memory copy. A clone may copy valid hash caches +because it initially has identical logical contents. Subsequent mutation of +the branch invalidates only the affected components. + +Copy-on-write backing storage may be evaluated later, but it should not be part +of the initial design: most branches are mutated immediately, which may turn +reference counting and deferred copying into overhead. + +An indexable tableau must not be structurally mutated while stored as a map +key. Mixture storage removes or clones a key before applying gates and inserts +the resulting tableau under its updated hash. + +## Sampling implications + +Column-major X/Z planes make fixed-qubit queries and bit-parallel gate updates +efficient. They are also compatible with scanning many generators during a +measurement. Temporary transposition makes elimination and row products +contiguous when required. + +This layout should be evaluated separately from higher-level sampling +algorithms. Stim also uses an inverse tableau and reference-frame sampling; +those algorithmic choices are not consequences of column-major storage and do +not belong in the PPVM trait system. + +## Prototype validation + +The prototype should include: + +- property tests comparing gates and measurements with the existing tableau; +- round-trip tests for column-major -> row-major -> column-major transpose; +- equality and hash tests across physical orientations; +- tests proving phase-only changes preserve the X/Z hash cache; +- tests proving padding never affects equality or hashing; +- benchmarks for one- and two-qubit Clifford gates; +- benchmarks for deterministic and random measurement paths; +- benchmarks for clone-and-mutate mixture branching; and +- benchmarks comparing permanent row-major, permanent column-major, and + temporary-transpose variants on representative circuits. + +## Open questions + +1. What block width and alignment should the first implementation use? +2. Should phases occupy one or two bits per generator in the tableau model? +3. Should loss be stored per qubit, per generator, or outside the core tableau? +4. Which operations should receive a transposition guard versus performing + column-strided work directly? +5. Does a dual-orientation representation outperform temporary transposition + for PPVM's measurement-heavy workloads? +6. Should PPVM ultimately store a forward or inverse tableau? diff --git a/docs/design/traits-2-configuration-and-hashing.md b/docs/design/traits-2-configuration-and-hashing.md new file mode 100644 index 000000000..5d242a2ae --- /dev/null +++ b/docs/design/traits-2-configuration-and-hashing.md @@ -0,0 +1,508 @@ +# `ppvm-traits-2`: type composition, indexable values, and cached hashing + +Status: design sketch + +## Motivation + +The current `ppvm_traits::Config` bundles choices from several unrelated +layers: + +- coefficient type; +- packed Pauli storage; +- Pauli-word representation; +- key hasher; +- truncation strategy; and +- concrete map implementation. + +This makes the foundational configuration specific to `PauliSum`, even though +the gate and noise traits are shared by other algorithms. In particular, a map +and a truncation strategy are not properties of quantum data. They are choices +made by a particular algorithm. + +The second trait-system experiment should separate: + +1. the coefficient type, passed directly as a generic parameter; +2. concrete quantum-data representations; +3. the hashing contract of values that can be used as keys; and +4. algorithm-specific storage and policy choices. + +The redesign should remain compile-time generic. It should not introduce +runtime trait objects or runtime dispatch for these choices. + +This proposal was compared against the current definitions in +`ppvm-traits/src/config.rs`, `traits/map.rs`, `traits/strategy.rs`, and +`traits/word_trait.rs`; `ppvm-pauli-sum/src/sum/data.rs`; the concrete word +types in `ppvm-pauli-word`; and `ppvm-tableau-sum`'s `EntryStore`, `VecStorage`, +and `MapStorage`. Existing names are retained below unless the abstraction or +responsibility changes. + +## Type-composition layers + +### Coefficient type + +There is no algorithm-agnostic `Config` trait. Algorithms use the coefficient +type directly: + +```rust +pub struct SomeAlgorithm { + // ... +} +``` + +A trait containing only `type Coeff` adds indirection without providing useful +composition. Maps, policies, Pauli-word storage, Pauli-word +implementations, and hashers are also selected independently rather than being +collected into a replacement global configuration trait. + +### Representation types + +There is no separate global storage configuration, and representation storage +does not appear as an associated type. A concrete value encapsulates its own +fields; generic algorithms use behavioral methods instead of naming or +inspecting the backing memory. + +`Word` is the common concept for an indexed algebraic monomial. The old +Pauli-word operations are not Pauli-specific: every supported word has a site +alphabet, an indexed extent, site access and mutation, and a weight: + +```rust +pub trait Word { + type Site; + + fn n_sites(&self) -> usize; + fn get(&self, index: usize) -> Self::Site; + fn set(&mut self, index: usize, site: Self::Site); + fn weight(&self) -> usize; +} +``` + +`Site` selects the operator alphabet without introducing `PauliWord` or +`FermionWord` subtraits. For example, the relevant concrete types may be: + +```rust +pub enum Pauli { + I, + X, + Y, + Z, +} + +pub enum LossySite { + Present(S), + Lost, +} + +pub struct FermionSite { + pub mode: usize, + pub action: FermionAction, +} +``` + +Thus an ordinary packed Pauli word implements `Word`, a lossy +wrapper implements `Word>`, and a future ordered +fermionic product can implement `Word`. A fermionic word's +index denotes factor order; `FermionSite` carries the physical mode. For a +dense Pauli word, the index is the qubit and `n_sites()` is its width. +`weight()` is the number of non-identity factors according to the concrete site +alphabet; an ordered representation that stores no explicit identities may +therefore have `weight() == n_sites()`. Implementations of `set()` preserve +their representation invariants and invalidate the affected hash components. + +The concrete `WithLoss` family provides loss mutation and +`loss_weight()` as inherent methods. Loss channels and loss-specific +truncation specialize directly on `WithLoss`; there is no one-implementation +`LossyPauliWord` capability trait. + +Concrete packed Pauli, lossy, phased, and hash-cache layouts are described in +[`word-data-structures.md`](word-data-structures.md). None of those layouts is +visible through `Word`. + +A tableau is an independent concrete representation. It does not contain a +public row type, implement `Word`, or select an associated word +implementation. Its X/Z matrices, phases, orientation, and contiguous backing +allocation are private implementation details described in +[`tableau-data-structure.md`](tableau-data-structure.md). + +### Behavioral traits + +Shared traits describe operations, not representation layout. Clifford gates +need no coefficient parameter. Operations that consume numeric parameters use +the coefficient type directly: + +```rust +pub trait Clifford { + fn h(&mut self, qubit: usize); + fn cnot(&mut self, control: usize, target: usize); + // ... +} + +pub trait RotationOne { + fn rx(&mut self, qubit: usize, theta: C); + // ... +} + +pub trait PauliError { + fn pauli_error(&mut self, qubit: usize, probabilities: [C; 3]); +} +``` + +The same concrete tableau may implement a numeric trait for every supported +coefficient type without storing that coefficient type. Measurement and reset +traits likewise expose behavior and result types without exposing tableau +rows, packed matrix blocks, or matrix orientation. + +There is no shared `TableauStorage` trait in the first design. If multiple +tableau implementations are later useful, each concrete type can implement +the same behavioral traits. A storage abstraction should only be introduced +after two implementations demonstrate a common interface. + +### Algorithm and storage parameters + +An algorithm should take its independent choices as direct type parameters. +An associated-type bundle is not useful merely because it replaces two type +parameters with one. In particular, there is no `PauliSumAlgorithm` trait that +bundles a term map with a policy: storage layout and policy are orthogonal +choices. + +The reusable sparse-sum shape is: + +```rust +pub struct OperatorSum +where + C: Coefficient, + W: Word + Indexable, + S: SumStorage, + P: Policy, +{ + storage: S, + policy: P, + n_sites: usize, +} +``` + +Here `C`, `W`, `S`, and `P` respectively mean coefficient domain, algebraic +word, concrete sparse-sum storage engine, and algorithm policy. Each +parameter has an independent meaning. Propagation methods select their algebra +through the site type, for example `W: Word` or +`W: Word`. + +`Policy` is the proposed name for the current `Strategy` concept. It retains +the current responsibilities: predicting initial capacity and truncating the +sum. Existing concrete strategies become policies without otherwise changing +their meaning; `NoStrategy` and `CombinedStrategy` become `NoPolicy` and +`CombinedPolicy`, while `MaxPauliWeight` and `CoefficientThreshold` keep their +established names: + +```rust +pub trait Policy: Default + Clone + Copy +where + W: Word + Indexable, + C: Coefficient, +{ + fn capacity(&self, n_sites: usize) -> usize; + + fn truncate(&self, map: &mut M) + where + M: ACMap; +} +``` + +`Policy` and its concrete implementations belong to the sparse-sum crate. This +removes the current split where the `Strategy` trait lives in `ppvm-traits` but +its concrete strategies live in `ppvm-pauli-sum`; the policy is not an +algorithm-agnostic `ppvm-traits-2` concern. + +`ACMap` remains the name of the associative coefficient-map capability already +implemented by `HashMap`, `IndexMap`, and `DashMap`. Its generic signature can +be simplified after `PauliStorage` and the separately supplied build hasher are +removed, but coefficient accumulation, iteration, insertion, retention, and +consumption are the same concept. `ACMap` moves with the sparse-sum engine (the +existing `ppvm-pauli-sum` initially) rather than being renamed or kept in the +algorithm-agnostic trait crate. Its existing capability names—such as +`ACMapBase`, `ACMapIter`, `ACMapAddAssign`, `ACMapInsert`, `ACMapRetain`, and +`ACMapConsume`—should also remain unless implementation work shows that two +capabilities should actually be merged or split. + +A `SumStorage` is a new abstraction extracted from the fields currently owned +directly by `PauliSum`: its maps and reusable workspace. It is an actual value, +not a marker configuration: + +```rust +pub trait SumStorage: Clone +where + W: Word + Indexable, + C: Coefficient, +{ + type Map: ACMap; + + fn data(&self) -> &Self::Map; + fn data_mut(&mut self) -> &mut Self::Map; + + fn map_insert(&mut self, f: F) + where + F: Fn(&W, &mut C) -> Option<(W, C)>; + + fn map_insert_multiple(&mut self, f: F) + where + F: Fn(&W, &mut C) -> Option>; + + fn map_add(&mut self, f: F) + where + F: Fn(&W, &C) -> (W, C); + + fn consume(&mut self); +} +``` + +The exact closure bounds and support for multiple produced terms remain an +implementation detail for the prototype. The important boundary is that the +trait preserves the current semantic operation names without exposing physical +auxiliary maps or scratch buffers. + +The generalized engine may be named `OperatorSum`, but the Pauli specialization +retains the existing domain-facing `PauliSum` name. This is a new internal +generalization, not a requirement to rename Pauli call sites. + +A classical tableau mixture follows the same principle and takes its +`EntryStore` directly rather than introducing a one-associated-type +`TableauMixtureAlgorithm` bundle: + +```rust +pub struct GeneralizedTableauSum +where + C: Coefficient, + T: Indexable, + S: EntryStore, +{ + entries: S, +} +``` + +`GeneralizedTableauSum` and `EntryStore` retain their current names because the +proposal does not change their underlying roles. `GeneralizedTableauSum` and +`OperatorSum` are both sparse linear combinations of indexable keys, so they +may eventually share an implementation. This iteration deliberately keeps them +separate: their mutation, branching, normalization, and storage requirements +have not yet been reduced to a proven common interface. The next design +iteration should look for the smallest useful common factor and merge only that +factor, rather than assuming that the two complete algorithms are identical. + +Every keyed store must use the build hasher associated with its key type. + +### Compatibility with current names + +The redesign is not a vocabulary reset. The following names are retained or +changed according to whether their underlying responsibility changes: + +| Current implementation | Proposal | Rationale | +| --- | --- | --- | +| `Config` | removed | The bundle itself is removed; this is not a rename. | +| `PauliWordTrait` | `Word` plus `Indexable` where used as a key | Word operations are generalized through `Word::Site`; hashing becomes a separate capability. | +| `n_qubits`, `get`, `set`, `weight` | `n_sites`, `get`, `set`, `weight` | Only the Pauli-specific extent name changes; the other operation names stay. | +| concrete `PauliWord` | `PauliWord` | The packed X/Z word is the same domain concept. | +| concrete `LossyPauliWord` | `LossyPauliWord` alias over `WithLoss` | The public concept stays; only the implementation becomes a generic wrapper. | +| `PhasedPauliWord` | `PhasedPauliWord` alias over `Phased` | The public Pauli name stays; the implementation becomes a word-generic wrapper. | +| `rehash` | `invalidate_hash` for lazy modes | Recalculation changes from eager mutation-time work to lazy demand-time work. | +| `Strategy` | `Policy` | Intentional terminology change requested for this redesign. | +| `ACMap` | `ACMap` | The associative coefficient map has the same role. | +| `PauliSum::data`, `map_insert`, `map_add` | same method names on `SumStorage` | These semantic operations already match the proposed boundary. | +| `PauliSum` map pair and `scratch` fields | `SumStorage` | A new abstraction is extracted from currently unnamed storage state. | +| `PauliSum` | `PauliSum` over generalized `OperatorSum` machinery | Pauli-facing code keeps its established name; `OperatorSum` names the new cross-algebra engine. | +| `GeneralizedTableauSum` | `GeneralizedTableauSum` | The classical mixture algorithm remains the same concept. | +| `EntryStore`, `VecStorage`, `MapStorage` | unchanged | The proposal uses the existing storage boundary and implementations. | +| `BuildHasher`, `HashFinalize` | unchanged | The hasher concepts remain, but ownership moves from `Config` to each indexable key type. | +| `PauliStorage` | removed | Packed backing storage becomes private to the concrete word representation. | + +Names such as `Word`, `Indexable`, `SumStorage`, `WithLoss`, and +`OperatorSum` are therefore new because they denote abstractions that do not +exist in the current implementation, not because the existing API is being +renamed wholesale. + +### Trait admission rule + +A proposed trait belongs in the design only when generic code consumes it and +there are multiple meaningful implementation families, or when it is an +established behavioral boundary implemented by different backends. A trait is +not justified merely to name inherent methods on one generic struct. + +This rule keeps: + +- `Indexable`, consumed by keyed stores and implemented by hash-enabled words + and tableaus; +- `Word`, consumed by the shared sparse-sum engine and propagation algorithms; +- gate and noise traits, implemented across propagation and tableau backends; +- `SumStorage` and `ACMap`, implemented by genuinely different storage engines + and collections; +- `EntryStore`, already implemented by `VecStorage` and `MapStorage`; and +- `Policy`, implemented by independent capacity and truncation behaviors. + +It rejects the removed global `Config`, `PauliSumAlgorithm`, +`TableauMixtureAlgorithm`, and `TableauStorage` traits, as well as word +subtraits named `PauliWord`, `FermionWord`, or `LossyPauliWord`. Their +distinctions are expressed by `Word::Site` or by concrete wrapper types instead +of one-alphabet subtraits; the concrete `PauliWord` and `LossyPauliWord` type +names remain available. + +### Sparse-sum branch staging + +A propagation rule can turn one term into multiple terms. For example, a +Pauli rotation may produce: + +```text +c P -> c cos(theta) P + c sin(theta) P' +``` + +The existing entry can be updated while the active map is traversed, but a new +key cannot generally be inserted into that same map during mutable iteration. +New terms must therefore be staged and merged after the traversal. New keys +may collide with each other or with existing keys, so the merge accumulates +their coefficients. + +Only one staging mechanism is required for correctness. The current engine +uses two because they serve different performance paths: + +- an auxiliary map supports whole-map rewrites, combines output collisions as + they are produced, and retains its allocation across operations; and +- a reusable `Vec<(W, C)>` scratch buffer stages additional terms when existing + entries remain in place, avoiding an auxiliary-map insertion followed by a + second map probe during the merge. + +Conceptually: + +```text +whole-map rewrite: active map -> auxiliary map -> swap +in-place branching: active map + scratch buffer -> merge into active map +``` + +Neither physical mechanism is part of the public storage contract. A default +`SumStorage` implementation may privately contain both maps and the reusable +vector, matching the current `PauliSum` layout. A simpler backend may implement +both `map_insert` and `map_add` using only an auxiliary map; another backend +may use per-thread buffers. Whether retaining both mechanisms is worthwhile is +a benchmark decision, not a type-system requirement. + +## Indexable values + +Hash-enabled `Word` values and `Tableau` can both be expensive, mostly-stable +map keys. Their hashing contract should be expressed independently of any map. + +The tentative common capability is: + +```rust +pub trait Indexable: Clone + Eq + Hash { + type BuildHasher: std::hash::BuildHasher + Clone + Default + HashFinalize; + type HashCache; + + fn invalidate_hash(&mut self); +} +``` + +The important points are: + +- `BuildHasher` retains the current associated-type name and moves from + `Config` to the key type; +- the build hasher is associated with the key type, not with a configuration + bundle; +- the hash cache is an associated type, not a framework-provided concrete + `HashCache` structure; +- the author of the concrete data type owns the cache layout and invalidation + implementation; and +- equality and hashing cover only structural key identity, never cache fields + or incidental runtime state. + +For example, a word implementation may choose `u64`, `Option`, an atomic +representation, or another type. `ppvm-traits-2` does not prescribe the +fields that store it. + +### Lazy hashing and interior mutability + +Rust's `Hash::hash` receives `&self`. A cache that is first populated from +inside `Hash::hash` therefore requires interior mutability. Possible concrete +representations include: + +- `Cell` plus `Cell` for single-threaded keys; +- atomics for keys that must remain `Sync`; or +- a plain `u64` plus `bool` when cache preparation is eager or explicitly + performed through `&mut self` before hashing. + +This is deliberately a representation-level decision. A cache using interior +mutability will generally prevent the containing value from being `Copy`. + +Multiple concurrent readers may compute the same invalid hash more than once; +that is acceptable as long as they publish the same structural hash safely. +Structural mutation still requires `&mut self`. + +### Key mutation invariant + +An indexable value must not be structurally mutated while it is stored as a map +key. Cache invalidation makes a value correct for its next insertion or lookup; +it cannot make in-place mutation of an existing map key valid. + +Structural fields should therefore be private in the new representations. +Mutation must go through operations that invalidate the affected cache, or +through mutation guards that invalidate on completion. + +## Concrete word hashing + +Every concrete `Word` owns its `BuildHasher`, cache representation, structural +hash algorithm, and invalidation logic through `Indexable`. Pauli words hash +their X/Z content, lossy words compose Pauli and loss components, and future +fermion words hash their ordered factors. Factor order is part of fermionic +identity. + +The trait layer does not expose packed storage to support hashing. Concrete +implementations hash their private fields and may apply hasher-specific +finalization internally. Detailed layouts and component invalidation rules are +in [`word-data-structures.md`](word-data-structures.md). + +## Tableau indexability + +A tableau may itself be used as a key by a classical-mixture algorithm, so the +concrete tableau implements `Indexable` directly and owns a tableau-specific +hasher and cache representation. This does not imply that a tableau is a +`Word`; they only share the `Indexable` key capability. + +The tableau's structural hash is composed from its logical X/Z matrix, phase +plane, and optional loss plane. It excludes RNG, padding, cache state, and +physical matrix orientation. Separate component caches allow phase-only +changes to avoid rehashing the X/Z matrix. Physical transposition is a layout +change, not a logical mutation, and does not invalidate the structural hash. + +The concrete memory layout, component invalidation table, and canonical hash +order live in [`tableau-data-structure.md`](tableau-data-structure.md) so they +do not leak into the shared trait-system design. + +## Expected generic composition + +The intended composition is explicit: + +```rust +OperatorSum +Tableau +GeneralizedTableauSum +``` + +Domain-specific aliases or wrappers can preserve `PauliSum` and introduce +`FermionSum` without rebuilding a monolithic configuration trait. + +## Non-goals for the first prototype + +- Migrating the existing crates to `ppvm-traits-2` immediately. +- Merging `GeneralizedTableauSum` and `OperatorSum` in this iteration; only a + smaller proven common factor should be considered in the next iteration. +- Defining one collection interface shared by all algorithms. +- Requiring every sparse-sum storage backend to physically contain both an + auxiliary map and a scratch buffer. +- Selecting a single cache representation for every key type. +- Preserving `Copy` at the expense of correct lazy caching. +- Adding runtime dispatch for storage, hashing, or algorithm policies. + +## Open design questions + +1. What is the minimal method surface of `Indexable` beyond the associated + `BuildHasher` and `HashCache` types? Cache mechanics should remain owned by + the concrete type, but algorithms may need a common invalidation operation. +2. Which cache representations must support `Send` and `Sync` in the first + prototype? +3. Do benchmarks justify retaining both the auxiliary-map and vector-staging + fast paths in the default sparse-sum storage backend? diff --git a/docs/design/word-data-structures.md b/docs/design/word-data-structures.md new file mode 100644 index 000000000..5ece8e5c3 --- /dev/null +++ b/docs/design/word-data-structures.md @@ -0,0 +1,357 @@ +# Word data structures + +Status: design sketch + +## Purpose + +This document describes concrete data structures for standalone algebraic +words. The shared trait design is specified in +[`traits-2-configuration-and-hashing.md`](traits-2-configuration-and-hashing.md). + +The trait-level `Word` is representation-free but defines the common indexed +product operations: + +```rust +pub trait Word { + type Site; + + fn n_sites(&self) -> usize; + fn get(&self, index: usize) -> Self::Site; + fn set(&mut self, index: usize, site: Self::Site); + fn weight(&self) -> usize; +} +``` + +There is no `Word::Storage` associated type. Packed arrays, loss masks, ordered +factors, hash fields, and validity flags are private fields of concrete types. +Generic propagation and collection code uses behavioral traits and never names +the backing memory. + +`Word` does not extend `Indexable`. A word used as an `ACMap` key implements +both traits, while a mutable intermediate such as `Phased<_, NoHash>` can still +implement `Word` without pretending to be a valid map key. This separation +replaces the current `REHASH = false` use case with an explicit non-key mode. + +This document initially focuses on: + +- packed Pauli words; +- lossy Pauli words; +- phased-word composition; and +- lazy structural hash caches. + +Fermion-word storage will receive a separate design when fermionic propagation +is implemented. It will use the same `Word` interface with a different `Site`: +the word index records product order, while the fermionic site value records +the physical mode and creation or annihilation action. + +`weight()` counts non-identity factors according to the selected site type. +For a representation that stores only non-identity factors, it may equal +`n_sites()`. `set()` is the shared structural mutation boundary and must +preserve concrete invariants and invalidate the affected hash components. + +## Logical Pauli model + +An ordinary Pauli word is a fixed-width tensor product over `I`, `X`, `Y`, and +`Z`. Each site is represented by two logical bits: + +| Pauli | X bit | Z bit | +| --- | --- | --- | +| `I` | 0 | 0 | +| `X` | 1 | 0 | +| `Z` | 0 | 1 | +| `Y` | 1 | 1 | + +A lossy Pauli word adds a `Lost` state. Loss is exclusive with the four Pauli +states; a lost site has canonical bits `(x, z, lost) = (0, 0, 1)`. + +The exact Pauli alphabet and its lossy extension are site types rather than +word subtraits: + +```rust +pub enum Pauli { + I, + X, + Y, + Z, +} + +pub enum LossySite { + Present(S), + Lost, +} +``` + +An ordinary word implements `Word`. `WithLoss` implements +`Word>` and returns `Lost` for marked sites. This keeps +the word interface independent of the chosen operator alphabet. + +## `PauliWord` packed representation + +The initial packed implementation stores parallel fixed-size X and Z arrays: + +```rust +pub struct PauliWord { + xbits: BitArray, + zbits: BitArray, + nqubits: usize, + hash_cache: M, + _phantom: PhantomData, +} +``` + +`A` is an implementation parameter such as `[u8; N]` or `[usize; N]`. It is +not exposed through `Word`. The implementation validates that `nqubits` fits +the arrays and ignores or canonicalizes unused high bits. + +The structural identity is: + +```text +(nqubits, logical X bits, logical Z bits) +``` + +Equality and hashing exclude unused capacity, cache contents, cache validity, +and the `PhantomData` marker. + +All fields are private. Mutations go through methods that preserve unused-bit +invariants and invalidate the hash when a logical Pauli site changes. + +## Lossy Pauli word + +Loss is best modeled as an orthogonal wrapper around a Pauli word: + +```rust +pub struct WithLoss +where + W: Word, +{ + word: W, + lbits: BitArray, + loss_hash: M, +} +``` + +`A` is the packed bit-array backing selected by the concrete type. The public +alias retains the current `LossyPauliWord` name: + +```rust +pub type LossyPauliWord = WithLoss, A>; +``` + +`WithLoss` is a justified new implementation name because the proposal changes +the current standalone lossy representation into a generic wrapper. The +established domain-facing alias does not need to change. + +Neither `A` nor the underlying word's array type appears in `Word`. + +### Canonical loss invariant + +A lost site must contain identity in the underlying Pauli word: + +```text +lost[q] = 1 => word[q] = I +``` + +`set_lost(q)` first sets the underlying word to `I`, then sets the loss bit. +`set(q, LossySite::Present(p))` clears the loss bit and then writes `p`. This +prevents multiple physical encodings from representing the same logical lossy +word. + +The structural identity is: + +```text +(underlying Pauli word, logical loss bits) +``` + +`weight()` counts `X`, `Y`, `Z`, and `Lost`; `loss_weight()` counts only lost +sites. + +### Loss-specific behavior + +Generic lossy Pauli propagation sees `LossySite::Lost` through `Word` and +preserves or skips the site according to the operation's semantics. Operations +that create, clear, or count loss use inherent `WithLoss` methods: + +```rust +impl WithLoss +where + W: Word, +{ + pub fn is_lost(&self, qubit: usize) -> bool; + pub fn set_lost(&mut self, qubit: usize); + pub fn clear_loss(&mut self, qubit: usize); + pub fn loss_weight(&self) -> usize; +} +``` + +Loss channels and maximum-loss-weight truncation specialize directly on the +generic wrapper family: + +```rust +impl LossChannel + for OperatorSum, S, P> +where + W: Word, +{ + // ... +} +``` + +There are no traits named `PauliWord`, `LossyPauliWord`, or `FermionWord`. +`PauliWord` and `LossyPauliWord` remain concrete domain type names. Algorithms +select the algebra through `Word::Site`; loss-only operations remain inherent +to `WithLoss`. + +## Phased words + +Phase is another orthogonal wrapper: + +```rust +pub struct Phased +where + W: Word, + M: HashMode, +{ + word: W, + phase: Phase, + hash_state: M::State, +} +``` + +`Phased` is the generalized wrapper implementation. Pauli-facing code retains +the established name through an alias: + +```rust +pub type PhasedPauliWord = Phased; +``` + +For Pauli use, `Phase` represents `+1`, `+i`, `-1`, and `-i`. The wrapper may +also be useful for other word algebras, but algebra-specific multiplication is +implemented only under the appropriate specialized word bound. + +Loss and phase compose without a new combined representation: + +```rust +PhasedPauliWord, A>, Mode> +``` + +The wrapper fields are private so phase mutation cannot bypass component-cache +invalidation. + +## Hash ownership + +Every hash-enabled word implements `Indexable` and selects: + +- a build hasher; +- an `Indexable::HashCache` associated type; +- concrete cache fields; and +- the algorithm that hashes its private structural data. + +The associated cache type does not require every implementation to use the +same representation. Examples include: + +- a plain unsigned integer with a separate validity flag for eager or explicit + recomputation; +- `Cell` plus `Cell` for lazy single-threaded hashing; +- atomics for a `Sync` key; and +- `()` for a mode that deliberately provides no cache. + +Because `Hash::hash` receives `&self`, a cache populated lazily from that method +requires interior mutability. Such a representation will generally prevent +the containing word from being `Copy`. + +## Component hashes + +Hash composition follows the logical wrappers: + +```text +packed Pauli hash = hash(nqubits, X bits, Z bits) +lossy hash = combine(Pauli hash, loss hash) +phased hash = combine(inner-word hash, phase hash) +phased lossy hash = combine(Pauli hash, loss hash, phase hash) +``` + +`combine` must be ordered and domain-separated. It must not be an +unqualified XOR of arbitrary component digests. + +The phase has only four values, so `CompositeHash` can normally compute its +contribution from a small table or mixer without another cache. `CachedHash` +is justified only if profiling identifies a caller that benefits from caching +the combined phased value. + +Loss masks may be large, so `WithLoss` can cache the loss component separately +from the Pauli component. A loss-only mutation then avoids rehashing X/Z. + +## Invalidation rules + +| Mutation | Pauli component | Loss component | Phase component | +| --- | --- | --- | --- | +| Change ordinary Pauli site | invalidate | preserve | preserve | +| Mark identity site lost | preserve | invalidate | preserve | +| Mark nonidentity site lost | invalidate | invalidate | preserve | +| Clear loss to identity | preserve | invalidate | preserve | +| Replace loss with Pauli | invalidate if nonidentity | invalidate | preserve | +| Change phase | preserve | preserve | recompute or invalidate | + +Constructors compute caches eagerly or mark them invalid according to the +selected cache mode. Cloning copies a valid cache because the clone initially +has identical structural contents. + +## Hash modes + +Wrappers that are sometimes used only as mutable intermediate values should be +generic over hash behavior: + +```rust +pub trait HashMode { + type State: Clone; +} + +pub struct NoHash; +pub struct CompositeHash; +pub struct CachedHash(PhantomData); +``` + +- `NoHash` stores no wrapper cache and does not implement `Hash` or + `Indexable` for the wrapper. +- `CompositeHash` implements `Hash` by combining cached inner components on + demand, with no combined cache. +- `CachedHash` stores a lazily or eagerly maintained combined cache. + +The concrete policy protocol should be finalized while implementing the first +two modes. It should not expose cache state to propagation algorithms. + +## Ordering and serialization + +`Eq`, `Ord`, `Hash`, display, parsing, and serialization must agree on logical +identity: + +- Pauli sites compare in a documented order. +- Loss participates after the underlying Pauli content or through an explicit + `LossySite` ordering. +- Phase participates only when the phased wrapper's identity includes it. +- Unused bits and cache state never participate. + +Serialization uses logical symbols and lengths, not raw native-word memory, so +it remains stable across storage widths and platforms. + +## Prototype validation + +The prototype should include: + +- round-trip parsing tests for ordinary and lossy symbols; +- property tests comparing packed operations with a simple symbol vector; +- tests enforcing `lost => underlying I` after every mutator; +- equality/hash agreement tests for all wrappers and modes; +- tests showing loss-only changes preserve the Pauli hash component; +- tests showing phase-only changes preserve Pauli and loss components; +- tests proving unused high bits do not affect identity; +- `Send`/`Sync` assertions for cache modes intended for concurrent maps; and +- benchmarks comparing eager, lazy, composite, and uncached hashing. + +## Open questions + +1. Should `WithLoss` permit a loss-mask storage width different from its inner + word's packed width? +2. Does any real phased-word caller benefit from a combined `CachedHash` mode? +3. Does `Indexable::HashCache` have a generic consumer, or should the cache + type also remain entirely private like word storage? From f2a5e7f2c94052c487d601eae3bf3e9f5a699924 Mon Sep 17 00:00:00 2001 From: David Plankensteiner Date: Thu, 23 Jul 2026 12:35:27 +0200 Subject: [PATCH 02/10] docs(design): reconcile traits 2 decisions --- docs/design/tableau-data-structure.md | 130 +++++++++-- .../traits-2-configuration-and-hashing.md | 127 ++++++----- docs/design/word-data-structures.md | 201 +++++++----------- 3 files changed, 254 insertions(+), 204 deletions(-) diff --git a/docs/design/tableau-data-structure.md b/docs/design/tableau-data-structure.md index 7b283359f..7d5379dcc 100644 --- a/docs/design/tableau-data-structure.md +++ b/docs/design/tableau-data-structure.md @@ -24,7 +24,7 @@ orientation. - Store the X/Z tableau bits in contiguous, aligned, bit-packed memory. - Make one- and two-qubit column operations efficient. - Permit temporary transposition for row-oriented elimination and collapse. -- Keep phases and optional loss state independently addressable and hashable. +- Keep phases and per-qubit loss state independently addressable and hashable. - Hash the logical tableau independently of its current physical orientation. - Avoid parameterizing the trait system by a row type or Pauli-word storage. - Leave room for SIMD-width and padding changes without changing public @@ -61,8 +61,7 @@ The logical state consists of: - an X matrix of shape `(2n, n)`; - a Z matrix of shape `(2n, n)`; - a phase plane of length `2n`; and -- for lossy variants, a loss plane of length `n` or another explicitly defined - logical shape. +- a per-qubit loss plane of length `n`. This model does not imply a physical row object. @@ -77,7 +76,7 @@ pub struct TableauData { x_offset: usize, z_offset: usize, phase_offset: usize, - loss_offset: Option, + loss_offset: usize, major_stride: usize, n_qubits: usize, orientation: Orientation, @@ -98,6 +97,47 @@ Padding must either be kept zero or excluded from equality and hashing. Zeroed padding is preferable because it permits bulk comparison and hashing of canonical ranges. +## Loss ownership + +The existing concrete `Tableau` always owns the per-qubit loss plane. This is a +capability of the same tableau type, not a `LossyTableau` variant or a +`Tableau` parameter: + +```rust +pub struct Tableau { + data: TableauData, + lost_count: usize, + // hash caches and RNG +} + +pub struct GeneralizedTableau { + tableau: Tableau, + coefficients: S, + // threshold and measurement record +} +``` + +`GeneralizedTableau` therefore has no separate `is_lost: Vec`. Its gate, +noise, and measurement algorithms query and mutate the loss plane owned by the +inner tableau. + +`lost_count` is derived metadata used to preserve a fast +`lost_count == 0` path. It is excluded from equality and hashing; debug builds +should verify that it equals the population count of the logical loss plane. +When no qubit is lost, gate kernels enter their existing lossless bulk path. +When loss is present, a one-qubit gate skips a lost target and a two-qubit gate +skips the operation if either target is lost, matching the current generalized +tableau semantics. Batch kernels should mask or skip lost targets without +allocating filtered target vectors. + +This ownership enables a pure Clifford-plus-loss simulation with the same +`Tableau`. Loss events use the pure Clifford collapse procedure before marking +the affected qubit lost. Generalized loss events still use the +coefficient-aware generalized measurement procedure before marking the loss; +moving the bit does not move that algorithm. The pure path covers loss models +whose conditional trajectory remains a stabilizer state; faithful +non-Clifford survival back-action remains generalized. + ## Column-major orientation The default orientation should make the generator dimension contiguous for a @@ -183,11 +223,12 @@ impl Tableau { pub fn h(&mut self, qubit: usize); pub fn cnot(&mut self, control: usize, target: usize); - pub fn measure_z(&mut self, qubit: usize) -> bool; + pub fn measure_z(&mut self, qubit: usize) -> Option; pub fn x_bit(&self, generator: usize, qubit: usize) -> bool; pub fn z_bit(&self, generator: usize, qubit: usize) -> bool; pub fn phase(&self, generator: usize) -> Phase; + pub fn is_lost(&self, qubit: usize) -> bool; } ``` @@ -200,6 +241,44 @@ There should be no `stabilizers_mut() -> &mut [Row]` or equivalent escape hatch that bypasses hash invalidation and orientation invariants. Specialized internal row operations operate through `TableauData` or a transposition guard. +## Measurement algorithms + +The Rust behavioral boundary is one loss-aware trait: + +```rust +pub trait Measure { + fn measure(&mut self, qubit: usize) -> Option; + + fn measure_many(&mut self, targets: &[usize]) -> Vec> { + targets.iter().map(|&q| self.measure(q)).collect() + } +} +``` + +`Some(false)` and `Some(true)` represent computational-basis outcomes; `None` +represents a lost qubit. This is the existing core representation used by +`GeneralizedTableau`. The Python binding may continue mapping it to +`MeasurementResult::{ZERO, ONE, LOST}`. The old public split between +`Measure -> bool` and `LossyMeasure -> Option` is removed. The bare +boolean Clifford measurement routine becomes a private helper called only +after the public implementation has established that the target is present. + +The common trait and result type do not imply a common measurement algorithm: + +- `Tableau::measure` checks the loss plane and then uses the pure Clifford + stabilizer measurement procedure. It does not decompose against a sparse + generalized-state coefficient basis. +- `GeneralizedTableau::measure` checks the inner tableau's loss plane, then + decomposes the measured Pauli into stabilizers and destabilizers and updates + the sparse coefficients. This path is fundamentally \(O(n^2)\). + +The physical tableau must make both stabilizer and destabilizer generators +available to the generalized decomposition, but that requirement must not be +promoted into the `Measure` trait or force the generalized algorithm onto the +pure Clifford implementation. Pure and generalized measurement performance +must be benchmarked separately; Stim's inverse-tableau measurement +optimizations are not complexity promises for the generalized algorithm. + ## Gate access patterns The implementation should categorize mutations by physical access and hash @@ -227,30 +306,31 @@ hasher and cache representations; neither is inherited from `PauliWord`. The structural hash is composed from independent logical components: ```text -tableau hash = combine(xz hash, phase hash[, loss hash]) +tableau hash = combine(xz hash, phase hash, loss hash) ``` The X and Z planes share an `xz_hash` cache because most Clifford mutations update them together. The phase plane has a separate cache so Pauli -conjugations and sign changes do not force a matrix rehash. Loss should use a -third cache if it is independently mutable and part of key identity. +conjugations and sign changes do not force a matrix rehash. The independently +mutable loss plane uses a third cache. ```rust pub struct Tableau { data: TableauData, - xz_hash: XzHashCache, - phase_hash: PhaseHashCache, - loss_hash: Option, + lost_count: usize, + xz_hash: OnceLock, + phase_hash: OnceLock, + loss_hash: OnceLock, rng: SmallRng, } ``` -The cache types above are concrete choices made by the tableau author. The -shared `Indexable::HashCache` associated type may name a small aggregate or -another representation chosen by the implementation. +The cache fields are private representation choices made by the tableau +author. `Indexable` exposes only the associated build hasher; it does not name +cache types or expose invalidation. Equality and hashing include logical qubit count, generator order, all logical -X/Z bits, phases, and loss state when present. They exclude: +X/Z bits, phases, and loss state. They exclude: - RNG state; - allocation capacity; @@ -266,14 +346,13 @@ The component invalidation rules are: | Direct phase change | preserve | invalidate | preserve | | `H`, `S`, `CNOT`, `CZ` | invalidate | invalidate if changed | preserve | | Row multiplication | invalidate | invalidate if changed | preserve | -| Mark or clear loss | preserve | preserve | invalidate | +| Toggle a loss bit | preserve | preserve | invalidate | | Physical transpose | preserve | preserve | preserve | | RNG update | preserve | preserve | preserve | The current `ppvm-tableau-sum` split between `word_fingerprint` and `phase_loss_hash` is evidence that component hashing matters. The new tableau -owns these components directly and separates loss as well when the logical -model permits it. +owns X/Z, phase, and loss components directly. ## Cloning and mixture use @@ -307,12 +386,18 @@ not belong in the PPVM trait system. The prototype should include: - property tests comparing gates and measurements with the existing tableau; +- differential tests for the pure Clifford and generalized measurement + algorithms, including lost targets; +- tests that gates skip lost targets and retain the `lost_count == 0` fast + path; - round-trip tests for column-major -> row-major -> column-major transpose; - equality and hash tests across physical orientations; - tests proving phase-only changes preserve the X/Z hash cache; - tests proving padding never affects equality or hashing; - benchmarks for one- and two-qubit Clifford gates; -- benchmarks for deterministic and random measurement paths; +- separate benchmarks for pure Clifford and generalized deterministic and + random measurement paths; +- benchmarks for lossless gates, sparse loss, and Clifford-plus-loss sampling; - benchmarks for clone-and-mutate mixture branching; and - benchmarks comparing permanent row-major, permanent column-major, and temporary-transpose variants on representative circuits. @@ -321,9 +406,8 @@ The prototype should include: 1. What block width and alignment should the first implementation use? 2. Should phases occupy one or two bits per generator in the tableau model? -3. Should loss be stored per qubit, per generator, or outside the core tableau? -4. Which operations should receive a transposition guard versus performing +3. Which operations should receive a transposition guard versus performing column-strided work directly? -5. Does a dual-orientation representation outperform temporary transposition +4. Does a dual-orientation representation outperform temporary transposition for PPVM's measurement-heavy workloads? -6. Should PPVM ultimately store a forward or inverse tableau? +5. Should PPVM ultimately store a forward or inverse tableau? diff --git a/docs/design/traits-2-configuration-and-hashing.md b/docs/design/traits-2-configuration-and-hashing.md index 5d242a2ae..2709ea042 100644 --- a/docs/design/traits-2-configuration-and-hashing.md +++ b/docs/design/traits-2-configuration-and-hashing.md @@ -98,20 +98,23 @@ pub struct FermionSite { } ``` -Thus an ordinary packed Pauli word implements `Word`, a lossy -wrapper implements `Word>`, and a future ordered -fermionic product can implement `Word`. A fermionic word's -index denotes factor order; `FermionSite` carries the physical mode. For a -dense Pauli word, the index is the qubit and `n_sites()` is its width. +Thus an ordinary packed Pauli word implements `Word`, a +concrete packed lossy word implements `Word>`, and a +future ordered fermionic product can implement `Word`. A +fermionic word's index denotes factor order; `FermionSite` carries the physical +mode. For a dense Pauli word, the index is the qubit and `n_sites()` is its +width. `weight()` is the number of non-identity factors according to the concrete site alphabet; an ordered representation that stores no explicit identities may therefore have `weight() == n_sites()`. Implementations of `set()` preserve their representation invariants and invalidate the affected hash components. -The concrete `WithLoss` family provides loss mutation and -`loss_weight()` as inherent methods. Loss channels and loss-specific -truncation specialize directly on `WithLoss`; there is no one-implementation -`LossyPauliWord` capability trait. +The concrete `LossyPauliWord` stores packed X, Z, and loss planes directly and +provides loss mutation and `loss_weight()` as inherent methods. Loss channels +and loss-specific truncation specialize directly on that concrete type; there +is no one-implementation `LossyPauliWord` capability trait. A generic loss +wrapper should be reconsidered only after a second real word representation +needs the same composition. Concrete packed Pauli, lossy, phased, and hash-cache layouts are described in [`word-data-structures.md`](word-data-structures.md). None of those layouts is @@ -144,12 +147,31 @@ pub trait RotationOne { pub trait PauliError { fn pauli_error(&mut self, qubit: usize, probabilities: [C; 3]); } + +pub trait Measure { + fn measure(&mut self, qubit: usize) -> Option; + + fn measure_many(&mut self, targets: &[usize]) -> Vec> { + targets.iter().map(|&q| self.measure(q)).collect() + } +} ``` The same concrete tableau may implement a numeric trait for every supported coefficient type without storing that coefficient type. Measurement and reset traits likewise expose behavior and result types without exposing tableau -rows, packed matrix blocks, or matrix orientation. +rows, packed matrix blocks, or matrix orientation. `Measure` is loss-aware for +both `Tableau` and `GeneralizedTableau`: `Some(false)` and `Some(true)` denote +computational-basis outcomes and `None` denotes a lost qubit. The former +`Measure -> bool` and `LossyMeasure -> Option` split is removed. +Python may continue translating this Rust representation to its +`MeasurementResult` enum. + +Sharing the result type does not share the measurement algorithm. `Tableau` +uses the pure Clifford measurement procedure. `GeneralizedTableau` performs +its coefficient-aware stabilizer/destabilizer decomposition and update, which +is always \(O(n^2)\). The shared behavioral trait must not force the latter +algorithm or its scratch state into the concrete tableau. There is no shared `TableauStorage` trait in the first design. If multiple tableau implementations are later useful, each concrete type can implement @@ -259,6 +281,20 @@ implementation detail for the prototype. The important boundary is that the trait preserves the current semantic operation names without exposing physical auxiliary maps or scratch buffers. +`SumStorage` owns the semantic whole-map operations and its reusable workspace. +It delegates to the lower-level `ACMap` batch kernels without restoring the +removed map-to-map `ACMapInsert::map_insert` method: + +```text +SumStorage::map_insert -> ACMapInsert::map_insert_vec +SumStorage::map_insert_multiple -> ACMapInsert::map_insert_multiple +SumStorage::map_add -> ACMapAddAssign::map_add_assign +``` + +This boundary is compatible with +`refactor/shrink-internal-trait-surface`: the higher-level sparse-sum operation +remains, while the dead low-level primitive stays removed. + The generalized engine may be named `OperatorSum`, but the Pauli specialization retains the existing domain-facing `PauliSum` name. This is a new internal generalization, not a requirement to rename Pauli call sites. @@ -300,9 +336,9 @@ changed according to whether their underlying responsibility changes: | `PauliWordTrait` | `Word` plus `Indexable` where used as a key | Word operations are generalized through `Word::Site`; hashing becomes a separate capability. | | `n_qubits`, `get`, `set`, `weight` | `n_sites`, `get`, `set`, `weight` | Only the Pauli-specific extent name changes; the other operation names stay. | | concrete `PauliWord` | `PauliWord` | The packed X/Z word is the same domain concept. | -| concrete `LossyPauliWord` | `LossyPauliWord` alias over `WithLoss` | The public concept stays; only the implementation becomes a generic wrapper. | -| `PhasedPauliWord` | `PhasedPauliWord` alias over `Phased` | The public Pauli name stays; the implementation becomes a word-generic wrapper. | -| `rehash` | `invalidate_hash` for lazy modes | Recalculation changes from eager mutation-time work to lazy demand-time work. | +| concrete `LossyPauliWord` | `LossyPauliWord` | The packed X/Z/loss representation remains concrete and flattened. | +| `PhasedPauliWord` | `PhasedPauliWord` alias over non-indexable `Phased` | The wrapper is generic over ordinary and lossy words but is not a production map key. | +| `rehash` | private cache invalidation | Recalculation changes from eager mutation-time work to lazy demand-time work without exposing cache mechanics through `Indexable`. | | `Strategy` | `Policy` | Intentional terminology change requested for this redesign. | | `ACMap` | `ACMap` | The associative coefficient map has the same role. | | `PauliSum::data`, `map_insert`, `map_add` | same method names on `SumStorage` | These semantic operations already match the proposed boundary. | @@ -310,11 +346,12 @@ changed according to whether their underlying responsibility changes: | `PauliSum` | `PauliSum` over generalized `OperatorSum` machinery | Pauli-facing code keeps its established name; `OperatorSum` names the new cross-algebra engine. | | `GeneralizedTableauSum` | `GeneralizedTableauSum` | The classical mixture algorithm remains the same concept. | | `EntryStore`, `VecStorage`, `MapStorage` | unchanged | The proposal uses the existing storage boundary and implementations. | -| `BuildHasher`, `HashFinalize` | unchanged | The hasher concepts remain, but ownership moves from `Config` to each indexable key type. | +| `BuildHasher` | associated with `Indexable` | Hasher ownership moves from `Config` to each indexable key type. | +| `HashFinalize` | removed from the shared contract | Concrete keys may finalize or compose hashes privately. | | `PauliStorage` | removed | Packed backing storage becomes private to the concrete word representation. | -Names such as `Word`, `Indexable`, `SumStorage`, `WithLoss`, and -`OperatorSum` are therefore new because they denote abstractions that do not +Names such as `Word`, `Indexable`, `SumStorage`, and `OperatorSum` are +therefore new because they denote abstractions that do not exist in the current implementation, not because the existing API is being renamed wholesale. @@ -339,7 +376,7 @@ This rule keeps: It rejects the removed global `Config`, `PauliSumAlgorithm`, `TableauMixtureAlgorithm`, and `TableauStorage` traits, as well as word subtraits named `PauliWord`, `FermionWord`, or `LossyPauliWord`. Their -distinctions are expressed by `Word::Site` or by concrete wrapper types instead +distinctions are expressed by `Word::Site` or by concrete types instead of one-alphabet subtraits; the concrete `PauliWord` and `LossyPauliWord` type names remain available. @@ -386,14 +423,11 @@ a benchmark decision, not a type-system requirement. Hash-enabled `Word` values and `Tableau` can both be expensive, mostly-stable map keys. Their hashing contract should be expressed independently of any map. -The tentative common capability is: +The common capability is intentionally minimal: ```rust pub trait Indexable: Clone + Eq + Hash { - type BuildHasher: std::hash::BuildHasher + Clone + Default + HashFinalize; - type HashCache; - - fn invalidate_hash(&mut self); + type BuildHasher: std::hash::BuildHasher + Clone + Default; } ``` @@ -403,34 +437,24 @@ The important points are: `Config` to the key type; - the build hasher is associated with the key type, not with a configuration bundle; -- the hash cache is an associated type, not a framework-provided concrete - `HashCache` structure; -- the author of the concrete data type owns the cache layout and invalidation - implementation; and +- cache layout and invalidation are private representation invariants of the + concrete type; and - equality and hashing cover only structural key identity, never cache fields or incidental runtime state. -For example, a word implementation may choose `u64`, `Option`, an atomic -representation, or another type. `ppvm-traits-2` does not prescribe the -fields that store it. +No generic consumer needs to name a cache type or request invalidation. +Structural mutation already occurs through `&mut self`, so each mutator can +clear the affected private cache as part of maintaining its concrete +invariants. ### Lazy hashing and interior mutability -Rust's `Hash::hash` receives `&self`. A cache that is first populated from -inside `Hash::hash` therefore requires interior mutability. Possible concrete -representations include: - -- `Cell` plus `Cell` for single-threaded keys; -- atomics for keys that must remain `Sync`; or -- a plain `u64` plus `bool` when cache preparation is eager or explicitly - performed through `&mut self` before hashing. - -This is deliberately a representation-level decision. A cache using interior -mutability will generally prevent the containing value from being `Copy`. - -Multiple concurrent readers may compute the same invalid hash more than once; -that is acceptable as long as they publish the same structural hash safely. -Structural mutation still requires `&mut self`. +Rust's `Hash::hash` receives `&self`, so shipped indexable words and tableaus +use private component `OnceLock` caches. `Hash::hash` may populate a cache +through shared access, while structural mutators clear affected cells through +their exclusive `&mut self` access. This preserves `Send + Sync` for the +shipped representations. `Indexable` itself does not require either +concurrency bound. ### Key mutation invariant @@ -444,8 +468,8 @@ through mutation guards that invalidate on completion. ## Concrete word hashing -Every concrete `Word` owns its `BuildHasher`, cache representation, structural -hash algorithm, and invalidation logic through `Indexable`. Pauli words hash +Every concrete `Word` owns its `BuildHasher`, private cache representation, +structural hash algorithm, and invalidation logic. Pauli words hash their X/Z content, lossy words compose Pauli and loss components, and future fermion words hash their ordered factors. Factor order is part of fermionic identity. @@ -463,7 +487,7 @@ hasher and cache representation. This does not imply that a tableau is a `Word`; they only share the `Indexable` key capability. The tableau's structural hash is composed from its logical X/Z matrix, phase -plane, and optional loss plane. It excludes RNG, padding, cache state, and +plane, and per-qubit loss plane. It excludes RNG, padding, cache state, and physical matrix orientation. Separate component caches allow phase-only changes to avoid rehashing the X/Z matrix. Physical transposition is a layout change, not a logical mutation, and does not invalidate the structural hash. @@ -493,16 +517,11 @@ Domain-specific aliases or wrappers can preserve `PauliSum` and introduce - Defining one collection interface shared by all algorithms. - Requiring every sparse-sum storage backend to physically contain both an auxiliary map and a scratch buffer. -- Selecting a single cache representation for every key type. +- Exposing cache representation or invalidation through `Indexable`. - Preserving `Copy` at the expense of correct lazy caching. - Adding runtime dispatch for storage, hashing, or algorithm policies. ## Open design questions -1. What is the minimal method surface of `Indexable` beyond the associated - `BuildHasher` and `HashCache` types? Cache mechanics should remain owned by - the concrete type, but algorithms may need a common invalidation operation. -2. Which cache representations must support `Send` and `Sync` in the first - prototype? -3. Do benchmarks justify retaining both the auxiliary-map and vector-staging +1. Do benchmarks justify retaining both the auxiliary-map and vector-staging fast paths in the default sparse-sum storage backend? diff --git a/docs/design/word-data-structures.md b/docs/design/word-data-structures.md index 5ece8e5c3..30d87f229 100644 --- a/docs/design/word-data-structures.md +++ b/docs/design/word-data-structures.md @@ -28,9 +28,9 @@ Generic propagation and collection code uses behavioral traits and never names the backing memory. `Word` does not extend `Indexable`. A word used as an `ACMap` key implements -both traits, while a mutable intermediate such as `Phased<_, NoHash>` can still -implement `Word` without pretending to be a valid map key. This separation -replaces the current `REHASH = false` use case with an explicit non-key mode. +both traits, while a mutable intermediate such as `Phased` can implement +`Word` without pretending to be a valid map key. This separation replaces the +current `REHASH = false` use case with an explicitly non-indexable type. This document initially focuses on: @@ -81,7 +81,7 @@ pub enum LossySite { } ``` -An ordinary word implements `Word`. `WithLoss` implements +An ordinary word implements `Word`. `LossyPauliWord` implements `Word>` and returns `Lost` for marked sites. This keeps the word interface independent of the chosen operator alphabet. @@ -90,18 +90,19 @@ the word interface independent of the chosen operator alphabet. The initial packed implementation stores parallel fixed-size X and Z arrays: ```rust -pub struct PauliWord { +pub struct PauliWord { xbits: BitArray, zbits: BitArray, nqubits: usize, - hash_cache: M, - _phantom: PhantomData, + hash_cache: OnceLock, + _hasher: PhantomData H>, } ``` -`A` is an implementation parameter such as `[u8; N]` or `[usize; N]`. It is -not exposed through `Word`. The implementation validates that `nqubits` fits -the arrays and ignores or canonicalizes unused high bits. +`A` is an implementation parameter such as `[u8; N]` or `[usize; N]`, and `H` +is the build hasher associated through `Indexable`. Neither is exposed through +`Word`. The implementation validates that `nqubits` fits the arrays and +ignores or canonicalizes unused high bits. The structural identity is: @@ -117,41 +118,36 @@ invariants and invalidate the hash when a logical Pauli site changes. ## Lossy Pauli word -Loss is best modeled as an orthogonal wrapper around a Pauli word: +The first prototype keeps the established lossy Pauli word as a flattened +packed representation: ```rust -pub struct WithLoss -where - W: Word, -{ - word: W, +pub struct LossyPauliWord { + xbits: BitArray, + zbits: BitArray, lbits: BitArray, - loss_hash: M, + nqubits: usize, + xz_hash_cache: OnceLock, + loss_hash_cache: OnceLock, + _hasher: PhantomData H>, } ``` -`A` is the packed bit-array backing selected by the concrete type. The public -alias retains the current `LossyPauliWord` name: - -```rust -pub type LossyPauliWord = WithLoss, A>; -``` - -`WithLoss` is a justified new implementation name because the proposal changes -the current standalone lossy representation into a generic wrapper. The -established domain-facing alias does not need to change. - -Neither `A` nor the underlying word's array type appears in `Word`. +Inlining all three planes avoids wrapper nesting and keeps the lossy hot path +and component hashes direct. A generic loss wrapper is not introduced until a +second real word representation demonstrates that it needs the same +composition. `A` and `H` remain private implementation parameters from the +perspective of `Word`. ### Canonical loss invariant -A lost site must contain identity in the underlying Pauli word: +A lost site must contain identity in its X/Z planes: ```text -lost[q] = 1 => word[q] = I +lost[q] = 1 => xbits[q] = 0 and zbits[q] = 0 ``` -`set_lost(q)` first sets the underlying word to `I`, then sets the loss bit. +`set_lost(q)` first clears the X/Z bits, then sets the loss bit. `set(q, LossySite::Present(p))` clears the loss bit and then writes `p`. This prevents multiple physical encodings from representing the same logical lossy word. @@ -159,7 +155,7 @@ word. The structural identity is: ```text -(underlying Pauli word, logical loss bits) +(nqubits, logical X bits, logical Z bits, logical loss bits) ``` `weight()` counts `X`, `Y`, `Z`, and `Lost`; `loss_weight()` counts only lost @@ -169,13 +165,10 @@ sites. Generic lossy Pauli propagation sees `LossySite::Lost` through `Word` and preserves or skips the site according to the operation's semantics. Operations -that create, clear, or count loss use inherent `WithLoss` methods: +that create, clear, or count loss use inherent `LossyPauliWord` methods: ```rust -impl WithLoss -where - W: Word, -{ +impl LossyPauliWord { pub fn is_lost(&self, qubit: usize) -> bool; pub fn set_lost(&mut self, qubit: usize); pub fn clear_loss(&mut self, qubit: usize); @@ -184,13 +177,11 @@ where ``` Loss channels and maximum-loss-weight truncation specialize directly on the -generic wrapper family: +concrete lossy word: ```rust -impl LossChannel - for OperatorSum, S, P> -where - W: Word, +impl LossChannel + for OperatorSum, S, P> { // ... } @@ -199,21 +190,19 @@ where There are no traits named `PauliWord`, `LossyPauliWord`, or `FermionWord`. `PauliWord` and `LossyPauliWord` remain concrete domain type names. Algorithms select the algebra through `Word::Site`; loss-only operations remain inherent -to `WithLoss`. +to `LossyPauliWord`. ## Phased words Phase is another orthogonal wrapper: ```rust -pub struct Phased +pub struct Phased where W: Word, - M: HashMode, { word: W, phase: Phase, - hash_state: M::State, } ``` @@ -221,43 +210,34 @@ where the established name through an alias: ```rust -pub type PhasedPauliWord = Phased; +pub type PhasedPauliWord = Phased; ``` For Pauli use, `Phase` represents `+1`, `+i`, `-1`, and `-i`. The wrapper may -also be useful for other word algebras, but algebra-specific multiplication is -implemented only under the appropriate specialized word bound. +wrap both ordinary and lossy Pauli words. It may also be useful for other word +algebras, but algebra-specific multiplication is implemented only under the +appropriate specialized word bound. Loss and phase compose without a new combined representation: ```rust -PhasedPauliWord, A>, Mode> +PhasedPauliWord> ``` -The wrapper fields are private so phase mutation cannot bypass component-cache -invalidation. +No phased word is a production map key in the first prototype, so `Phased` +does not implement `Hash` or `Indexable` and stores no hash mode or cache. ## Hash ownership -Every hash-enabled word implements `Indexable` and selects: +Every hash-enabled word implements `Indexable`, associates its build hasher, +and privately owns the fields and algorithm used to cache its structural +hash. Cache representation and invalidation are not exposed through +`Indexable`. -- a build hasher; -- an `Indexable::HashCache` associated type; -- concrete cache fields; and -- the algorithm that hashes its private structural data. - -The associated cache type does not require every implementation to use the -same representation. Examples include: - -- a plain unsigned integer with a separate validity flag for eager or explicit - recomputation; -- `Cell` plus `Cell` for lazy single-threaded hashing; -- atomics for a `Sync` key; and -- `()` for a mode that deliberately provides no cache. - -Because `Hash::hash` receives `&self`, a cache populated lazily from that method -requires interior mutability. Such a representation will generally prevent -the containing word from being `Copy`. +The shipped indexable words use component `OnceLock` caches. `Hash::hash` +can populate them through `&self`; structural mutators clear affected cells +through `&mut self`. This preserves `Send + Sync` without imposing either +bound on the `Indexable` trait. ## Component hashes @@ -266,59 +246,27 @@ Hash composition follows the logical wrappers: ```text packed Pauli hash = hash(nqubits, X bits, Z bits) lossy hash = combine(Pauli hash, loss hash) -phased hash = combine(inner-word hash, phase hash) -phased lossy hash = combine(Pauli hash, loss hash, phase hash) ``` `combine` must be ordered and domain-separated. It must not be an unqualified XOR of arbitrary component digests. -The phase has only four values, so `CompositeHash` can normally compute its -contribution from a small table or mixer without another cache. `CachedHash` -is justified only if profiling identifies a caller that benefits from caching -the combined phased value. - -Loss masks may be large, so `WithLoss` can cache the loss component separately -from the Pauli component. A loss-only mutation then avoids rehashing X/Z. +Loss masks may be large, so `LossyPauliWord` caches the loss component +separately from the X/Z component. A loss-only mutation then avoids rehashing +X/Z. `Phased` is absent from this composition because it is not indexable. ## Invalidation rules -| Mutation | Pauli component | Loss component | Phase component | -| --- | --- | --- | --- | -| Change ordinary Pauli site | invalidate | preserve | preserve | -| Mark identity site lost | preserve | invalidate | preserve | -| Mark nonidentity site lost | invalidate | invalidate | preserve | -| Clear loss to identity | preserve | invalidate | preserve | -| Replace loss with Pauli | invalidate if nonidentity | invalidate | preserve | -| Change phase | preserve | preserve | recompute or invalidate | - -Constructors compute caches eagerly or mark them invalid according to the -selected cache mode. Cloning copies a valid cache because the clone initially -has identical structural contents. - -## Hash modes - -Wrappers that are sometimes used only as mutable intermediate values should be -generic over hash behavior: - -```rust -pub trait HashMode { - type State: Clone; -} - -pub struct NoHash; -pub struct CompositeHash; -pub struct CachedHash(PhantomData); -``` - -- `NoHash` stores no wrapper cache and does not implement `Hash` or - `Indexable` for the wrapper. -- `CompositeHash` implements `Hash` by combining cached inner components on - demand, with no combined cache. -- `CachedHash` stores a lazily or eagerly maintained combined cache. +| Mutation | X/Z component | Loss component | +| --- | --- | --- | +| Change ordinary Pauli site | invalidate | preserve | +| Mark identity site lost | preserve | invalidate | +| Mark nonidentity site lost | invalidate | invalidate | +| Clear loss to identity | preserve | invalidate | +| Replace loss with Pauli | invalidate if nonidentity | invalidate | -The concrete policy protocol should be finalized while implementing the first -two modes. It should not expose cache state to propagation algorithms. +Constructors leave caches empty. Cloning may copy a valid cached value because +the clone initially has identical structural contents. ## Ordering and serialization @@ -328,7 +276,8 @@ identity: - Pauli sites compare in a documented order. - Loss participates after the underlying Pauli content or through an explicit `LossySite` ordering. -- Phase participates only when the phased wrapper's identity includes it. +- Phase participates in equality and serialization for `Phased`, but not in + map-key hashing because the wrapper is not indexable. - Unused bits and cache state never participate. Serialization uses logical symbols and lengths, not raw native-word memory, so @@ -340,18 +289,16 @@ The prototype should include: - round-trip parsing tests for ordinary and lossy symbols; - property tests comparing packed operations with a simple symbol vector; -- tests enforcing `lost => underlying I` after every mutator; -- equality/hash agreement tests for all wrappers and modes; -- tests showing loss-only changes preserve the Pauli hash component; -- tests showing phase-only changes preserve Pauli and loss components; +- tests enforcing `lost => X/Z identity` after every mutator; +- equality/hash agreement tests for ordinary and lossy indexable words; +- tests showing loss-only changes preserve the X/Z hash component; +- equality and serialization tests for ordinary and lossy `Phased` values; - tests proving unused high bits do not affect identity; -- `Send`/`Sync` assertions for cache modes intended for concurrent maps; and -- benchmarks comparing eager, lazy, composite, and uncached hashing. +- `Send`/`Sync` assertions for shipped indexable words; and +- benchmarks comparing uncached structural hashing with the private lazy + component caches. ## Open questions -1. Should `WithLoss` permit a loss-mask storage width different from its inner - word's packed width? -2. Does any real phased-word caller benefit from a combined `CachedHash` mode? -3. Does `Indexable::HashCache` have a generic consumer, or should the cache - type also remain entirely private like word storage? +1. Should the loss plane use the same packed array width as the X/Z planes, or + a separately selected private width? From 7796e5a62e79ec25c494b6a13a9a8ea2c62168c9 Mon Sep 17 00:00:00 2001 From: Roger-luo Date: Thu, 30 Jul 2026 17:48:00 -0400 Subject: [PATCH 03/10] docs(design): add SoA batch and hash-join execution contract Extend the traits-2 proposal with a batch-first map contract that frames the branch-staging merge as a hash join with aggregation, specified for memory prefetching, SIMD, multi-threading, and GPU offload. - Reframe the merge as build/probe-with-group-by and pin its cost model to memory latency at out-of-cache sizes. - Cite Chen, Ailamaki, Gibbons & Mowry, "Improving Hash Join Performance through Prefetching" (ICDE 2004; TODS 2007), as the origin of group and software-pipelined prefetching and the cost model the contract serves. - Add a structure-of-arrays batch contract: KeyBatch/TermBatch split into key, precomputed-hash, and coefficient columns; ACMapBatch upsert/probe over columns; a KeyColumn capability behind a new Columnar trait that is kept separate from the minimal Indexable (no cache/column type leaks through the hashing contract). - Separate term production from insertion so gates emit a TermBatch rather than mutating through a per-element closure, enabling prefetch, SIMD, radix partitioning, and device offload. - Document the concrete packed-Pauli/LossyPauliWord plane layout, alignment/padding, hash_into agreement, and gather semantics in word-data-structures.md, per the docs' layout-lives-with-the- representation rule. - Update the compatibility table, non-goals, and open questions (plane granularity and per-target alignment remain open). Co-authored-by: Claude Opus 4.8 (1M context) --- .../traits-2-configuration-and-hashing.md | 243 ++++++++++++++++++ docs/design/word-data-structures.md | 60 +++++ 2 files changed, 303 insertions(+) diff --git a/docs/design/traits-2-configuration-and-hashing.md b/docs/design/traits-2-configuration-and-hashing.md index 2709ea042..aa0e6395d 100644 --- a/docs/design/traits-2-configuration-and-hashing.md +++ b/docs/design/traits-2-configuration-and-hashing.md @@ -295,6 +295,12 @@ This boundary is compatible with `refactor/shrink-internal-trait-surface`: the higher-level sparse-sum operation remains, while the dead low-level primitive stays removed. +Those `ACMap` kernels are the scalar spelling of a bulk operation. The +batch-first, structure-of-arrays layout the kernels consume on hot paths — and +that admits prefetched, SIMD, threaded, and offloaded backends — is specified in +[Batch execution and the hash-join contract](#batch-execution-and-the-hash-join-contract) +below. + The generalized engine may be named `OperatorSum`, but the Pauli specialization retains the existing domain-facing `PauliSum` name. This is a new internal generalization, not a requirement to rename Pauli call sites. @@ -349,6 +355,7 @@ changed according to whether their underlying responsibility changes: | `BuildHasher` | associated with `Indexable` | Hasher ownership moves from `Config` to each indexable key type. | | `HashFinalize` | removed from the shared contract | Concrete keys may finalize or compose hashes privately. | | `PauliStorage` | removed | Packed backing storage becomes private to the concrete word representation. | +| (new) | `Columnar`, `KeyColumn`, `KeyBatch`, `TermBatch`, `ACMapBatch` | Structure-of-arrays batch/hash-join contract, kept as a separate trait so `Indexable` stays minimal; see [Batch execution and the hash-join contract](#batch-execution-and-the-hash-join-contract). | Names such as `Word`, `Indexable`, `SumStorage`, and `OperatorSum` are therefore new because they denote abstractions that do not @@ -418,6 +425,218 @@ both `map_insert` and `map_add` using only an auxiliary map; another backend may use per-thread buffers. Whether retaining both mechanisms is worthwhile is a benchmark decision, not a type-system requirement. +## Batch execution and the hash-join contract + +The refactor's stated goal is a map contract friendly to CPU SIMD, memory +prefetching, multi-threading, and GPUs. None of those are properties of a +particular collection; they are properties of the *shape of work the collection +is asked to do*. This section fixes that shape — bulk, and hash-join-shaped — +and leaves every implementation choice open, matching the proposal's "contract +now, benchmark the backend later" stance. It keeps `Indexable` minimal: the +structure-of-arrays capability is a separate trait, not another associated type +on the key's hashing contract. + +### The merge is a hash join + +The branch-staging merge described above — probe each produced term against the +current sum, accumulate its coefficient on a match, insert it on a miss — is a +*hash join with aggregation* (equivalently a group-by aggregate). The current +operator map is the build side; the terms a gate produces are the probe side; +coefficient accumulation is the aggregate. It is the hottest operation in +propagation. + +Naming it explicitly matters because its performance is governed by the same +cost model as a database hash join: at the sizes of interest the build table +does not fit in cache, so each probe is a random access that misses to main +memory, and the probe phase is bound by memory latency rather than by +arithmetic. + +The `SumStorage` closures (`map_insert`, `map_add`) and the `ACMap` kernels they +delegate to still describe *what* the merge computes. But a closure applied to +one `(key, coeff)` at a time exposes no batch to prefetch, no homogeneous run to +vectorize, no partition to distribute across threads, and no bulk kernel to hand +to a device. Whatever the eventual implementation, the *layout* the kernels +consume must be a batch, or those backends cannot be written against it. + +### Prefetching the probe phase + +The canonical treatment of the memory-latency problem is Chen, Ailamaki, +Gibbons, and Mowry, *Improving Hash Join Performance through Prefetching* (ICDE +2004; extended in ACM TODS 32(3), 2007). Its insight is that one probe cannot be +made faster — the chain `hash -> load bucket -> compare -> load payload` is +fully data-dependent — but many *independent* probes can be overlapped. The +paper gives two schedules: + +- **group prefetching**: take a group of probe keys, issue the bucket prefetch + for all of them, then do the comparisons once the lines have arrived; and +- **software-pipelined prefetching**: stagger the stages of consecutive groups + so the prefetches of later keys are in flight while earlier keys are compared. + +Both turn latency-bound probing into throughput-bound probing by keeping many +cache misses outstanding at once, up to the hardware's fill-buffer limit. +Neither is expressible through a scalar lookup; both need the whole group of +keys up front. This is the concrete reason the layout is batch-first. + +The private structural-hash cache (the shipped words' `OnceLock` +components) is a prerequisite: because each key already carries its computed +hash, a bulk `hash_into` mostly gathers cached values, so the group-prefetch +loop reduces to "prefetch the bucket now, confirm later" with no hashing on the +critical path. + +### The batch contract + +Work is presented as a *structure-of-arrays* batch of terms, with the execution +strategy left entirely to the implementation. A batch is columns split along the +two phases of the join, so probe and aggregation touch disjoint memory: + +```rust +/// Keys plus their precomputed structural hashes, in parallel columns. The +/// probe side of the join; it carries no coefficients, so a probe streams only +/// key and hash memory. +pub struct KeyBatch { + keys: W::Column, // structure-of-arrays key planes, owned by the word type + hashes: Vec, // one structural hash per key, parallel to `keys` +} + +/// A `KeyBatch` with the coefficient column attached: the produced terms +/// awaiting merge. Coefficients are a separate column, touched only when a +/// probe resolves to an aggregate. +pub struct TermBatch { + keys: KeyBatch, + coeffs: Vec, +} +``` + +The key column is itself structure-of-arrays and is owned by the concrete key +type, because only that type knows its planes (a packed Pauli word splits into +an X-bit plane block and a Z-bit plane block; the flattened `LossyPauliWord` +adds a loss plane). That capability is a **separate trait**, so `Indexable` +stays minimal and no column type leaks through the hashing contract: + +```rust +/// A key that can be laid out as a structure-of-arrays column. Separate from +/// `Indexable` so the minimal hashing contract is unchanged: a batched key is +/// both `Indexable` (a valid map key) and `Columnar` (has a column layout). +pub trait Columnar: Indexable { + type Column: KeyColumn; +} + +pub trait KeyColumn: Default + Clone { + type Key: Columnar; + + fn len(&self) -> usize; + fn with_capacity(n: usize) -> Self; + + /// Append one produced key; the column keeps each plane contiguous. + fn push(&mut self, key: Self::Key); + + /// Bulk structural hash of the whole column into a parallel hash column. + /// Where per-plane SIMD hashing lives, and what feeds the group-prefetch + /// loop; it must agree bit for bit with the scalar `Hash` of each key. + fn hash_into(&self, out: &mut [u64]); + + /// Join confirm: compare element `i` against a build-side key after a hash + /// or tag match, without materializing the whole element. + fn key_eq(&self, i: usize, other: &Self::Key) -> bool; + + /// Select or permute elements into a new column: the primitive for radix + /// partitioning across threads, compaction during truncation, and staging a + /// sub-batch for a device. Operates plane by plane, never scalar. + fn gather(&self, indices: &[u32]) -> Self; + + /// Scalar materialization of one element — a naive backend's fallback, + /// never the hot path. + fn get(&self, i: usize) -> Self::Key; +} +``` + +The bulk map operations then consume columns. They are the batch spelling of the +existing `ACMap` kernels, added alongside them rather than replacing the +semantic `SumStorage` methods: + +```rust +pub trait ACMapBatch +where + W: Word + Columnar, + C: Coefficient, +{ + /// Merge a batch into the sum: for each term, accumulate onto an existing + /// key or insert a new one. The build/probe-with-group-by of a hash join. + /// The implementation owns whether it runs scalar, group- or + /// pipeline-prefetched, SIMD-vectorized, hash-partitioned across threads, + /// or offloaded; the contract fixes only the result and the layout. + fn upsert_batch(&mut self, batch: &TermBatch); + + /// Read-only probe of a key column, for the overlap and expectation paths. + /// Takes a `KeyBatch` so the precomputed hash column drives the prefetch + /// with no coefficient column in the working set. + fn probe_batch(&self, keys: &KeyBatch, out: &mut [Option]); +} +``` + +Three points make this layout the performant one rather than a cosmetic +reshuffle: + +- **Column separation follows the join phases.** The probe touches keys and + hashes; aggregation touches coefficients. Separate columns keep the probe's + bandwidth and cache footprint minimal and let a backend place the coefficient + column on a different device or NUMA node from the keys. +- **The hash column is precomputed and contiguous.** The group-prefetch kernel + streams the hash column, prefetches the matching buckets, and only then + confirms against the key column — the private hash cache above keeps that + column cheap to fill. +- **Plane-level structure-of-arrays inside the key column** lets a backend hash + and compare the same machine-word lane across many keys with vertical SIMD, + and gives a GPU coalesced loads. The concrete plane layout, alignment, and + padding live in [`word-data-structures.md`](word-data-structures.md); they are + never visible through `KeyColumn`. + +`upsert_batch` is where the branch-staging merge lands: the reusable scratch +buffer of the previous section becomes a `TermBatch` — the probe side of the +join — and the `SumStorage` merge desugars to it. The scalar `SumStorage` +closures remain the semantic surface for a naive backend, but generic hot paths +build a `TermBatch` and hand it to the batch kernel. + +Gate and rotation traits change accordingly, and this is the interface change +users feel first: term *production* is separated from term *insertion*. A +rotation appends produced terms into a `TermBatch` — filling the key column and +the coefficient column as it goes — instead of mutating the map through a +callback. That decoupling, plus the columnar layout, is what lets the produced +batch be prefetched, vectorized, partitioned, or shipped to a device before it +ever touches the table, and it keeps the propagation rule (the physics) +independent of the merge strategy (the systems concern). + +### What the batch contract asks of the other contracts + +- `Indexable` is unchanged and stays minimal. The batch surface lives on the + separate `Columnar` trait, which a key type implements only when it is used in + a batch. A packed Pauli word's column is its X and Z plane blocks and the + flattened `LossyPauliWord` adds a loss plane; `Phased` is not indexable, so + it is not `Columnar` and never appears in a batch. +- `Word`: unaffected — `set` and friends still mutate one value; columns are + built by appending produced keys, not by mutating in place. +- `Policy`: truncation already operates on the whole map and is naturally bulk; + it should be expressible as a batch retain (via `KeyColumn::gather`) so it + composes with a partitioned or offloaded table. +- `SumStorage`: its staging fields become the join's probe-side buffer and its + merge desugars to `upsert_batch`. Whether it keeps both an auxiliary map and a + scratch buffer stays a backend decision, unchanged from above. + +### Parallel and offloaded backends + +Recognizing the operation as a hash join fixes the parallel and GPU stories +without further interface changes. A partitioned hash join radix-partitions both +sides by the high bits of the key hash so each partition is a disjoint +sub-table; partitions then merge independently — one per thread, or one per GPU +block — with no cross-partition synchronization. The batch contract is precisely +what a partitioner consumes: `upsert_batch` may internally `gather` a +`TermBatch` into per-partition batches and run them concurrently, and a device +backend may copy a `TermBatch` across the host/device boundary and run the probe +as a kernel. None of this is visible in the contract; all of it is precluded by +a scalar one. This is the partition-then-merge shape the `DashMap` backend +already gestures at, now stated as the contract that makes such backends +interchangeable rather than separate code paths. + ## Indexable values Hash-enabled `Word` values and `Tableau` can both be expensive, mostly-stable @@ -520,8 +739,32 @@ Domain-specific aliases or wrappers can preserve `PauliSum` and introduce - Exposing cache representation or invalidation through `Indexable`. - Preserving `Copy` at the expense of correct lazy caching. - Adding runtime dispatch for storage, hashing, or algorithm policies. +- Committing to a single batch-execution backend. The hash-join contract admits + scalar, prefetched, SIMD, thread-partitioned, and device backends; which one + ships is a later benchmark decision, not a type-system decision. +- Implementing the prefetch or partition schedule itself in this iteration. The + batch contract is defined here; the group/pipeline schedule and partitioner + are implementation work behind it. ## Open design questions 1. Do benchmarks justify retaining both the auxiliary-map and vector-staging fast paths in the default sparse-sum storage backend? +2. At what granularity does the key column split into planes, and how are those + planes padded and aligned — to the widest supported SIMD lane, to a cache + line, or to a device's coalescing width? The contract fixes that the column + is structure-of-arrays; the plane granularity and alignment are open, and may + need to vary by target. +3. What batch (group) size does each backend use, and should the contract expose + it at all, or keep it entirely internal to the implementation? +4. Should `probe_batch` yield coefficients, or opaque slot handles a later read + resolves, so a caller can probe once and then mutate the located slot? + +## References + +- Shimin Chen, Anastassia Ailamaki, Phillip B. Gibbons, and Todd C. Mowry. + "Improving Hash Join Performance through Prefetching." *Proceedings of the + 20th International Conference on Data Engineering (ICDE)*, 2004. Extended + version: *ACM Transactions on Database Systems* 32(3), Article 17, 2007. The + origin of group prefetching and software-pipelined prefetching for the probe + phase, and the cost model the batch contract is built to serve. diff --git a/docs/design/word-data-structures.md b/docs/design/word-data-structures.md index 30d87f229..5d72634aa 100644 --- a/docs/design/word-data-structures.md +++ b/docs/design/word-data-structures.md @@ -239,6 +239,12 @@ can populate them through `&self`; structural mutators clear affected cells through `&mut self`. This preserves `Send + Sync` without imposing either bound on the `Indexable` trait. +The same component caches back the bulk `hash_into` on a word's key column (see +[Key columns](#key-columns-structure-of-arrays-batches)): filling a batch's hash +column mostly gathers already-cached values, so the group-prefetch loop in the +[batch contract](traits-2-configuration-and-hashing.md#batch-execution-and-the-hash-join-contract) +pays no hashing on its critical path. + ## Component hashes Hash composition follows the logical wrappers: @@ -268,6 +274,55 @@ X/Z. `Phased` is absent from this composition because it is not indexable. Constructors leave caches empty. Cloning may copy a valid cached value because the clone initially has identical structural contents. +## Key columns (structure-of-arrays batches) + +The bulk map contract in +[`traits-2-configuration-and-hashing.md`](traits-2-configuration-and-hashing.md#batch-execution-and-the-hash-join-contract) +consumes a structure-of-arrays column of keys through the `Columnar` trait's +associated `Column`. Each concrete word owns the planar layout of its column; +generic code sees only the `KeyColumn` capability, never the planes. `Columnar` +is separate from `Indexable`, so this adds no surface to the minimal hashing +contract. + +For the packed Pauli word the column is two plane blocks and a shared width, +with the hash column stored parallel to it: + +```text +x_plane: [ word 0 X-bits | word 1 X-bits | ... | word M-1 X-bits ] +z_plane: [ word 0 Z-bits | word 1 Z-bits | ... | word M-1 Z-bits ] +nqubits: shared width +hashes: [ h(0) | h(1) | ... | h(M-1) ] (in the owning KeyBatch) +``` + +The flattened `LossyPauliWord` extends its column with a parallel loss-bit +plane, matching its X/Z/loss structural identity. `Phased` has no column: it +is not indexable, so it is never a batch key. + +The layout is chosen for the three access patterns the trait contract must +serve: + +- **Vertical SIMD.** A contiguous plane lets a backend hash or compare the same + machine-word lane across many keys in one vector instruction, instead of + gathering scattered `[u8; N]` words. +- **Coalesced device access.** A contiguous plane maps to a coalesced load for a + GPU warp; an array of packed words would not. +- **Bandwidth-minimal probe.** The probe streams the X/Z planes and the hash + column and never loads coefficients, which live in the caller's `TermBatch`. + +Two representation choices are therefore fixed here rather than in the trait +layer: + +- planes are stored as flat plane blocks, not `Vec<[u8; N]>`, and each key's + plane slot is padded to the alignment the backend's widest SIMD lane (or a + device's coalescing width) requires; and +- `hash_into` computes the whole hash column from the planes and from the + private `OnceLock` component caches, so the plane-parallel hash and the + scalar `Hash` in [Component hashes](#component-hashes) must agree bit for bit. + +`gather` selects or permutes columns by index for radix partitioning, truncation +compaction, and device staging; it copies plane by plane and never materializes +scalar words. + ## Ordering and serialization `Eq`, `Ord`, `Hash`, display, parsing, and serialization must agree on logical @@ -302,3 +357,8 @@ The prototype should include: 1. Should the loss plane use the same packed array width as the X/Z planes, or a separately selected private width? +2. What plane alignment and padding does the key column use, and must it vary by + target (for example AVX-512 vs NEON vs a GPU coalescing width)? A per-target + column would need the plane block parameterized by alignment. +3. Does the `LossyPauliWord` column store its loss plane inline beside X/Z (one + allocation, one `gather`), matching the flattened scalar representation? From 9e680e90957b016d42e6e4d5526474e20addb074 Mon Sep 17 00:00:00 2001 From: Roger-luo Date: Thu, 30 Jul 2026 18:37:47 -0400 Subject: [PATCH 04/10] docs(design): decompose Coefficient, collapse Sum with alias bundle Address the Step 2 (type-composition) design gaps surfaced in review: - Split the Coefficient god-trait: cutoff moves to Policy (value type now exposes only magnitude()), sin_cos moves to a separate Angle domain so the rotation angle is no longer welded to the coefficient. RotationOne gains a defaulted angle parameter (A = C). - Give SumStorage associated Word/Coeff types, collapsing the engine to Sum and removing the phantom word parameter. - Rename OperatorSum -> Sum; reintroduce ergonomics as a defaulted type-alias family (PauliSum/LossyPauliSum/FermionSum) that is explicitly not load-bearing at the trait level, so it avoids Config's two sins. - Drop the Policy: Copy bound (contradicted the proposal's own non-goal). - Document the n_sites width invariant with a debug_assert. - Update the LossChannel specialization to select on S::Word. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../traits-2-configuration-and-hashing.md | 191 +++++++++++++++--- docs/design/word-data-structures.md | 10 +- 2 files changed, 166 insertions(+), 35 deletions(-) diff --git a/docs/design/traits-2-configuration-and-hashing.md b/docs/design/traits-2-configuration-and-hashing.md index aa0e6395d..7bdb00827 100644 --- a/docs/design/traits-2-configuration-and-hashing.md +++ b/docs/design/traits-2-configuration-and-hashing.md @@ -38,7 +38,7 @@ responsibility changes. ## Type-composition layers -### Coefficient type +### Coefficient, angle, and truncation There is no algorithm-agnostic `Config` trait. Algorithms use the coefficient type directly: @@ -54,6 +54,57 @@ composition. Maps, policies, Pauli-word storage, Pauli-word implementations, and hashers are also selected independently rather than being collected into a replacement global configuration trait. +The decomposition must not stop at `Config`; the current `Coefficient` trait is +itself a bundle. Its own documentation states that it "bundles every arithmetic +operation," and it carries two responsibilities that are not value-domain +arithmetic: `sin_cos` (a rotation concern) and `cutoff` (a truncation concern — +the very policy this redesign extracts). `Coefficient` is therefore split into a +value domain, a separate angle domain, and a policy predicate. + +The value domain keeps only arithmetic and a magnitude, replacing `cutoff` with +a property that a policy can threshold: + +```rust +/// Value-domain arithmetic only — no rotation, no truncation. +pub trait Coefficient: + PartialEq + Clone + num::Zero + + Neg + + Add + Sub + + Mul + Mul + + AddAssign + MulAssign + + std::iter::Sum + Send + Sync +{ + fn mul_sign(&self, sign: i8) -> Self; + fn half(&self) -> Self; + + /// Nonnegative magnitude. Exposes a property of the value for a `Policy` to + /// threshold; it does not itself decide any cutoff. Replaces `cutoff`. + fn magnitude(&self) -> f64; +} +``` + +The rotation angle becomes a separate domain, so it is no longer welded to the +coefficient. The angle type defaults to the coefficient, recovering today's +`rx(theta: C)` for the common case while permitting an `f64`-coefficient sum +driven by a symbolic or parametric angle: + +```rust +/// A rotation angle that yields `(sin, cos)` in coefficient domain `C`. +pub trait Angle { + fn sin_cos(&self) -> (C, C); +} + +impl Angle for f64 { + fn sin_cos(&self) -> (f64, f64) { num::traits::Float::sin_cos(*self) } +} +``` + +The rotation traits (see [Behavioral traits](#behavioral-traits)) take the angle +domain as a defaulted parameter. Whether the "concrete coefficient, symbolic +angle" combination is supported or deliberately forbidden (via a `where A = C` +bound) is an explicit decision recorded in the open questions, not a silent +property of one fused trait. + ### Representation types There is no separate global storage configuration, and representation storage @@ -139,8 +190,8 @@ pub trait Clifford { // ... } -pub trait RotationOne { - fn rx(&mut self, qubit: usize, theta: C); +pub trait RotationOne = C> { + fn rx(&mut self, qubit: usize, theta: A); // ... } @@ -186,27 +237,75 @@ parameters with one. In particular, there is no `PauliSumAlgorithm` trait that bundles a term map with a policy: storage layout and policy are orthogonal choices. -The reusable sparse-sum shape is: +The choices that are *intrinsic to a storage instance* — which word it keys on +and which coefficient it accumulates — are associated types of that storage +rather than free parameters, the same way `HashMap` fixes `K` and `V` per +concrete map. `SumStorage` (defined below) therefore exposes `type Word` and +`type Coeff`, and the reusable sparse-sum shape needs only its storage and its +policy: ```rust -pub struct OperatorSum +pub struct Sum where - C: Coefficient, - W: Word + Indexable, - S: SumStorage, - P: Policy, + S: SumStorage, + P: Policy, { storage: S, policy: P, + /// Invariant: every key `k` in `storage` satisfies `k.n_sites() == + /// n_sites`. An empty sum has no word to derive the width from, so the + /// field is carried explicitly and checked by a `debug_assert!` on every + /// insertion path. n_sites: usize, } ``` -Here `C`, `W`, `S`, and `P` respectively mean coefficient domain, algebraic -word, concrete sparse-sum storage engine, and algorithm policy. Each -parameter has an independent meaning. Propagation methods select their algebra -through the site type, for example `W: Word` or -`W: Word`. +`Word` is no longer a free parameter with no home; it is `S::Word`, so there is +no phantom axis and no way to pair a storage type with the wrong word. This also +resolves the tension with the earlier note that associated types are not +justified merely to shrink a parameter count: `Word` and `Coeff` are genuinely +intrinsic to a storage instance, so making them associated types is proper +encapsulation, not a false bundle. Propagation methods still select their +algebra through the site type by bounding `S::Word: Word` or +`Word`. + +#### The convenience bundle + +Collapsing to `Sum` restores ergonomics through a type-alias family, not a +foundational trait. Each alias fixes a domain's axes and defaults the common +knobs, so a one-token `PauliSum` name returns: + +```rust +pub type PauliSum = Sum, P>; +pub type LossyPauliSum = Sum, P>; +pub type FermionSum = Sum, P>; +``` + +A user who wants to pin a reusable configuration writes one `type` line instead +of an `impl Config`: + +```rust +type MyPauliSum = Sum>, MaxPauliWeight>; +``` + +This is deliberately a bundle, and it is safe to be one because it commits +neither of `Config`'s two sins: + +- **It is not load-bearing at the trait level.** The shared behavioral traits + (`Clifford`, `RotationOne`, `PauliError`, `Measure`) are implemented on `Sum` + and take `S::Coeff` directly. No gate or noise trait is generic over the + alias, so an algorithm that needs only a coefficient never names storage or + policy. +- **Its axes stay independently overridable.** Because it is a plain alias over + defaulted parameters, changing one axis is `Sum`, not a new + trait impl. `Config`'s failure was fusing orthogonal axes into one impl you + had to rewrite wholesale to vary a single choice; a defaulted alias has the + opposite property. + +The name `Sum` shares a bare spelling with the `std::iter::Sum` *trait* that +`Coefficient` bounds on. They coexist because one is a type and one is a trait, +and the bound is written fully qualified; `PauliSum` and `FermionSum` remain the +public domain-facing names in any case. `Policy` is the proposed name for the current `Strategy` concept. It retains the current responsibilities: predicting initial capacity and truncating the @@ -216,7 +315,7 @@ their meaning; `NoStrategy` and `CombinedStrategy` become `NoPolicy` and established names: ```rust -pub trait Policy: Default + Clone + Copy +pub trait Policy: Default + Clone where W: Word + Indexable, C: Coefficient, @@ -229,6 +328,32 @@ where } ``` +`Policy` does not require `Copy`. The bound bought nothing — policies are used +through `&self` — and would forbid a stateful policy such as a per-region +threshold vector or an adaptive budget; requiring it would also contradict this +proposal's own non-goal of preserving `Copy` at the expense of correctness. + +Truncation policy also reclaims the cutoff decision from `Coefficient`. The +value type now exposes only `magnitude()`, a property of the value; the policy +owns the comparison: + +```rust +#[derive(Default, Clone)] +pub struct CoefficientThreshold { + pub threshold: f64, +} + +impl Policy for CoefficientThreshold { + fn capacity(&self, _n_sites: usize) -> usize { + 0 + } + + fn truncate>(&self, map: &mut M) { + map.retain(|_word, coeff| coeff.magnitude() >= self.threshold); + } +} +``` + `Policy` and its concrete implementations belong to the sparse-sum crate. This removes the current split where the `Strategy` trait lives in `ppvm-traits` but its concrete strategies live in `ppvm-pauli-sum`; the policy is not an @@ -250,27 +375,25 @@ directly by `PauliSum`: its maps and reusable workspace. It is an actual value, not a marker configuration: ```rust -pub trait SumStorage: Clone -where - W: Word + Indexable, - C: Coefficient, -{ - type Map: ACMap; +pub trait SumStorage: Clone { + type Word: Word + Indexable; + type Coeff: Coefficient; + type Map: ACMap; fn data(&self) -> &Self::Map; fn data_mut(&mut self) -> &mut Self::Map; fn map_insert(&mut self, f: F) where - F: Fn(&W, &mut C) -> Option<(W, C)>; + F: Fn(&Self::Word, &mut Self::Coeff) -> Option<(Self::Word, Self::Coeff)>; fn map_insert_multiple(&mut self, f: F) where - F: Fn(&W, &mut C) -> Option>; + F: Fn(&Self::Word, &mut Self::Coeff) -> Option>; fn map_add(&mut self, f: F) where - F: Fn(&W, &C) -> (W, C); + F: Fn(&Self::Word, &Self::Coeff) -> (Self::Word, Self::Coeff); fn consume(&mut self); } @@ -301,9 +424,9 @@ that admits prefetched, SIMD, threaded, and offloaded backends — is specified [Batch execution and the hash-join contract](#batch-execution-and-the-hash-join-contract) below. -The generalized engine may be named `OperatorSum`, but the Pauli specialization -retains the existing domain-facing `PauliSum` name. This is a new internal -generalization, not a requirement to rename Pauli call sites. +The generalized engine is named `Sum`, but the Pauli specialization retains the +existing domain-facing `PauliSum` name (a defaulted type alias over `Sum`). This +is a new internal generalization, not a requirement to rename Pauli call sites. A classical tableau mixture follows the same principle and takes its `EntryStore` directly rather than introducing a one-associated-type @@ -322,7 +445,7 @@ where `GeneralizedTableauSum` and `EntryStore` retain their current names because the proposal does not change their underlying roles. `GeneralizedTableauSum` and -`OperatorSum` are both sparse linear combinations of indexable keys, so they +`Sum` are both sparse linear combinations of indexable keys, so they may eventually share an implementation. This iteration deliberately keeps them separate: their mutation, branching, normalization, and storage requirements have not yet been reduced to a proven common interface. The next design @@ -345,11 +468,13 @@ changed according to whether their underlying responsibility changes: | concrete `LossyPauliWord` | `LossyPauliWord` | The packed X/Z/loss representation remains concrete and flattened. | | `PhasedPauliWord` | `PhasedPauliWord` alias over non-indexable `Phased` | The wrapper is generic over ordinary and lossy words but is not a production map key. | | `rehash` | private cache invalidation | Recalculation changes from eager mutation-time work to lazy demand-time work without exposing cache mechanics through `Indexable`. | -| `Strategy` | `Policy` | Intentional terminology change requested for this redesign. | +| `Strategy` | `Policy` | Intentional terminology change requested for this redesign; the `Copy` bound is dropped. | +| `Coefficient::cutoff` | removed | The truncation predicate moves to `Policy`; `Coefficient::magnitude` exposes only the value property the policy thresholds. | +| `Coefficient::sin_cos` | `Angle` | The rotation angle becomes a separate domain, defaulting to the coefficient type. | | `ACMap` | `ACMap` | The associative coefficient map has the same role. | | `PauliSum::data`, `map_insert`, `map_add` | same method names on `SumStorage` | These semantic operations already match the proposed boundary. | | `PauliSum` map pair and `scratch` fields | `SumStorage` | A new abstraction is extracted from currently unnamed storage state. | -| `PauliSum` | `PauliSum` over generalized `OperatorSum` machinery | Pauli-facing code keeps its established name; `OperatorSum` names the new cross-algebra engine. | +| `PauliSum` | `PauliSum` over generalized `Sum` machinery | Pauli-facing code keeps its established name; `Sum` names the new cross-algebra engine. | | `GeneralizedTableauSum` | `GeneralizedTableauSum` | The classical mixture algorithm remains the same concept. | | `EntryStore`, `VecStorage`, `MapStorage` | unchanged | The proposal uses the existing storage boundary and implementations. | | `BuildHasher` | associated with `Indexable` | Hasher ownership moves from `Config` to each indexable key type. | @@ -357,7 +482,7 @@ changed according to whether their underlying responsibility changes: | `PauliStorage` | removed | Packed backing storage becomes private to the concrete word representation. | | (new) | `Columnar`, `KeyColumn`, `KeyBatch`, `TermBatch`, `ACMapBatch` | Structure-of-arrays batch/hash-join contract, kept as a separate trait so `Indexable` stays minimal; see [Batch execution and the hash-join contract](#batch-execution-and-the-hash-join-contract). | -Names such as `Word`, `Indexable`, `SumStorage`, and `OperatorSum` are +Names such as `Word`, `Indexable`, `SumStorage`, and `Sum` are therefore new because they denote abstractions that do not exist in the current implementation, not because the existing API is being renamed wholesale. @@ -720,7 +845,7 @@ do not leak into the shared trait-system design. The intended composition is explicit: ```rust -OperatorSum +Sum // e.g. PauliSum = Sum> Tableau GeneralizedTableauSum ``` @@ -731,7 +856,7 @@ Domain-specific aliases or wrappers can preserve `PauliSum` and introduce ## Non-goals for the first prototype - Migrating the existing crates to `ppvm-traits-2` immediately. -- Merging `GeneralizedTableauSum` and `OperatorSum` in this iteration; only a +- Merging `GeneralizedTableauSum` and `Sum` in this iteration; only a smaller proven common factor should be considered in the next iteration. - Defining one collection interface shared by all algorithms. - Requiring every sparse-sum storage backend to physically contain both an diff --git a/docs/design/word-data-structures.md b/docs/design/word-data-structures.md index 5d72634aa..554a3b5a4 100644 --- a/docs/design/word-data-structures.md +++ b/docs/design/word-data-structures.md @@ -180,13 +180,19 @@ Loss channels and maximum-loss-weight truncation specialize directly on the concrete lossy word: ```rust -impl LossChannel - for OperatorSum, S, P> +impl LossChannel for Sum +where + S: SumStorage>, + P: Policy, { // ... } ``` +The specialization is expressed through the storage's associated `Word` type +rather than a free word parameter: a `Sum` is a `LossChannel` exactly when its +`SumStorage` keys on a `LossyPauliWord`. + There are no traits named `PauliWord`, `LossyPauliWord`, or `FermionWord`. `PauliWord` and `LossyPauliWord` remain concrete domain type names. Algorithms select the algebra through `Word::Site`; loss-only operations remain inherent From 33b67c7915d9d98eb973416f4caf488ef510b222 Mon Sep 17 00:00:00 2001 From: Roger-luo Date: Thu, 30 Jul 2026 19:26:21 -0400 Subject: [PATCH 05/10] docs(design): make Word read-only, add Sp-vs-phase Pauli algebra traits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolve the Step 3 (Word / Site) design gaps from review: - Make Word a read-only inspection trait (n_sites, get, weight, iter); remove set. This stops Word from overclaiming as the propagation interface (the real kernels are sub-site) and lets ordered algebras implement it honestly, where in-place positional set is ill-defined. - Relocate mutation to Pauli-specific traits, decomposed along the Clifford = Sp(2n,2) join phase structure: - PauliBits: mutable single-vector X/Z access; hosts rotation kernels. - SymplecticColumns (role-independent-semantics bit-plane column ops) and PhaseTrack (role-dependent Z4-word vs Z2+g-tableau phase), kept separate to mirror Sp and its extension. - Clifford: blanket impl over the two primitives — one audited copy of the symplectic sign logic, shared by PhasedPauliWord and Tableau. - StabilizerFrame: role-exclusive tableau ops (measure, row_multiply). - PauliSum consumes the one-row action pointwise (Clifford re-keys the map); word-side Mul / key identity stay inherent. - Update the compatibility table, trait admission rule, batch-contract note, and the word/tableau data-structure docs to match. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/design/tableau-data-structure.md | 22 +++ .../traits-2-configuration-and-hashing.md | 159 ++++++++++++++++-- docs/design/word-data-structures.md | 16 +- 3 files changed, 177 insertions(+), 20 deletions(-) diff --git a/docs/design/tableau-data-structure.md b/docs/design/tableau-data-structure.md index 7d5379dcc..2bb430067 100644 --- a/docs/design/tableau-data-structure.md +++ b/docs/design/tableau-data-structure.md @@ -298,6 +298,28 @@ This table describes logical mutations. A gate may determine that no phase bit changed and preserve the phase cache in that special case, but conservative component invalidation is correct for the first implementation. +### Shared algebra traits + +The gate rows above are exactly the Pauli algebra primitives defined in +[`traits-2-configuration-and-hashing.md`](traits-2-configuration-and-hashing.md#pauli-algebra-traits-symplectic-structure-and-phase), +realized at tableau width. The tableau implements: + +- `SymplecticColumns` — `swap_xz`, `xor_x_col`, `xor_z_col` as SIMD-block + operations over the `2n` generator rows (the "column pair" access above); +- `PhaseTrack` — the \(\mathbb{Z}_2\) sign plane plus the Aaronson–Gottesman `g` + rule for row products; and +- `StabilizerFrame` — the role-exclusive operations (measurement, pivot search, + `row_multiply`, canonicalization) that read the rows as a stabilizer/ + destabilizer basis. + +Implementing the first two yields the blanket `Clifford` impl for free, so the +tableau shares one audited copy of the symplectic sign logic with +`PhasedPauliWord`; only the phase algebra and the frame operations are +tableau-specific. `swap_xz` at tableau width is a block swap and `xor_x_col` is a +whole-column `⊕`, which is why the column-major layout matters: the shared gate +*sequence* is width-agnostic, but the tableau's implementation of each primitive +is the SIMD path the word never needs. + ## Structural hashing A tableau used in a classical mixture is an `Indexable` key. It owns its own diff --git a/docs/design/traits-2-configuration-and-hashing.md b/docs/design/traits-2-configuration-and-hashing.md index 7bdb00827..f33a4785a 100644 --- a/docs/design/traits-2-configuration-and-hashing.md +++ b/docs/design/traits-2-configuration-and-hashing.md @@ -112,9 +112,18 @@ does not appear as an associated type. A concrete value encapsulates its own fields; generic algorithms use behavioral methods instead of naming or inspecting the backing memory. -`Word` is the common concept for an indexed algebraic monomial. The old -Pauli-word operations are not Pauli-specific: every supported word has a site -alphabet, an indexed extent, site access and mutation, and a weight: +`Word` is the common **read-only** concept for an indexed algebraic monomial. It +is an *inspection* interface — extent, per-site read, weight, and iteration — +consumed by display, serialization, tests, and the sparse-sum plumbing. It is +deliberately **not** the propagation interface: the real gate kernels operate at +*sub-site* (individual X/Z bit) granularity and are algebra-specific, so they +live on the Pauli-specific traits in +[Pauli algebra traits](#pauli-algebra-traits-symplectic-structure-and-phase), +not on `Word`. Keeping `Word` free of mutation also lets an ordered algebra (a +normal-ordered fermionic product) implement it honestly: an in-place +`set(index, site)` at a factor position would violate that algebra's canonical +form, so mutation is relocated to each algebra's own traits rather than asserted +universally here. ```rust pub trait Word { @@ -122,8 +131,8 @@ pub trait Word { fn n_sites(&self) -> usize; fn get(&self, index: usize) -> Self::Site; - fn set(&mut self, index: usize, site: Self::Site); fn weight(&self) -> usize; + fn iter(&self) -> impl Iterator; } ``` @@ -154,11 +163,14 @@ concrete packed lossy word implements `Word>`, and a future ordered fermionic product can implement `Word`. A fermionic word's index denotes factor order; `FermionSite` carries the physical mode. For a dense Pauli word, the index is the qubit and `n_sites()` is its -width. +width. Because `Word` is read-only, these differing index meanings (qubit vs +factor order) never have to agree on a shared `set` contract. `weight()` is the number of non-identity factors according to the concrete site alphabet; an ordered representation that stores no explicit identities may -therefore have `weight() == n_sites()`. Implementations of `set()` preserve -their representation invariants and invalidate the affected hash components. +therefore have `weight() == n_sites()`. It remains a Pauli-motivated read (the +`MaxPauliWeight` policy needs it) that other algebras may define as is natural. +Structural mutation, and the hash-component invalidation it triggers, belong to +the algebra-specific mutation traits below, not to `Word`. The concrete `LossyPauliWord` stores packed X, Z, and loss planes directly and provides loss mutation and `loss_weight()` as inherent methods. Loss channels @@ -229,7 +241,112 @@ tableau implementations are later useful, each concrete type can implement the same behavioral traits. A storage abstraction should only be introduced after two implementations demonstrate a common interface. -### Algorithm and storage parameters +`Clifford` is not implemented by hand on each type. It is a *derived* behavioral +trait, blanket-implemented once over the Pauli algebra primitives described +next; `RotationOne` is implemented on `PauliSum` in terms of `PauliBits`, also +below. + +### Pauli algebra traits: symplectic structure and phase + +`Word` is read-only, so the mutation that gate propagation performs lives on +Pauli-specific traits. Their shape is dictated by the algebra rather than +invented: a Pauli operator modulo phase is a vector in the symplectic space +\(\mathrm{GF}(2)^{2n}\) (the X and Z bit planes), Pauli multiplication is +vector addition (`⊕`), commutation is the symplectic form +\(\omega(P,Q) = x_P\cdot z_Q \oplus z_P\cdot x_Q\), and the Clifford group is the +central extension \(\mathrm{Sp}(2n,2) \ltimes \text{phases}\). That factorization +sorts every gate operation into two buckets, and the traits follow the buckets. + +**Role-independent (the \(\mathrm{Sp}\) part).** Conjugating a Pauli by a +Clifford gate does the same bit-plane algebra whether the operator is a lone +word (one row) or one of a tableau's \(2n\) stabilizer/destabilizer generators. +`H` swaps the X and Z columns; `S` does `z ⊕= x`; `CNOT` does +`x_t ⊕= x_c, z_c ⊕= z_t`. This logic is written **once**. + +**Role-dependent (the phase extension).** A standalone phased Pauli lives in +\(\mathbb{Z}_4\) (`Y = iXZ` needs the `i`); a stabilizer tableau stores a +\(\mathbb{Z}_2\) sign and recovers the `i`'s through the Aaronson–Gottesman `g` +rule during row multiplication. Same gate, different phase algebra — so phase +bookkeeping is written **per type**. + +Two primitive traits capture the two buckets, kept separate so they mirror +\(\mathrm{Sp}\) and its extension (and so a future phase-free classical register +can implement only the first): + +```rust +/// Sp-part: bit-plane column algebra. `PhasedPauliWord` uses 1-bit columns; +/// `Tableau` uses SIMD blocks over its 2n rows. Same meaning, different width. +pub trait SymplecticColumns { + fn n_qubits(&self) -> usize; + fn swap_xz(&mut self, q: usize); + fn xor_x_col(&mut self, ctrl: usize, tgt: usize); + fn xor_z_col(&mut self, tgt: usize, ctrl: usize); +} + +/// Extension-part: the phase algebra. Z4 for a phased word, Z2 + g for a tableau. +pub trait PhaseTrack { + fn flip_phase_where_xz(&mut self, q: usize); + fn cnot_phase(&mut self, ctrl: usize, tgt: usize); + // ...one phase delta per gate; the tableau's g-rule lives behind these +} +``` + +`Clifford` is then the role-independent structure, blanket-implemented once. The +*sequence* of primitives per gate is identical across roles even though the +phase primitive it calls is not — the single audited copy of the symplectic sign +logic that would otherwise be duplicated and drift: + +```rust +impl Clifford for T { + fn h(&mut self, q: usize) { self.flip_phase_where_xz(q); self.swap_xz(q); } + fn cnot(&mut self, c: usize, t: usize) { + self.cnot_phase(c, t); + self.xor_x_col(c, t); + self.xor_z_col(c, t); + } + // ... +} +``` + +The role-*exclusive* operations — those that interpret the rows as a symplectic +basis rather than as independent operators — do not belong in the shared tower. +They are a tableau-only trait a word never implements: + +```rust +pub trait StabilizerFrame { + fn measure(&mut self, qubit: usize) -> Option; + fn row_multiply(&mut self, src: usize, dst: usize); // uses the g-rule + // pivot search, canonicalization, transposition guard, ... +} +``` + +The implementers are `PhasedPauliWord` (`SymplecticColumns + PhaseTrack`, one +row) and `Tableau` (all three). Note that `PauliSum` is deliberately *not* an +implementer: a Clifford gate on a sum re-keys every term (each Pauli maps to a +different Pauli, so the map is rebuilt, not updated in place), so the sum applies +the one-row action pointwise and drains each term's phase delta to its +coefficient. Non-Clifford rotations, which branch one term into several, stay on +a separate mutation primitive: + +```rust +/// Mutable single-vector X/Z access — a point of GF(2)^{2n}. Hosts the +/// rotation/branching kernels, which flip individual bits and ship the sign to +/// the coefficient. Implemented by `PauliWord` and `LossyPauliWord`. +pub trait PauliBits: Word { + fn x_bit(&self, i: usize) -> bool; + fn z_bit(&self, i: usize) -> bool; + fn set_x_bit(&mut self, i: usize, v: bool); // invalidates the hash lazily + fn set_z_bit(&mut self, i: usize, v: bool); + fn is_lost(&self, i: usize) -> bool { false } // LossyPauliWord overrides +} +``` + +`PauliBits` is the narrow bit-level slice of the retired `PauliWordTrait`, +separated from key identity (`Indexable`), inspection (`Word`), and phase — and +it passes the trait admission rule because generic rotation kernels consume it +across two implementers. Loss *reads* are a defaulted predicate; loss *writes* +and the binary Pauli product (`Mul`) stay inherent on the concrete words, since +each has a single implementer. An algorithm should take its independent choices as direct type parameters. An associated-type bundle is not useful merely because it replaces two type @@ -462,8 +579,11 @@ changed according to whether their underlying responsibility changes: | Current implementation | Proposal | Rationale | | --- | --- | --- | | `Config` | removed | The bundle itself is removed; this is not a rename. | -| `PauliWordTrait` | `Word` plus `Indexable` where used as a key | Word operations are generalized through `Word::Site`; hashing becomes a separate capability. | -| `n_qubits`, `get`, `set`, `weight` | `n_sites`, `get`, `set`, `weight` | Only the Pauli-specific extent name changes; the other operation names stay. | +| `PauliWordTrait` | split into `Word` (read-only inspection), `Indexable` (key), `PauliBits` (mutation) | The old bundle is decomposed by concern; bit-level mutation is a narrow Pauli-specific trait, hashing is separate, inspection is algebra-agnostic. | +| `n_qubits`, `get`, `set`, `weight` | `n_sites`, `get`, `weight` on `Word`; `set` removed | `Word` is read-only; positional mutation (`set`) moves to `PauliBits`, since it is ill-defined for ordered algebras. | +| bit accessors `get_xbit`/`set_xbit`/… | `PauliBits::x_bit`/`set_x_bit`/… | The rotation hot path is sub-site; it keeps a dedicated Pauli trait rather than the generic `Word`. | +| word-level Clifford (blanket over `PauliWordTrait`) | `Clifford` blanket over `SymplecticColumns` + `PhaseTrack` | The symplectic sign logic is written once and shared by `PhasedPauliWord` and `Tableau`. | +| (new) | `SymplecticColumns`, `PhaseTrack`, `StabilizerFrame` | The `Sp ⋉ phase` decomposition: role-independent column algebra, role-dependent phase, role-exclusive frame ops. | | concrete `PauliWord` | `PauliWord` | The packed X/Z word is the same domain concept. | | concrete `LossyPauliWord` | `LossyPauliWord` | The packed X/Z/loss representation remains concrete and flattened. | | `PhasedPauliWord` | `PhasedPauliWord` alias over non-indexable `Phased` | The wrapper is generic over ordinary and lossy words but is not a production map key. | @@ -498,7 +618,13 @@ This rule keeps: - `Indexable`, consumed by keyed stores and implemented by hash-enabled words and tableaus; -- `Word`, consumed by the shared sparse-sum engine and propagation algorithms; +- `Word`, consumed by display, serialization, tests, and the sparse-sum + plumbing as a read-only inspection interface (not the propagation interface); +- `PauliBits`, consumed by the rotation kernels and implemented by `PauliWord` + and `LossyPauliWord`; +- `SymplecticColumns` and `PhaseTrack`, consumed by the blanket `Clifford` and + implemented by `PhasedPauliWord` and `Tableau`; +- `StabilizerFrame`, the role-exclusive tableau operations; - gate and noise traits, implemented across propagation and tableau backends; - `SumStorage` and `ACMap`, implemented by genuinely different storage engines and collections; @@ -508,9 +634,11 @@ This rule keeps: It rejects the removed global `Config`, `PauliSumAlgorithm`, `TableauMixtureAlgorithm`, and `TableauStorage` traits, as well as word subtraits named `PauliWord`, `FermionWord`, or `LossyPauliWord`. Their -distinctions are expressed by `Word::Site` or by concrete types instead +alphabet distinctions are expressed by `Word::Site` or by concrete types instead of one-alphabet subtraits; the concrete `PauliWord` and `LossyPauliWord` type -names remain available. +names remain available. Note the contrast with `PauliBits`: it is admitted not +as an alphabet subtrait but as a narrow *bit-mutation* capability that genuinely +has multiple implementers and generic consumers. ### Sparse-sum branch staging @@ -738,8 +866,9 @@ independent of the merge strategy (the systems concern). a batch. A packed Pauli word's column is its X and Z plane blocks and the flattened `LossyPauliWord` adds a loss plane; `Phased` is not indexable, so it is not `Columnar` and never appears in a batch. -- `Word`: unaffected — `set` and friends still mutate one value; columns are - built by appending produced keys, not by mutating in place. +- `Word`: unaffected — it is read-only inspection either way. Per-value mutation + happens through `PauliBits` on a single word; columns are built by appending + produced keys, not by mutating in place. - `Policy`: truncation already operates on the whole map and is naturally bulk; it should be expressible as a batch retain (via `KeyColumn::gather`) so it composes with a partitioned or offloaded table. diff --git a/docs/design/word-data-structures.md b/docs/design/word-data-structures.md index 554a3b5a4..6cbb94e04 100644 --- a/docs/design/word-data-structures.md +++ b/docs/design/word-data-structures.md @@ -8,8 +8,10 @@ This document describes concrete data structures for standalone algebraic words. The shared trait design is specified in [`traits-2-configuration-and-hashing.md`](traits-2-configuration-and-hashing.md). -The trait-level `Word` is representation-free but defines the common indexed -product operations: +The trait-level `Word` is representation-free and **read-only**: it defines the +common indexed *inspection* operations, not mutation. Structural mutation lives +on algebra-specific traits (`PauliBits`, `SymplecticColumns`, `PhaseTrack`; see +[`traits-2-configuration-and-hashing.md`](traits-2-configuration-and-hashing.md#pauli-algebra-traits-symplectic-structure-and-phase)): ```rust pub trait Word { @@ -17,8 +19,8 @@ pub trait Word { fn n_sites(&self) -> usize; fn get(&self, index: usize) -> Self::Site; - fn set(&mut self, index: usize, site: Self::Site); fn weight(&self) -> usize; + fn iter(&self) -> impl Iterator; } ``` @@ -46,8 +48,12 @@ the physical mode and creation or annihilation action. `weight()` counts non-identity factors according to the selected site type. For a representation that stores only non-identity factors, it may equal -`n_sites()`. `set()` is the shared structural mutation boundary and must -preserve concrete invariants and invalidate the affected hash components. +`n_sites()`. The structural mutation boundary is `PauliBits::set_x_bit` / +`set_z_bit` (and the `SymplecticColumns` / `PhaseTrack` column primitives), each +of which preserves concrete invariants and lazily invalidates the affected hash +components. `PauliWord` and `LossyPauliWord` implement `PauliBits`; +`PhasedPauliWord` additionally implements `SymplecticColumns + PhaseTrack`, so it +picks up the blanket `Clifford` impl. ## Logical Pauli model From 8566b3d2160f0e85af55ea850283972bd67ed55e Mon Sep 17 00:00:00 2001 From: Roger-luo Date: Thu, 30 Jul 2026 19:45:07 -0400 Subject: [PATCH 06/10] docs(design): make the finalized digest first-class in Indexable Resolve the Step 4 (Indexable / hashing) design gaps from review: - Replace `type BuildHasher` with `Indexable::key_hash() -> u64`, the one load-bearing value the whole design depends on. Scalar `Hash`, the map's hasher, and batch `hash_into` are all defined in terms of it, collapsing three implicit contracts into one explicit method with three equations. - Drop the per-key `BuildHasher`: the direct-digest model leaves the map no hasher to choose, so the internal digest algorithm becomes a private representation parameter of the concrete key (the `H` in PauliWord), on the same footing as the storage array `A`. - Retain HashFinalize but relocate it to a private ppvm-pauli-word helper; the per-algorithm/per-width avalanche fold runs inside key_hash(), and the public contract is only "key_hash() is well distributed" (property-tested). - Add the pass-through storage contract: provided SumStorage backends consume key_hash() directly via an IdentityBuildHasher, guaranteed rather than left to the user, so no second hash re-correlates the folded bits. - Document that SumStorage's `&Word` (not `&mut Word`) closure signatures are the structural half of the key-mutation-invariant enforcement. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/design/tableau-data-structure.md | 6 +- .../traits-2-configuration-and-hashing.md | 113 ++++++++++++++---- docs/design/word-data-structures.md | 35 ++++-- 3 files changed, 116 insertions(+), 38 deletions(-) diff --git a/docs/design/tableau-data-structure.md b/docs/design/tableau-data-structure.md index 2bb430067..2c8284a8e 100644 --- a/docs/design/tableau-data-structure.md +++ b/docs/design/tableau-data-structure.md @@ -348,8 +348,10 @@ pub struct Tableau { ``` The cache fields are private representation choices made by the tableau -author. `Indexable` exposes only the associated build hasher; it does not name -cache types or expose invalidation. +author. `Indexable` exposes only the finalized digest via `key_hash()` — which +here composes the component caches (`combine(xz_hash, phase_hash, loss_hash)`) +and applies the tableau's own finalization fold; it does not name cache types or +expose invalidation. Equality and hashing include logical qubit count, generator order, all logical X/Z bits, phases, and loss state. They exclude: diff --git a/docs/design/traits-2-configuration-and-hashing.md b/docs/design/traits-2-configuration-and-hashing.md index f33a4785a..2dc80976d 100644 --- a/docs/design/traits-2-configuration-and-hashing.md +++ b/docs/design/traits-2-configuration-and-hashing.md @@ -541,6 +541,31 @@ that admits prefetched, SIMD, threaded, and offloaded backends — is specified [Batch execution and the hash-join contract](#batch-execution-and-the-hash-join-contract) below. +#### The pass-through storage contract + +Because a key's [`key_hash()`](#indexable-values) is already the finalized, +avalanche-quality digest, the provided `SumStorage` backends **guarantee** that +their inner map consumes it directly, through an identity pass-through hasher: + +```rust +#[derive(Default, Clone)] +pub struct IdentityHasher(u64); +impl std::hash::Hasher for IdentityHasher { + fn write_u64(&mut self, n: u64) { self.0 = n; } // store the digest + fn write(&mut self, _: &[u8]) { unreachable!() } // keys only write a u64 + fn finish(&self) -> u64 { self.0 } // hand it back verbatim +} +``` + +The shipped `HashStore` / `DashStore` therefore instantiate their `HashMap` / +`DashMap` with `IdentityBuildHasher`, so `finish() == key.key_hash()` and the +digest reaches hashbrown untouched. This is part of the storage contract, not a +user responsibility: the user selects the key's *internal* digest algorithm (the +`H` in `PauliWord`), which governs distribution quality, and never selects +a map hasher — the direct-digest model leaves none to choose. Re-hashing the +digest with a general hasher would be wasted work and could re-correlate the +low bits the key's finalization fold just decorrelated. + The generalized engine is named `Sum`, but the Pauli specialization retains the existing domain-facing `PauliSum` name (a defaulted type alias over `Sum`). This is a new internal generalization, not a requirement to rename Pauli call sites. @@ -569,7 +594,9 @@ have not yet been reduced to a proven common interface. The next design iteration should look for the smallest useful common factor and merge only that factor, rather than assuming that the two complete algorithms are identical. -Every keyed store must use the build hasher associated with its key type. +Every keyed store must consume its key's `key_hash()` digest directly, through +the identity pass-through described in +[The pass-through storage contract](#the-pass-through-storage-contract). ### Compatibility with current names @@ -597,8 +624,9 @@ changed according to whether their underlying responsibility changes: | `PauliSum` | `PauliSum` over generalized `Sum` machinery | Pauli-facing code keeps its established name; `Sum` names the new cross-algebra engine. | | `GeneralizedTableauSum` | `GeneralizedTableauSum` | The classical mixture algorithm remains the same concept. | | `EntryStore`, `VecStorage`, `MapStorage` | unchanged | The proposal uses the existing storage boundary and implementations. | -| `BuildHasher` | associated with `Indexable` | Hasher ownership moves from `Config` to each indexable key type. | -| `HashFinalize` | removed from the shared contract | Concrete keys may finalize or compose hashes privately. | +| `Config::BuildHasher` | removed; `Indexable::key_hash() -> u64` | The direct-digest model leaves the map no hasher to choose; the key's internal algorithm is a private representation parameter, and the finalized digest is exposed as a value. | +| `HashFinalize` | retained, but private to `ppvm-pauli-word` | The per-algorithm/per-width finalization fold still runs inside `key_hash()`; it leaves the algebra-agnostic contract. | +| (new) | `IdentityBuildHasher` + pass-through `SumStorage` contract | The provided storage consumes `key_hash()` directly instead of re-hashing it. | | `PauliStorage` | removed | Packed backing storage becomes private to the concrete word representation. | | (new) | `Columnar`, `KeyColumn`, `KeyBatch`, `TermBatch`, `ACMapBatch` | Structure-of-arrays batch/hash-join contract, kept as a separate trait so `Indexable` stays minimal; see [Batch execution and the hash-join contract](#batch-execution-and-the-hash-join-contract). | @@ -785,7 +813,8 @@ pub trait KeyColumn: Default + Clone { /// Bulk structural hash of the whole column into a parallel hash column. /// Where per-plane SIMD hashing lives, and what feeds the group-prefetch - /// loop; it must agree bit for bit with the scalar `Hash` of each key. + /// loop. Its obligation is now an equation, not a phrase: `out[i]` must + /// equal the `i`-th key's `Indexable::key_hash()`. fn hash_into(&self, out: &mut [u64]); /// Join confirm: compare element `i` against a build-side key after a hash @@ -896,20 +925,38 @@ interchangeable rather than separate code paths. Hash-enabled `Word` values and `Tableau` can both be expensive, mostly-stable map keys. Their hashing contract should be expressed independently of any map. -The common capability is intentionally minimal: +The common capability is intentionally minimal, and it makes the one +load-bearing value — the finalized structural digest — first class: ```rust pub trait Indexable: Clone + Eq + Hash { - type BuildHasher: std::hash::BuildHasher + Clone + Default; + /// The finalized structural digest of this key: avalanche-quality in both + /// the low bits (the hashbrown bucket) and the top 7 (the control tag), + /// so it can be consumed *directly* as the map hash. Contracts: + /// + /// * `Hash for Self` is exactly `state.write_u64(self.key_hash())`; + /// * structurally equal keys return equal digests; and + /// * `KeyColumn::hash_into` reproduces this value bit for bit. + /// + /// This exposes the digest *value*, not the cache mechanics — there is no + /// cache type or invalidation hook in the contract. + fn key_hash(&self) -> u64; } ``` The important points are: -- `BuildHasher` retains the current associated-type name and moves from - `Config` to the key type; -- the build hasher is associated with the key type, not with a configuration - bundle; +- the digest is used **directly** as the map hash — the map does not re-hash it + (see [the pass-through storage contract](#the-pass-through-storage-contract)), + so the value `key_hash()` returns must already be well distributed; +- the *choice of internal hash algorithm* (`fxhash`, `gxhash`, …) is a private + representation parameter of the concrete key (the `H` in `PauliWord`), + not an associated type on `Indexable`. There is no per-key `BuildHasher` on + the contract, because the direct-digest model leaves the map no hasher to + pick; +- avalanche quality is the key's responsibility: a weak-but-fast algorithm on a + short key must apply a finalization fold inside `key_hash()` (see + [Concrete word hashing](#concrete-word-hashing)); - cache layout and invalidation are private representation invariants of the concrete type; and - equality and hashing cover only structural key identity, never cache fields @@ -935,29 +982,47 @@ An indexable value must not be structurally mutated while it is stored as a map key. Cache invalidation makes a value correct for its next insertion or lookup; it cannot make in-place mutation of an existing map key valid. -Structural fields should therefore be private in the new representations. -Mutation must go through operations that invalidate the affected cache, or -through mutation guards that invalidate on completion. +This invariant is not left entirely to discipline: the sparse-sum engine +enforces its structural half through the closure signatures. `SumStorage`'s +`map_insert` and `map_add` hand the closure `&Word` — a *shared* borrow of the +live key — and expect a new `(Word, Coeff)` back, so the propagation kernels +physically cannot mutate a key in place; they clone, mutate the clone through +`PauliBits`, and the engine re-inserts under the updated digest. The `&Word` +(not `&mut Word`) in those signatures is a deliberate safety choice, not an +incidental one. + +Structural fields should nonetheless be private in the new representations, and +any remaining mutation must go through operations that invalidate the affected +cache, or through mutation guards that invalidate on completion. ## Concrete word hashing -Every concrete `Word` owns its `BuildHasher`, private cache representation, -structural hash algorithm, and invalidation logic. Pauli words hash -their X/Z content, lossy words compose Pauli and loss components, and future -fermion words hash their ordered factors. Factor order is part of fermionic -identity. +Every concrete `Word` owns its internal hash algorithm, private cache +representation, and invalidation logic, and produces the finalized digest that +`Indexable::key_hash` returns. Pauli words hash their X/Z content, lossy words +compose Pauli and loss components, and future fermion words hash their ordered +factors. Factor order is part of fermionic identity. The trait layer does not expose packed storage to support hashing. Concrete -implementations hash their private fields and may apply hasher-specific -finalization internally. Detailed layouts and component invalidation rules are -in [`word-data-structures.md`](word-data-structures.md). +implementations hash their private fields and apply a **finalization fold** +internally before caching, so `key_hash()` is avalanche-quality even for a short +key consumed directly as the hashbrown bucket-and-tag. That fold is per-algorithm +and per-width — a couple of `fxhash` rounds on an `[u8; 8]` word leave the low +bits correlated and need `raw ^ (raw >> 32)`, whereas AES-based `gxhash` +avalanches an 8-byte key already and folds nothing. The retained `HashFinalize` +helper that encodes those rules is a **private utility inside `ppvm-pauli-word`**, +not part of the algebra-agnostic `Indexable` contract; the public contract is +only "`key_hash()` is well distributed," checked by a distribution property test +rather than the type system. Detailed layouts and component invalidation rules +are in [`word-data-structures.md`](word-data-structures.md). ## Tableau indexability A tableau may itself be used as a key by a classical-mixture algorithm, so the -concrete tableau implements `Indexable` directly and owns a tableau-specific -hasher and cache representation. This does not imply that a tableau is a -`Word`; they only share the `Indexable` key capability. +concrete tableau implements `Indexable` directly, owning its internal hash +algorithm and cache representation and returning its own finalized +`key_hash()`. This does not imply that a tableau is a `Word`; they only share +the `Indexable` key capability. The tableau's structural hash is composed from its logical X/Z matrix, phase plane, and per-qubit loss plane. It excludes RNG, padding, cache state, and diff --git a/docs/design/word-data-structures.md b/docs/design/word-data-structures.md index 6cbb94e04..0b110e14d 100644 --- a/docs/design/word-data-structures.md +++ b/docs/design/word-data-structures.md @@ -106,9 +106,12 @@ pub struct PauliWord { ``` `A` is an implementation parameter such as `[u8; N]` or `[usize; N]`, and `H` -is the build hasher associated through `Indexable`. Neither is exposed through -`Word`. The implementation validates that `nqubits` fits the arrays and -ignores or canonicalizes unused high bits. +is the private internal digest algorithm (`fxhash`, `gxhash`, …) used to compute +`key_hash()`. `H` is no longer a `BuildHasher` associated on `Indexable` — the +direct-digest model gives the map no hasher to pick — but a representation +parameter of the concrete word, on the same footing as `A`. Neither is exposed +through `Word` or `Indexable`. The implementation validates that `nqubits` fits +the arrays and ignores or canonicalizes unused high bits. The structural identity is: @@ -241,15 +244,23 @@ does not implement `Hash` or `Indexable` and stores no hash mode or cache. ## Hash ownership -Every hash-enabled word implements `Indexable`, associates its build hasher, -and privately owns the fields and algorithm used to cache its structural -hash. Cache representation and invalidation are not exposed through -`Indexable`. - -The shipped indexable words use component `OnceLock` caches. `Hash::hash` -can populate them through `&self`; structural mutators clear affected cells -through `&mut self`. This preserves `Send + Sync` without imposing either -bound on the `Indexable` trait. +Every hash-enabled word implements `Indexable` and privately owns its internal +digest algorithm, the fields used to cache its structural hash, and the +finalization fold that makes `key_hash()` avalanche-quality. Cache +representation, the algorithm choice, and invalidation are not exposed through +`Indexable`, which surfaces only the finalized digest value via `key_hash()`. + +The shipped indexable words use component `OnceLock` caches. `key_hash()` +(and the `Hash` impl, which is just `state.write_u64(self.key_hash())`) can +populate them through `&self`; structural mutators clear affected cells through +`&mut self`. This preserves `Send + Sync` without imposing either bound on the +`Indexable` trait. + +The finalization fold is applied per-algorithm and per-width by the private +`HashFinalize` helper: narrow storage under a weak hasher (`[u8; 8]` + +`fxhash`) folds `raw ^ (raw >> 32)` so the low bits — hashbrown's bucket — +decorrelate, while a strong hasher (`gxhash`) folds nothing. This helper lives +in `ppvm-pauli-word`, not in the algebra-agnostic trait crate. The same component caches back the bulk `hash_into` on a word's key column (see [Key columns](#key-columns-structure-of-arrays-batches)): filling a batch's hash From f89274ac8530b22c8cb1583d9fd9a445ef1408de Mon Sep 17 00:00:00 2001 From: Roger-luo Date: Thu, 30 Jul 2026 20:13:49 -0400 Subject: [PATCH 07/10] docs(design): re-derive the map as a graded algebra over C[W] MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Recast the storage/ACMap layer from the algebraic structure (Step 5): - Replace the flat ACMapBase/Iter/AddAssign/Insert/Retain/Consume split with layers graded by algebraic strength over the free module C[W]: L0 Support, L1 Accumulate+Reduce, L2 Scale, L3 Pair, L4 Multiply. The minimum is L0+L1 (a finitely-supported W->C you accumulate and reduce). Each layer names a distinct algebraic property and consumer. - Add L4 Multiply (the group-algebra product / convolution), bounded on ComplexCoefficient since the Pauli product injects phase. - Truncation leaves the algebra: it breaks module exactness, so it is a non-algebraic Retain capability consumed by Policy, not part of ACMap. - Make reduce() first-class: exact-zero pruning is canonicalization to finite-support form, run only at finalize after all coefficients for a word are accumulated (never inline) — correctness by construction, and a bulk stream-compaction each backend specializes. - Design columnar from day one: two backends (HashStore AoS, ColumnStore SoA) over the same graded traits, with four no-AoS-leak rules so the SIMD/GPU store is expressible immediately. Each layer maps to a bulk data-parallel primitive (scale=vector map, reduce=prefix-sum compaction, accumulate=hash-join build, pair=probe, multiply=outer product). - Unify map_add/map_insert into one SumStorage::apply(producer): Clifford, rotation, and multiply are TermProducers feeding accumulate; the Vec staging leak is gone, and the Step 6 batch types become the columnar spelling of Accumulate/Pair. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../traits-2-configuration-and-hashing.md | 338 ++++++++++++------ 1 file changed, 235 insertions(+), 103 deletions(-) diff --git a/docs/design/traits-2-configuration-and-hashing.md b/docs/design/traits-2-configuration-and-hashing.md index 2dc80976d..28b97ae60 100644 --- a/docs/design/traits-2-configuration-and-hashing.md +++ b/docs/design/traits-2-configuration-and-hashing.md @@ -441,10 +441,16 @@ where fn truncate(&self, map: &mut M) where - M: ACMap; + M: Retain; } ``` +`truncate` bounds on `Retain`, not the map algebra: dropping terms still in the +support breaks module exactness, so it is the one non-algebraic map operation +and lives on its own capability that `Policy` — not the algebra — consumes. +`Retain` is `fn retain(&mut self, keep: impl Fn(&W, &C) -> bool)`, implemented by +both storage backends. + `Policy` does not require `Copy`. The bound bought nothing — policies are used through `&self` — and would forbid a stateful policy such as a per-region threshold vector or an adaptive budget; requiring it would also contradict this @@ -465,7 +471,7 @@ impl Policy for CoefficientThreshold 0 } - fn truncate>(&self, map: &mut M) { + fn truncate>(&self, map: &mut M) { map.retain(|_word, coeff| coeff.magnitude() >= self.threshold); } } @@ -476,16 +482,151 @@ removes the current split where the `Strategy` trait lives in `ppvm-traits` but its concrete strategies live in `ppvm-pauli-sum`; the policy is not an algorithm-agnostic `ppvm-traits-2` concern. -`ACMap` remains the name of the associative coefficient-map capability already -implemented by `HashMap`, `IndexMap`, and `DashMap`. Its generic signature can -be simplified after `PauliStorage` and the separately supplied build hasher are -removed, but coefficient accumulation, iteration, insertion, retention, and -consumption are the same concept. `ACMap` moves with the sparse-sum engine (the -existing `ppvm-pauli-sum` initially) rather than being renamed or kept in the -algorithm-agnostic trait crate. Its existing capability names—such as -`ACMapBase`, `ACMapIter`, `ACMapAddAssign`, `ACMapInsert`, `ACMapRetain`, and -`ACMapConsume`—should also remain unless implementation work shows that two -capabilities should actually be merged or split. +#### The map is a graded algebra over `C[W]` + +The associative coefficient map — implemented today by `HashMap`, `IndexMap`, +and `DashMap` — is, algebraically, the **free `C`-module on the set of words**: +a finitely-supported function `W ⇀ C`, an element of `C[W]`. Every operation it +must support is a module (or, at the top, ring) operation, so its capabilities +are **graded by algebraic strength** rather than named ad hoc. This replaces the +flat `ACMapBase` / `ACMapIter` / `ACMapAddAssign` / `ACMapInsert` / +`ACMapRetain` / `ACMapConsume` split — which the design's own admission rule +would reject as six grandfathered names — with layers, each justified by a +distinct algebraic property *and* a distinct consumer: + +| Layer | Algebra | Trait | Justifying consumer | +| --- | --- | --- | --- | +| L0 | finite partial function `W ⇀ C` | `Support` | everything; read-out / export | +| L1 | abelian-monoid formation + canonical support | `Accumulate` (`accumulate_batch` + `reduce`) | forming any linear combination | +| L2 | the `C`-module action | `Scale` | normalization, global factors | +| L3 | the bilinear form | `Pair` (`probe_batch`, `overlap`) | overlap / expectation read-out | +| L4 | the ring (group-algebra) product | `Multiply` | operator composition, squaring an observable | + +The **minimum** is L0 + L1: a finitely-supported `W ⇀ C` you can accumulate +into and reduce. That *is* the algebraic essence of a sparse sum. Truncation is +deliberately **absent** from this table: dropping terms that are still in the +support is an approximation that breaks module exactness, so it is not an +algebraic operation — it belongs to `Policy` (a `Retain` capability), sitting +outside the algebra. That is why truncation was always awkward to place. + +```rust +/// L0 — the container. Note: no `&mut (W, C)` and no `&mut [C]` slot access is +/// exposed, so a columnar (structure-of-arrays) backend is expressible. `iter` +/// is read-only export; a SoA backend synthesizes the pairs from its columns. +pub trait Support { + type Word: Word + Indexable; + type Coeff: Coefficient; + fn len(&self) -> usize; + fn get(&self, key: &Self::Word) -> Option; + fn iter(&self) -> impl Iterator; +} + +/// L1 — the module core: form linear combinations, then canonicalize. +pub trait Accumulate: Support { + /// Build side of the hash join: merge a produced batch, accumulating onto an + /// existing key or inserting a new one. Columnar in; the scalar + /// `accumulate(w, c)` is provided sugar over a batch of one. + fn accumulate_batch(&mut self, terms: &TermBatch); + + /// Canonicalize to reduced finite-support form: drop every key whose + /// coefficient `is_zero()`. First-class and run **only here** — see below. + fn reduce(&mut self); +} + +/// L2 — the C-module action: a pure elementwise map over the coefficients. +pub trait Scale: Support { + fn scale(&mut self, s: &Self::Coeff); // ∀ w. c_w *= s +} + +/// L3 — the bilinear form: the read side of the hash join. +pub trait Pair: Support { + fn probe_batch(&self, keys: &KeyBatch, out: &mut [Option]); + fn overlap(&self, other: &Self) -> Self::Coeff; +} + +/// L4 — the ring product. The Pauli product injects powers of `i`, so the +/// coefficient must absorb phase: `Multiply` is bounded on `ComplexCoefficient`. +pub trait Multiply: Accumulate +where + Self::Coeff: ComplexCoefficient, +{ + fn multiply_into(&self, other: &Self, acc: &mut Self); +} +``` + +#### `reduce()` is first-class, and runs only at finalize + +Reduce (drop zero coefficients) is **not** an inline check during accumulation. +A coefficient `c1 + c2 + c3` is one value; the fact that a partial sum `c1 + c2` +transiently hits zero during a merge is not the element reaching zero. Pruning +mid-accumulation would delete a key that a later contribution re-creates — churn +at best, and unsafe if it mutates the map during the traversal that is still +producing those contributions. So canonicalization runs **once, after all +coefficients for every word are accumulated**. Making `reduce` a named +operation guarantees that by construction, and it earns its place three ways: + +- **Correctness** — there is no inline drop to get wrong (the caution above). +- **It is a bulk primitive with a backend-specific implementation** — a scalar + `retain(|_, c| !c.is_zero())` on `HashStore`, but a prefix-sum **stream + compaction** kernel on a GPU column store. Folded into `consume`/swap it could + not have a bulk form. +- **It amortizes** — several `accumulate_batch` calls (or a `multiply_into` + outer product) can precede a single `reduce`, instead of canonicalizing after + every step. + +#### Columnar from day one: two backends, no AoS leak + +The whole point of this refactor is to unblock SIMD / GPU execution, so the +layers are designed to admit a **structure-of-arrays** accumulator from the +start — not as a later retrofit. Two backends must both be expressible against +the same traits: + +- **`HashStore`** (array-of-structs, `hashbrown`): implements every layer with + scalar kernels. `accumulate_batch` scatters the batch through scalar inserts; + `reduce` is a `retain`; `scale` is a scalar loop. Correct and fast for the CPU + scalar path, but a dead end for the GPU. +- **`ColumnStore`** (structure-of-arrays, for SIMD / GPU): coefficients in one + contiguous column, keys in plane blocks. `accumulate_batch` is a + radix-partitioned, group-prefetched hash-join build; `scale` is one vectorized + `*=` over the coefficient column; `reduce` is a prefix-sum compaction; + `probe_batch` is a prefetched probe with coalesced gathers. + +Both are expressible **only because no trait signature leaks the array-of-structs +layout**. The rules the whole storage contract holds to: + +1. coefficients are a contiguous column and keys are plane blocks (SoA) — so L2 + Scale, L1 Reduce, and L3 Pair vectorize and GPU loads coalesce; +2. term I/O is a columnar `TermBatch` / `KeyBatch` (see + [The batch contract](#the-batch-contract)), never a scalar per-element + callback — so `HashStore` accepts a batch and processes it scalar internally + while `ColumnStore` vectorizes it, from the *same* call; +3. the mutating operations are whole-map (`scale`, `reduce`) or batch + (`accumulate_batch`), never per-slot; and +4. **no signature exposes `&mut (W, C)` or `&mut [C]`** — the one rule that keeps + the AoS layout from becoming load-bearing and foreclosing `ColumnStore`. + +The columnar term types (`TermBatch`, `KeyBatch`, `KeyColumn`) that these layers +consume are the same ones specified in +[Batch execution and the hash-join contract](#batch-execution-and-the-hash-join-contract); +that section is the columnar spelling of L1 and L3, and the graded layers here +are where it plugs into the algebra. + +#### Every gate is a producer feeding `accumulate` + +`map_add` and `map_insert` are not two different map operations; they are one +(`accumulate`) fed by two different **producers**. Clifford, rotation, and +multiply all reduce to "produce `(w, c)` terms, `accumulate_batch`, `reduce`": + +| Algebra op | Producer | Term shape | +| --- | --- | --- | +| Clifford (H/S/CNOT) | pushforward along a Pauli bijection `w ↦ φ(w)` | one term per input (injective, no collision) | +| Rotation / noise | extend-by-linearity `w ↦ cos·w + sin·w'` | a small fan-out per input (branch) | +| Multiply (L4) | outer product over two operands' support | one term per `(v, w)` pair | + +The producer difference lives entirely on the **term-production side** — the +`TermSink` of [The batch contract](#the-batch-contract) — not in the map, which +only ever accumulates produced batches. That is what dissolves the old +`map_add` / `map_insert` split and its `Vec<(W, C)>` staging leak. A `SumStorage` is a new abstraction extracted from the fields currently owned directly by `PauliSum`: its maps and reusable workspace. It is an actual value, @@ -495,49 +636,38 @@ not a marker configuration: pub trait SumStorage: Clone { type Word: Word + Indexable; type Coeff: Coefficient; - type Map: ACMap; + type Map: Accumulate; fn data(&self) -> &Self::Map; fn data_mut(&mut self) -> &mut Self::Map; - fn map_insert(&mut self, f: F) - where - F: Fn(&Self::Word, &mut Self::Coeff) -> Option<(Self::Word, Self::Coeff)>; + /// Apply a term producer — a gate — into the map. The producer emits its + /// terms into a `TermSink`; the storage owns whether that sink is a scratch + /// `Vec`, a columnar `TermBatch`, or per-thread buffers, hands the staged + /// batch to `Accumulate::accumulate_batch`, then `reduce`s. This one method + /// subsumes the former `map_add` / `map_insert` / `map_insert_multiple`: + /// they differed only in their producer, not in the map operation. + fn apply>(&mut self, producer: P); - fn map_insert_multiple(&mut self, f: F) - where - F: Fn(&Self::Word, &mut Self::Coeff) -> Option>; - - fn map_add(&mut self, f: F) - where - F: Fn(&Self::Word, &Self::Coeff) -> (Self::Word, Self::Coeff); - - fn consume(&mut self); + fn reduce(&mut self); } ``` -The exact closure bounds and support for multiple produced terms remain an -implementation detail for the prototype. The important boundary is that the -trait preserves the current semantic operation names without exposing physical -auxiliary maps or scratch buffers. - -`SumStorage` owns the semantic whole-map operations and its reusable workspace. -It delegates to the lower-level `ACMap` batch kernels without restoring the -removed map-to-map `ACMapInsert::map_insert` method: - -```text -SumStorage::map_insert -> ACMapInsert::map_insert_vec -SumStorage::map_insert_multiple -> ACMapInsert::map_insert_multiple -SumStorage::map_add -> ACMapAddAssign::map_add_assign -``` - -This boundary is compatible with -`refactor/shrink-internal-trait-surface`: the higher-level sparse-sum operation -remains, while the dead low-level primitive stays removed. - -Those `ACMap` kernels are the scalar spelling of a bulk operation. The -batch-first, structure-of-arrays layout the kernels consume on hot paths — and -that admits prefetched, SIMD, threaded, and offloaded backends — is specified in +`SumStorage` owns the reusable workspace and the sink. It no longer exposes a +family of closure methods returning `Option>` — that return type +leaked the scratch-`Vec` staging mechanism into the contract. Instead a gate is +a `TermProducer` that pushes into a `TermSink`, so the staging representation +(auxiliary map, scratch vector, columnar batch, per-thread buffers) is entirely +the storage's private choice, and the same gate kernel drives the scalar +`HashStore` and the columnar `ColumnStore` unchanged. `apply` desugars to +`accumulate_batch` + `reduce` on the underlying `Accumulate` map; the +whole-map rewrite (Clifford) and the branching expansion (rotation) are two +`TermProducer`s, per the producer table above. + +Because the batch a producer stages is exactly the structure-of-arrays layout +prefetched, SIMD, threaded, and offloaded backends consume, the execution +schedule for that batch — group prefetching, radix partitioning, device +offload — is specified in [Batch execution and the hash-join contract](#batch-execution-and-the-hash-join-contract) below. @@ -618,8 +748,8 @@ changed according to whether their underlying responsibility changes: | `Strategy` | `Policy` | Intentional terminology change requested for this redesign; the `Copy` bound is dropped. | | `Coefficient::cutoff` | removed | The truncation predicate moves to `Policy`; `Coefficient::magnitude` exposes only the value property the policy thresholds. | | `Coefficient::sin_cos` | `Angle` | The rotation angle becomes a separate domain, defaulting to the coefficient type. | -| `ACMap` | `ACMap` | The associative coefficient map has the same role. | -| `PauliSum::data`, `map_insert`, `map_add` | same method names on `SumStorage` | These semantic operations already match the proposed boundary. | +| `ACMap` (`ACMapBase`/`Iter`/`AddAssign`/`Insert`/`Retain`/`Consume`) | graded `Support` / `Accumulate` / `Scale` / `Pair` / `Multiply` | The six flat names are replaced by algebra layers over `C[W]`; `Retain` (truncation) leaves the algebra for `Policy`. | +| `PauliSum::map_insert`, `map_add` | one `SumStorage::apply(producer)` → `Accumulate::accumulate_batch` + `reduce` | They differed only in their producer, not the map op; the `Vec<(W,C)>` staging leak is removed. | | `PauliSum` map pair and `scratch` fields | `SumStorage` | A new abstraction is extracted from currently unnamed storage state. | | `PauliSum` | `PauliSum` over generalized `Sum` machinery | Pauli-facing code keeps its established name; `Sum` names the new cross-algebra engine. | | `GeneralizedTableauSum` | `GeneralizedTableauSum` | The classical mixture algorithm remains the same concept. | @@ -628,7 +758,8 @@ changed according to whether their underlying responsibility changes: | `HashFinalize` | retained, but private to `ppvm-pauli-word` | The per-algorithm/per-width finalization fold still runs inside `key_hash()`; it leaves the algebra-agnostic contract. | | (new) | `IdentityBuildHasher` + pass-through `SumStorage` contract | The provided storage consumes `key_hash()` directly instead of re-hashing it. | | `PauliStorage` | removed | Packed backing storage becomes private to the concrete word representation. | -| (new) | `Columnar`, `KeyColumn`, `KeyBatch`, `TermBatch`, `ACMapBatch` | Structure-of-arrays batch/hash-join contract, kept as a separate trait so `Indexable` stays minimal; see [Batch execution and the hash-join contract](#batch-execution-and-the-hash-join-contract). | +| (new) | `Columnar`, `KeyColumn`, `KeyBatch`, `TermBatch`, `TermSink`/`TermProducer` | Structure-of-arrays term types; the columnar spelling of `Accumulate`/`Pair`. See [Batch execution and the hash-join contract](#batch-execution-and-the-hash-join-contract). | +| (new) | `HashStore` (AoS) and `ColumnStore` (SoA) backends | Two `SumStorage` backends over the same graded traits — the SoA one is expressible from day one because no signature leaks AoS. | Names such as `Word`, `Indexable`, `SumStorage`, and `Sum` are therefore new because they denote abstractions that do not @@ -654,10 +785,13 @@ This rule keeps: implemented by `PhasedPauliWord` and `Tableau`; - `StabilizerFrame`, the role-exclusive tableau operations; - gate and noise traits, implemented across propagation and tableau backends; -- `SumStorage` and `ACMap`, implemented by genuinely different storage engines - and collections; +- `SumStorage`, and the graded map layers `Support` / `Accumulate` / `Scale` / + `Pair` / `Multiply`, each admitted by a distinct algebraic property *and* a + distinct consumer, and implemented by the scalar `HashStore` and the columnar + `ColumnStore`; - `EntryStore`, already implemented by `VecStorage` and `MapStorage`; and -- `Policy`, implemented by independent capacity and truncation behaviors. +- `Policy`, implemented by independent capacity and truncation behaviors, and + consuming the non-algebraic `Retain` capability rather than the map algebra. It rejects the removed global `Config`, `PauliSumAlgorithm`, `TableauMixtureAlgorithm`, and `TableauStorage` traits, as well as word @@ -699,12 +833,13 @@ whole-map rewrite: active map -> auxiliary map -> swap in-place branching: active map + scratch buffer -> merge into active map ``` -Neither physical mechanism is part of the public storage contract. A default -`SumStorage` implementation may privately contain both maps and the reusable -vector, matching the current `PauliSum` layout. A simpler backend may implement -both `map_insert` and `map_add` using only an auxiliary map; another backend -may use per-thread buffers. Whether retaining both mechanisms is worthwhile is -a benchmark decision, not a type-system requirement. +Neither physical mechanism is part of the public storage contract. Both are the +`TermSink` a `TermProducer` pushes into: a default `SumStorage` may privately +hold both an auxiliary map and the reusable vector, matching the current +`PauliSum` layout; a simpler backend may sink everything through one auxiliary +map; a columnar backend sinks into a `TermBatch`; another may use per-thread +buffers. Whether retaining both mechanisms is worthwhile is a benchmark +decision, not a type-system requirement. ## Batch execution and the hash-join contract @@ -732,12 +867,13 @@ does not fit in cache, so each probe is a random access that misses to main memory, and the probe phase is bound by memory latency rather than by arithmetic. -The `SumStorage` closures (`map_insert`, `map_add`) and the `ACMap` kernels they -delegate to still describe *what* the merge computes. But a closure applied to -one `(key, coeff)` at a time exposes no batch to prefetch, no homogeneous run to -vectorize, no partition to distribute across threads, and no bulk kernel to hand -to a device. Whatever the eventual implementation, the *layout* the kernels -consume must be a batch, or those backends cannot be written against it. +`SumStorage::apply` and the `Accumulate`/`Pair` layers describe *what* the merge +computes. But a producer that emitted one `(key, coeff)` at a time would expose +no batch to prefetch, no homogeneous run to vectorize, no partition to distribute +across threads, and no bulk kernel to hand to a device. That is why a +`TermProducer` stages into a `TermSink` and the map consumes an +`accumulate_batch`: the *layout* the merge consumes is a batch, so those backends +can be written against it. ### Prefetching the probe phase @@ -832,28 +968,23 @@ pub trait KeyColumn: Default + Clone { } ``` -The bulk map operations then consume columns. They are the batch spelling of the -existing `ACMap` kernels, added alongside them rather than replacing the -semantic `SumStorage` methods: +The bulk map operations that consume these columns are not a new trait: they are +exactly the columnar methods of the graded algebra layers — +[`Accumulate::accumulate_batch`](#the-map-is-a-graded-algebra-over-cw) is the +build side and [`Pair::probe_batch`](#the-map-is-a-graded-algebra-over-cw) is the +read side. Restated here at the column level for reference: ```rust -pub trait ACMapBatch -where - W: Word + Columnar, - C: Coefficient, -{ - /// Merge a batch into the sum: for each term, accumulate onto an existing - /// key or insert a new one. The build/probe-with-group-by of a hash join. - /// The implementation owns whether it runs scalar, group- or - /// pipeline-prefetched, SIMD-vectorized, hash-partitioned across threads, - /// or offloaded; the contract fixes only the result and the layout. - fn upsert_batch(&mut self, batch: &TermBatch); - - /// Read-only probe of a key column, for the overlap and expectation paths. - /// Takes a `KeyBatch` so the precomputed hash column drives the prefetch - /// with no coefficient column in the working set. - fn probe_batch(&self, keys: &KeyBatch, out: &mut [Option]); -} +// Accumulate::accumulate_batch — the build/probe-with-group-by of a hash join. +// The implementation owns whether it runs scalar, group- or pipeline-prefetched, +// SIMD-vectorized, hash-partitioned across threads, or offloaded; the contract +// fixes only the result and the columnar layout. +fn accumulate_batch(&mut self, batch: &TermBatch); + +// Pair::probe_batch — read-only probe of a key column, for the overlap and +// expectation paths. Takes a `KeyBatch` so the precomputed hash column drives +// the prefetch with no coefficient column in the working set. +fn probe_batch(&self, keys: &KeyBatch, out: &mut [Option]); ``` Three points make this layout the performant one rather than a cosmetic @@ -873,17 +1004,17 @@ reshuffle: padding live in [`word-data-structures.md`](word-data-structures.md); they are never visible through `KeyColumn`. -`upsert_batch` is where the branch-staging merge lands: the reusable scratch +`accumulate_batch` is where the branch-staging merge lands: the reusable scratch buffer of the previous section becomes a `TermBatch` — the probe side of the -join — and the `SumStorage` merge desugars to it. The scalar `SumStorage` -closures remain the semantic surface for a naive backend, but generic hot paths -build a `TermBatch` and hand it to the batch kernel. +join. A naive backend may still have its `TermSink` collect into a scalar `Vec` +and loop, but generic hot paths build a columnar `TermBatch` and hand it to +`accumulate_batch`. Gate and rotation traits change accordingly, and this is the interface change users feel first: term *production* is separated from term *insertion*. A -rotation appends produced terms into a `TermBatch` — filling the key column and -the coefficient column as it goes — instead of mutating the map through a -callback. That decoupling, plus the columnar layout, is what lets the produced +`TermProducer` (a rotation, a Clifford re-key, a multiply operand) appends +produced terms into a `TermSink` — filling the key column and the coefficient +column as it goes — instead of mutating the map through a callback. That decoupling, plus the columnar layout, is what lets the produced batch be prefetched, vectorized, partitioned, or shipped to a device before it ever touches the table, and it keeps the propagation rule (the physics) independent of the merge strategy (the systems concern). @@ -901,9 +1032,10 @@ independent of the merge strategy (the systems concern). - `Policy`: truncation already operates on the whole map and is naturally bulk; it should be expressible as a batch retain (via `KeyColumn::gather`) so it composes with a partitioned or offloaded table. -- `SumStorage`: its staging fields become the join's probe-side buffer and its - merge desugars to `upsert_batch`. Whether it keeps both an auxiliary map and a - scratch buffer stays a backend decision, unchanged from above. +- `SumStorage`: its staging fields become the join's probe-side buffer (the + `TermSink`) and its `apply` desugars to `accumulate_batch` + `reduce`. Whether + it keeps both an auxiliary map and a scratch buffer stays a backend decision, + unchanged from above. ### Parallel and offloaded backends @@ -912,7 +1044,7 @@ without further interface changes. A partitioned hash join radix-partitions both sides by the high bits of the key hash so each partition is a disjoint sub-table; partitions then merge independently — one per thread, or one per GPU block — with no cross-partition synchronization. The batch contract is precisely -what a partitioner consumes: `upsert_batch` may internally `gather` a +what a partitioner consumes: `accumulate_batch` may internally `gather` a `TermBatch` into per-partition batches and run them concurrently, and a device backend may copy a `TermBatch` across the host/device boundary and run the probe as a kernel. None of this is visible in the contract; all of it is precluded by @@ -983,13 +1115,13 @@ key. Cache invalidation makes a value correct for its next insertion or lookup; it cannot make in-place mutation of an existing map key valid. This invariant is not left entirely to discipline: the sparse-sum engine -enforces its structural half through the closure signatures. `SumStorage`'s -`map_insert` and `map_add` hand the closure `&Word` — a *shared* borrow of the -live key — and expect a new `(Word, Coeff)` back, so the propagation kernels -physically cannot mutate a key in place; they clone, mutate the clone through -`PauliBits`, and the engine re-inserts under the updated digest. The `&Word` -(not `&mut Word`) in those signatures is a deliberate safety choice, not an -incidental one. +enforces its structural half through the producer interface. A `TermProducer` +reads live keys through a *shared* `&Word` borrow and *emits* new `(Word, Coeff)` +terms into the `TermSink`; it is never handed `&mut Word` for a stored key. So +the propagation kernels physically cannot mutate a key in place — they clone, +mutate the clone through `PauliBits`, and the engine accumulates it under the +updated digest. That the producer only ever *reads* keys and *produces* terms is +a deliberate safety choice, not an incidental one. Structural fields should nonetheless be private in the new representations, and any remaining mutation must go through operations that invalidate the affected From 89f89a3f1b139882ce6c923257d068af7fffd519 Mon Sep 17 00:00:00 2001 From: Roger-luo Date: Thu, 30 Jul 2026 20:40:56 -0400 Subject: [PATCH 08/10] docs(design): inverse tableau, RNG injection, and one algebra for both Resolve Step 7 (tableau) and unify the tableau side with Pauli propagation: - Relax the graded algebra from C[Word] to C[Key] over any Indexable key. The Word bound leaked Pauli-specificity; C[K] is the free module over any index set. Pauli propagation re-adds Word/PauliBits on its own methods. This makes PauliSum and the stabilizer mixture the SAME Sum engine: PauliSum = Sum>, TableauMixture = Sum> (replacing GeneralizedTableauSum). A gate is a TermProducer that re-keys each key via the key's own Clifford impl. This is the "smallest common factor" the prior iteration deferred. - Add TermProducer/RekeyProducer with the zero-overhead guarantees the hot loop needs: generic (never dyn), #[inline] produce, pre-sized sink, so the gate body folds into the accumulate loop with no per-term allocation. - Add KeyProduct (the L4 key-product capability) and Multiply's bound. - Commit to the inverse tableau: measurement becomes a row read in the gate's column-major orientation, eliminating per-measurement transpose; the inverse is a private representation of Tableau (traits unchanged). - Hoist the RNG out of Tableau: inject at measure(&mut, q, rng); clone is pure data, so mixture branches get independent streams (fixes the correlated-sampling bug); Tableau is trivially Send + Sync. - StabilizerFrame holds frame primitives (pivot/row_multiply/canonicalize); Measure is the public algorithm built on them. - State that bulk hash/compare requires canonical orientation, enforced by the transposition guard's &mut borrow; budget the non-square transpose's square-block padding + scratch rather than assuming it away. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/design/tableau-data-structure.md | 90 +++++++--- .../traits-2-configuration-and-hashing.md | 169 ++++++++++++++---- 2 files changed, 203 insertions(+), 56 deletions(-) diff --git a/docs/design/tableau-data-structure.md b/docs/design/tableau-data-structure.md index 2c8284a8e..6645a4e44 100644 --- a/docs/design/tableau-data-structure.md +++ b/docs/design/tableau-data-structure.md @@ -37,8 +37,11 @@ orientation. - Standardizing a general-purpose matrix trait in `ppvm-traits-2`. - Maintaining both row-major and column-major copies before benchmarks show that the memory and synchronization cost is worthwhile. -- Deciding in this document whether PPVM should store a forward or inverse - tableau. Orientation and inversion are independent design choices. + +Orientation (column- vs row-major) and inversion (forward vs inverse) are +independent choices; this document now fixes both — column-major storage of the +**inverse** tableau (see [Inverse tableau](#inverse-tableau)) — because that +pairing is what lets gates and measurement share one orientation. ## Logical model @@ -105,9 +108,9 @@ capability of the same tableau type, not a `LossyTableau` variant or a ```rust pub struct Tableau { - data: TableauData, + data: TableauData, // the *inverse* tableau (see below), column-major lost_count: usize, - // hash caches and RNG + // hash caches only — NO rng field; see "Randomness is injected" } pub struct GeneralizedTableau { @@ -175,9 +178,15 @@ assuming the same total performance from layout alone. ## Temporary transposition -Row multiplication and elimination need long generator rows and therefore -prefer the opposite orientation. The initial design should transpose the X/Z -quadrants temporarily instead of storing two permanent copies: +With the inverse tableau, ordinary measurement is a row read in the canonical +column-major orientation, so it does **not** transpose — the per-measurement +thrash that motivated a permanent second copy is gone. What still prefers the +opposite orientation is *bulk* row multiplication and elimination (canonical +form, batched frame work). Those receive a temporary transpose of the X/Z +quadrants rather than a permanently stored second copy. Because a non-square +`2n × n` bit-matrix transpose is not a swap, the guard is expected to work over +square, padded blocks (as Stim does) and may use a scratch buffer; that padding +and scratch are budgeted here rather than assumed away: ```rust pub enum Orientation { @@ -247,10 +256,10 @@ The Rust behavioral boundary is one loss-aware trait: ```rust pub trait Measure { - fn measure(&mut self, qubit: usize) -> Option; + fn measure(&mut self, qubit: usize, rng: &mut R) -> Option; - fn measure_many(&mut self, targets: &[usize]) -> Vec> { - targets.iter().map(|&q| self.measure(q)).collect() + fn measure_many(&mut self, targets: &[usize], rng: &mut R) -> Vec> { + targets.iter().map(|&q| self.measure(q, rng)).collect() } } ``` @@ -263,6 +272,31 @@ represents a lost qubit. This is the existing core representation used by boolean Clifford measurement routine becomes a private helper called only after the public implementation has established that the target is present. +### Randomness is injected, not stored + +`measure` takes `rng: &mut R`; the tableau owns **no** RNG field. Storing the +RNG inside the tableau is a correctness hazard: classical-mixture branching +clones a tableau into two branches, and a cloned RNG would make both branches +draw the *same* random outcomes — a silent sampling bias. With randomness +injected, `clone` is a pure data copy (no stream to duplicate) and the caller +derives an independent stream per branch (`SmallRng::from_rng(&mut *rng)`). It +also makes `Tableau` trivially `Send + Sync` and removes RNG from the +structural-hash exclusion list — there is nothing to exclude. + +### Inverse tableau + +`Tableau` stores the **inverse** tableau. In a forward tableau, measurement must +scan a column to find an anticommuting generator and then eliminate — row work +in the opposite orientation from the column-major gate path, forcing a transpose +per measurement. The inverse tableau makes the anticommutation structure a +**row read in the same column-major orientation gates already use**, so gates +and measurement no longer fight over layout and the per-measurement transpose +disappears. The inverse is a *private representation choice* of `Tableau`: the +`SymplecticColumns` / `PhaseTrack` impls encode the inverse update rule and +`StabilizerFrame` reads the inverse, but the behavioral traits above them +(`Clifford`, `Measure`) are unchanged. Reference-frame sampling composes on top +for the sampling workload. + The common trait and result type do not imply a common measurement algorithm: - `Tableau::measure` checks the loss plane and then uses the pure Clifford @@ -343,7 +377,7 @@ pub struct Tableau { xz_hash: OnceLock, phase_hash: OnceLock, loss_hash: OnceLock, - rng: SmallRng, + // no rng — randomness is injected at `measure` (see above) } ``` @@ -353,14 +387,21 @@ here composes the component caches (`combine(xz_hash, phase_hash, loss_hash)`) and applies the tableau's own finalization fold; it does not name cache types or expose invalidation. +Because equality and hashing compare *canonical ranges* in bulk (zeroed +padding), they require the tableau to be in its **canonical column-major +orientation**. This is not left to discipline: the transposition guard holds +`&mut self` for its whole lifetime, so the borrow checker forbids any shared +`&self` hash or comparison while the tableau is transposed. Hashing therefore +only ever observes the canonical orientation. + Equality and hashing include logical qubit count, generator order, all logical X/Z bits, phases, and loss state. They exclude: -- RNG state; - allocation capacity; - alignment padding; - cache values and validity flags; and - current physical orientation. + (There is no RNG state to exclude — it is not stored.) The component invalidation rules are: @@ -372,7 +413,6 @@ The component invalidation rules are: | Row multiplication | invalidate | invalidate if changed | preserve | | Toggle a loss bit | preserve | preserve | invalidate | | Physical transpose | preserve | preserve | preserve | -| RNG update | preserve | preserve | preserve | The current `ppvm-tableau-sum` split between `word_fingerprint` and `phase_loss_hash` is evidence that component hashing matters. The new tableau @@ -381,7 +421,10 @@ owns X/Z, phase, and loss components directly. ## Cloning and mixture use Classical-mixture branching can clone tableaus frequently. Contiguous backing -storage makes cloning a bulk memory copy. A clone may copy valid hash caches +storage makes cloning a bulk memory copy. Because the RNG is not stored (it is +injected at `measure`), the clone is *pure data*: there is no random stream to +duplicate, so branches cannot end up statistically correlated — the caller +derives an independent stream per branch. A clone may copy valid hash caches because it initially has identical logical contents. Subsequent mutation of the branch invalidates only the affected components. @@ -401,9 +444,10 @@ measurement. Temporary transposition makes elimination and row products contiguous when required. This layout should be evaluated separately from higher-level sampling -algorithms. Stim also uses an inverse tableau and reference-frame sampling; -those algorithmic choices are not consequences of column-major storage and do -not belong in the PPVM trait system. +algorithms. PPVM follows Stim in storing the inverse tableau (see +[Inverse tableau](#inverse-tableau)), and reference-frame sampling composes on +top; both are internal choices of the concrete `Tableau` and do not surface in +the PPVM trait system. ## Prototype validation @@ -432,6 +476,12 @@ The prototype should include: 2. Should phases occupy one or two bits per generator in the tableau model? 3. Which operations should receive a transposition guard versus performing column-strided work directly? -4. Does a dual-orientation representation outperform temporary transposition - for PPVM's measurement-heavy workloads? -5. Should PPVM ultimately store a forward or inverse tableau? +4. Does a dual-orientation representation still add anything now that the inverse + tableau keeps measurement in the gate orientation, or is temporary + transposition needed only for rare bulk elimination? + +**Resolved:** PPVM stores the **inverse** tableau (see +[Inverse tableau](#inverse-tableau)). This keeps measurement a row read in the +same column-major orientation gates use, so the per-measurement transpose is +gone and the guard is reserved for rare bulk row work. Reference-frame sampling +composes on top for measurement-dominated sampling workloads. diff --git a/docs/design/traits-2-configuration-and-hashing.md b/docs/design/traits-2-configuration-and-hashing.md index 28b97ae60..913405bb3 100644 --- a/docs/design/traits-2-configuration-and-hashing.md +++ b/docs/design/traits-2-configuration-and-hashing.md @@ -310,13 +310,19 @@ impl Clifford for T { The role-*exclusive* operations — those that interpret the rows as a symplectic basis rather than as independent operators — do not belong in the shared tower. -They are a tableau-only trait a word never implements: +They are a tableau-only trait a word never implements. `StabilizerFrame` holds +the frame **primitives**, not `measure` itself: `measure` is the public +`Measure` trait, and its two algorithms (`Tableau` pure-Clifford, +`GeneralizedTableau` coefficient-aware `O(n^2)`) are built *on* these primitives: ```rust pub trait StabilizerFrame { - fn measure(&mut self, qubit: usize) -> Option; - fn row_multiply(&mut self, src: usize, dst: usize); // uses the g-rule - // pivot search, canonicalization, transposition guard, ... + /// Find a generator that anticommutes with the measured Pauli (the pivot). + fn anticommuting_pivot(&self, qubit: usize) -> Option; + /// Multiply generator `src` into `dst` (uses the Aaronson–Gottesman g-rule). + fn row_multiply(&mut self, src: usize, dst: usize); + /// Restore canonical form after elimination. + fn canonicalize(&mut self); } ``` @@ -354,12 +360,11 @@ parameters with one. In particular, there is no `PauliSumAlgorithm` trait that bundles a term map with a policy: storage layout and policy are orthogonal choices. -The choices that are *intrinsic to a storage instance* — which word it keys on -and which coefficient it accumulates — are associated types of that storage -rather than free parameters, the same way `HashMap` fixes `K` and `V` per -concrete map. `SumStorage` (defined below) therefore exposes `type Word` and -`type Coeff`, and the reusable sparse-sum shape needs only its storage and its -policy: +The choices that are *intrinsic to a storage instance* — which key it maps and +which coefficient it accumulates — are associated types of that storage rather +than free parameters, the same way `HashMap` fixes `K` and `V` per concrete +map. `SumStorage` (defined below) therefore exposes `type Key` and `type Coeff`, +and the reusable sparse-sum shape needs only its storage and its policy: ```rust pub struct Sum @@ -482,51 +487,60 @@ removes the current split where the `Strategy` trait lives in `ppvm-traits` but its concrete strategies live in `ppvm-pauli-sum`; the policy is not an algorithm-agnostic `ppvm-traits-2` concern. -#### The map is a graded algebra over `C[W]` +#### The map is a graded algebra over `C[K]` The associative coefficient map — implemented today by `HashMap`, `IndexMap`, -and `DashMap` — is, algebraically, the **free `C`-module on the set of words**: -a finitely-supported function `W ⇀ C`, an element of `C[W]`. Every operation it -must support is a module (or, at the top, ring) operation, so its capabilities -are **graded by algebraic strength** rather than named ad hoc. This replaces the -flat `ACMapBase` / `ACMapIter` / `ACMapAddAssign` / `ACMapInsert` / -`ACMapRetain` / `ACMapConsume` split — which the design's own admission rule -would reject as six grandfathered names — with layers, each justified by a -distinct algebraic property *and* a distinct consumer: +and `DashMap` — is, algebraically, the **free `C`-module on a set of keys `K`**: +a finitely-supported function `K ⇀ C`, an element of `C[K]`. The keys are +whatever is `Indexable` — `PauliWord` for `PauliSum`, `Tableau` for a stabilizer +mixture — so the *same* algebra serves both. Every operation it must support is a +module (or, at the top, ring) operation, so its capabilities are **graded by +algebraic strength** rather than named ad hoc. This replaces the flat +`ACMapBase` / `ACMapIter` / `ACMapAddAssign` / `ACMapInsert` / `ACMapRetain` / +`ACMapConsume` split — which the design's own admission rule would reject as six +grandfathered names — with layers, each justified by a distinct algebraic +property *and* a distinct consumer: | Layer | Algebra | Trait | Justifying consumer | | --- | --- | --- | --- | -| L0 | finite partial function `W ⇀ C` | `Support` | everything; read-out / export | +| L0 | finite partial function `K ⇀ C` | `Support` | everything; read-out / export | | L1 | abelian-monoid formation + canonical support | `Accumulate` (`accumulate_batch` + `reduce`) | forming any linear combination | | L2 | the `C`-module action | `Scale` | normalization, global factors | | L3 | the bilinear form | `Pair` (`probe_batch`, `overlap`) | overlap / expectation read-out | -| L4 | the ring (group-algebra) product | `Multiply` | operator composition, squaring an observable | +| L4 | the ring (group-algebra) product | `Multiply` (needs a key product) | operator composition, squaring an observable | -The **minimum** is L0 + L1: a finitely-supported `W ⇀ C` you can accumulate +The **minimum** is L0 + L1: a finitely-supported `K ⇀ C` you can accumulate into and reduce. That *is* the algebraic essence of a sparse sum. Truncation is deliberately **absent** from this table: dropping terms that are still in the support is an approximation that breaks module exactness, so it is not an algebraic operation — it belongs to `Policy` (a `Retain` capability), sitting outside the algebra. That is why truncation was always awkward to place. +The key type is **`Indexable`, not `Word`**: `C[K]` is the free module over any +index set, so the algebra needs only a valid map key. Requiring `Word` here would +leak Pauli-specificity into the general algebra and block the tableau mixture, +whose key is a `Tableau` (an `Indexable` that is not a `Word`). Pauli-specific +propagation re-adds the `Word` / `PauliBits` bound on *its* methods, not on the +algebra. + ```rust -/// L0 — the container. Note: no `&mut (W, C)` and no `&mut [C]` slot access is +/// L0 — the container. Note: no `&mut (K, C)` and no `&mut [C]` slot access is /// exposed, so a columnar (structure-of-arrays) backend is expressible. `iter` /// is read-only export; a SoA backend synthesizes the pairs from its columns. pub trait Support { - type Word: Word + Indexable; + type Key: Indexable; // PauliWord, LossyPauliWord, or Tableau type Coeff: Coefficient; fn len(&self) -> usize; - fn get(&self, key: &Self::Word) -> Option; - fn iter(&self) -> impl Iterator; + fn get(&self, key: &Self::Key) -> Option; + fn iter(&self) -> impl Iterator; } /// L1 — the module core: form linear combinations, then canonicalize. pub trait Accumulate: Support { /// Build side of the hash join: merge a produced batch, accumulating onto an /// existing key or inserting a new one. Columnar in; the scalar - /// `accumulate(w, c)` is provided sugar over a batch of one. - fn accumulate_batch(&mut self, terms: &TermBatch); + /// `accumulate(k, c)` is provided sugar over a batch of one. + fn accumulate_batch(&mut self, terms: &TermBatch); /// Canonicalize to reduced finite-support form: drop every key whose /// coefficient `is_zero()`. First-class and run **only here** — see below. @@ -535,19 +549,30 @@ pub trait Accumulate: Support { /// L2 — the C-module action: a pure elementwise map over the coefficients. pub trait Scale: Support { - fn scale(&mut self, s: &Self::Coeff); // ∀ w. c_w *= s + fn scale(&mut self, s: &Self::Coeff); // ∀ k. c_k *= s } /// L3 — the bilinear form: the read side of the hash join. pub trait Pair: Support { - fn probe_batch(&self, keys: &KeyBatch, out: &mut [Option]); + fn probe_batch(&self, keys: &KeyBatch, out: &mut [Option]); fn overlap(&self, other: &Self) -> Self::Coeff; } -/// L4 — the ring product. The Pauli product injects powers of `i`, so the -/// coefficient must absorb phase: `Multiply` is bounded on `ComplexCoefficient`. +/// A key whose set carries a product — the group/monoid structure that lifts +/// `C[K]` from a module to an algebra. `PauliWord` implements it (the phased +/// Pauli product `v·w = ±i^k (v⊕w)`); a bare `Tableau` mixture key does not. +pub trait KeyProduct: Indexable { + /// Product of two keys, with the phase it produces (folded onto the coeff). + fn key_mul(&self, other: &Self) -> (Self, Phase); +} + +/// L4 — the ring product. The only layer that needs the *key* to carry a +/// product; it stays optional and is not implemented for a key type that has +/// none. For Paulis the product injects powers of `i`, so the coefficient must +/// absorb phase: bounded on `ComplexCoefficient`. pub trait Multiply: Accumulate where + Self::Key: KeyProduct, Self::Coeff: ComplexCoefficient, { fn multiply_into(&self, other: &Self, acc: &mut Self); @@ -628,15 +653,82 @@ The producer difference lives entirely on the **term-production side** — the only ever accumulates produced batches. That is what dissolves the old `map_add` / `map_insert` split and its `Vec<(W, C)>` staging leak. +A producer is a **monomorphized, inlinable** type, never `dyn` — this is a hot +loop, so the abstraction must compile to nothing: + +```rust +pub trait TermProducer { + /// Push the produced terms for one existing (key, coeff) into the sink. + fn produce>(&self, key: &K, coeff: &C, sink: &mut S); +} + +/// Bijective re-key (Clifford): one produced term per input. +pub struct RekeyProducer { f: F } // ZST when the closure captures by copy + +impl (K, C)> TermProducer for RekeyProducer { + #[inline(always)] + fn produce>(&self, key: &K, coeff: &C, sink: &mut S) { + let (k, c) = (self.f)(key, coeff); + sink.push(k, c); + } +} +``` + +Because `apply

` and `produce` are generic, each gate call site +monomorphizes and the `#[inline]` body folds into the accumulate loop; the sink's +column is pre-sized from `Policy::capacity`, so there is no per-term allocation. +The only inherent cost is the `key.clone()` inside the closure — a stack copy for +`PauliWord`, a memcpy for `Tableau` — which a hand-written loop pays too. + +#### One engine, two key types: Pauli propagation and the tableau mixture + +Because the algebra is over `Key: Indexable` and a gate is a `TermProducer` that +transforms each key, **Pauli propagation and the classical stabilizer mixture are +the same `Sum` engine** — they differ only in the key type and the key's own +conjugation: + +```rust +pub type PauliSum = Sum, P>; +pub type TableauMixture = Sum, P>; +// TableauMixture replaces the former GeneralizedTableauSum. + +// A Clifford gate on ANY sum: re-key each key via its own `Clifford` impl. +impl Clifford for Sum +where + S: SumStorage, + S::Key: Clifford + Clone, // PauliWord's is symplectic bits; Tableau's is the inverse-tableau update + P: Policy, +{ + fn h(&mut self, q: usize) { + self.apply(RekeyProducer::new(|k: &S::Key, c: &S::Coeff| { + let mut next = k.clone(); + next.h(q); // dispatched to the key type's Clifford impl + (next, c.clone()) // (+ any phase the conjugation puts on the coeff) + })); + } + // s, cnot, cz likewise +} +``` + +For `S::Key = PauliWord`, `next.h(q)` is the Step-3 `PauliBits` / `SymplecticColumns` +bit op; for `S::Key = Tableau`, it is the tableau's own `Clifford` impl over its +inverse-tableau storage. Same engine, same producer, same `accumulate` / `reduce` +— only the key's `Clifford` implementation varies. This is the "smallest useful +common factor" the earlier iteration deferred: it is the key-agnostic graded +algebra, which Step 5 already built and this section unlocks by relaxing the key +bound from `Word` to `Indexable`. The mixture's coefficient-aware `O(n^2)` +measurement remains a `Tableau`-specific algorithm (via `StabilizerFrame`), layered +on top rather than merged into the engine. + A `SumStorage` is a new abstraction extracted from the fields currently owned directly by `PauliSum`: its maps and reusable workspace. It is an actual value, not a marker configuration: ```rust pub trait SumStorage: Clone { - type Word: Word + Indexable; + type Key: Indexable; // PauliWord for PauliSum, Tableau for a mixture type Coeff: Coefficient; - type Map: Accumulate; + type Map: Accumulate; fn data(&self) -> &Self::Map; fn data_mut(&mut self) -> &mut Self::Map; @@ -647,7 +739,12 @@ pub trait SumStorage: Clone { /// batch to `Accumulate::accumulate_batch`, then `reduce`s. This one method /// subsumes the former `map_add` / `map_insert` / `map_insert_multiple`: /// they differed only in their producer, not in the map operation. - fn apply>(&mut self, producer: P); + /// + /// `P` is a type parameter, never `dyn`, so `apply` monomorphizes and the + /// producer's `produce` (marked `#[inline]`) folds into the accumulate loop: + /// after compilation this is a tight loop with the gate body inlined and no + /// per-term allocation, identical to a hand-written loop. + fn apply>(&mut self, producer: P); fn reduce(&mut self); } @@ -752,7 +849,7 @@ changed according to whether their underlying responsibility changes: | `PauliSum::map_insert`, `map_add` | one `SumStorage::apply(producer)` → `Accumulate::accumulate_batch` + `reduce` | They differed only in their producer, not the map op; the `Vec<(W,C)>` staging leak is removed. | | `PauliSum` map pair and `scratch` fields | `SumStorage` | A new abstraction is extracted from currently unnamed storage state. | | `PauliSum` | `PauliSum` over generalized `Sum` machinery | Pauli-facing code keeps its established name; `Sum` names the new cross-algebra engine. | -| `GeneralizedTableauSum` | `GeneralizedTableauSum` | The classical mixture algorithm remains the same concept. | +| `GeneralizedTableauSum` | `TableauMixture = Sum, P>` | The mixture is `C[Tableau]` — the same graded algebra as `PauliSum`, keyed on `Tableau`; storage unifies while the `O(n^2)` measurement stays tableau-specific. | | `EntryStore`, `VecStorage`, `MapStorage` | unchanged | The proposal uses the existing storage boundary and implementations. | | `Config::BuildHasher` | removed; `Indexable::key_hash() -> u64` | The direct-digest model leaves the map no hasher to choose; the key's internal algorithm is a private representation parameter, and the finalized digest is exposed as a value. | | `HashFinalize` | retained, but private to `ppvm-pauli-word` | The per-algorithm/per-width finalization fold still runs inside `key_hash()`; it leaves the algebra-agnostic contract. | From f50bfade621d677e21b9d451c821ead9a44b852e Mon Sep 17 00:00:00 2001 From: Roger-luo Date: Thu, 30 Jul 2026 21:07:55 -0400 Subject: [PATCH 09/10] docs(design): drop SumStorage/owned workspace; backends are containers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Correct two over-designs surfaced while checking the generalized tableau: - No SumStorage trait and no owned workspace. The reusable auxiliary map was an artifact of the old mutate-while-iterate model; the producer/TermBatch model reads the map through & and writes into a separate TermSink, so no aux map is needed. Bundling one into storage would double every sum's allocation and clone it on every mixture branch. `Sum` now owns only storage + policy + n_sites; `apply` is a thin method producing into a transient batch, with reuse an opt-in driver optimization (PauliSum), never forced on the frequently-cloned tableau side. - Backends are the containers themselves. The graded algebra traits are impl'd directly on Vec<(K,C)> (small support) and HashMap (large) — no VecStore/HashStore newtypes, only type aliases for readability. ColumnStore is the one backend that must be a new struct (SoA planes). - Add the generalized tableau as the third instantiation: C[Bitstring]. Its current SparseVector trait is already a second hand-rolled copy of the algebra (add_or_insert = accumulate, retain = Retain, iter = Support); the amplitude vector is Sum, CoefficientThreshold>, Clifford gates hit the frame, non-Clifford gates branch the amplitudes. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/design/tableau-data-structure.md | 13 +- .../traits-2-configuration-and-hashing.md | 287 ++++++++++-------- 2 files changed, 172 insertions(+), 128 deletions(-) diff --git a/docs/design/tableau-data-structure.md b/docs/design/tableau-data-structure.md index 6645a4e44..86ec41de0 100644 --- a/docs/design/tableau-data-structure.md +++ b/docs/design/tableau-data-structure.md @@ -113,16 +113,19 @@ pub struct Tableau { // hash caches only — NO rng field; see "Randomness is injected" } -pub struct GeneralizedTableau { - tableau: Tableau, - coefficients: S, - // threshold and measurement record +pub struct GeneralizedTableau { + frame: Tableau, // the Clifford frame U; owns loss + amplitudes: Sum)>, P>, // C[bitstring] — the graded algebra + // measurement record } ``` `GeneralizedTableau` therefore has no separate `is_lost: Vec`. Its gate, noise, and measurement algorithms query and mutate the loss plane owned by the -inner tableau. +inner `frame` tableau. Its `amplitudes` field is the graded-algebra `Sum` over +bitstring keys (see +[A third instantiation](traits-2-configuration-and-hashing.md#a-third-instantiation-the-generalized-tableau)): +Clifford gates update `frame` only, and non-Clifford gates branch `amplitudes`. `lost_count` is derived metadata used to preserve a fast `lost_count == 0` path. It is excluded from equality and hashing; debug builds diff --git a/docs/design/traits-2-configuration-and-hashing.md b/docs/design/traits-2-configuration-and-hashing.md index 913405bb3..f2caf0a4d 100644 --- a/docs/design/traits-2-configuration-and-hashing.md +++ b/docs/design/traits-2-configuration-and-hashing.md @@ -363,33 +363,36 @@ choices. The choices that are *intrinsic to a storage instance* — which key it maps and which coefficient it accumulates — are associated types of that storage rather than free parameters, the same way `HashMap` fixes `K` and `V` per concrete -map. `SumStorage` (defined below) therefore exposes `type Key` and `type Coeff`, -and the reusable sparse-sum shape needs only its storage and its policy: +map. `Accumulate` (the graded map trait, below) therefore exposes `type Key` and +`type Coeff`, and the reusable sparse-sum shape needs only its storage container +and its policy: + +The storage parameter is simply an `Accumulate` container (the graded algebra of +[The map is a graded algebra over `C[K]`](#the-map-is-a-graded-algebra-over-ck)) +— there is no separate `SumStorage` trait and, crucially, **no owned workspace**: ```rust pub struct Sum where - S: SumStorage, - P: Policy, + S: Accumulate, + P: Policy, { - storage: S, + storage: S, // Vec<(K, C)>, HashMap, or ColumnStore policy: P, /// Invariant: every key `k` in `storage` satisfies `k.n_sites() == - /// n_sites`. An empty sum has no word to derive the width from, so the + /// n_sites`. An empty sum has no key to derive the width from, so the /// field is carried explicitly and checked by a `debug_assert!` on every /// insertion path. n_sites: usize, } ``` -`Word` is no longer a free parameter with no home; it is `S::Word`, so there is -no phantom axis and no way to pair a storage type with the wrong word. This also -resolves the tension with the earlier note that associated types are not -justified merely to shrink a parameter count: `Word` and `Coeff` are genuinely -intrinsic to a storage instance, so making them associated types is proper -encapsulation, not a false bundle. Propagation methods still select their -algebra through the site type by bounding `S::Word: Word` or -`Word`. +`Sum` owns **only** its storage, policy, and width — no auxiliary map, no scratch +buffer. Clone is therefore pure data, which matters because a stabilizer mixture +clones frequently. The key and coefficient are `S::Key` / `S::Coeff` (associated +types of `Accumulate`), so there is no phantom axis and no way to pair a storage +type with the wrong key. Pauli propagation re-adds the `Word` / `PauliBits` bound +on *its* methods (`S::Key: Word`), not on the engine. #### The convenience bundle @@ -398,16 +401,18 @@ foundational trait. Each alias fixes a domain's axes and defaults the common knobs, so a one-token `PauliSum` name returns: ```rust -pub type PauliSum = Sum, P>; -pub type LossyPauliSum = Sum, P>; -pub type FermionSum = Sum, P>; +// HashMapStore = HashMap; the storage is the +// bare container, the alias only names it and bakes in the pass-through hasher. +pub type PauliSum = Sum, P>; +pub type LossyPauliSum = Sum, P>; +pub type FermionSum = Sum, P>; ``` A user who wants to pin a reusable configuration writes one `type` line instead of an `impl Config`: ```rust -type MyPauliSum = Sum>, MaxPauliWeight>; +type MyPauliSum = Sum, IdentityBuildHasher>, MaxPauliWeight>; ``` This is deliberately a bundle, and it is safe to be one because it commits @@ -592,38 +597,40 @@ operation guarantees that by construction, and it earns its place three ways: - **Correctness** — there is no inline drop to get wrong (the caution above). - **It is a bulk primitive with a backend-specific implementation** — a scalar - `retain(|_, c| !c.is_zero())` on `HashStore`, but a prefix-sum **stream - compaction** kernel on a GPU column store. Folded into `consume`/swap it could + `retain(|_, c| !c.is_zero())` on a `HashMap`, but a prefix-sum **stream + compaction** kernel on a `ColumnStore`. Folded into `consume`/swap it could not have a bulk form. - **It amortizes** — several `accumulate_batch` calls (or a `multiply_into` outer product) can precede a single `reduce`, instead of canonicalizing after every step. -#### Columnar from day one: two backends, no AoS leak +#### Backends are containers; columnar is expressible from day one -The whole point of this refactor is to unblock SIMD / GPU execution, so the -layers are designed to admit a **structure-of-arrays** accumulator from the -start — not as a later retrofit. Two backends must both be expressible against -the same traits: +The graded traits are `impl`'d **directly on the container** — no wrapper types. +The lookup strategy is a *collection* choice by support size, and the memory +layout is a separate *layout* choice by execution target: -- **`HashStore`** (array-of-structs, `hashbrown`): implements every layer with - scalar kernels. `accumulate_batch` scatters the batch through scalar inserts; - `reduce` is a `retain`; `scale` is a scalar loop. Correct and fast for the CPU - scalar path, but a dead end for the GPU. -- **`ColumnStore`** (structure-of-arrays, for SIMD / GPU): coefficients in one - contiguous column, keys in plane blocks. `accumulate_batch` is a - radix-partitioned, group-prefetched hash-join build; `scale` is one vectorized - `*=` over the coefficient column; `reduce` is a prefix-sum compaction; - `probe_batch` is a prefetched probe with coalesced gathers. +- **`Vec<(K, C)>`** — an unsorted coordinate list, linear-scan `accumulate`. Best + for small support (the `GeneralizedTableau` amplitude vector); this is today's + `SparseVector`. +- **`HashMap`** — hash-join `accumulate`. Best for + large support (`PauliSum`); this is today's `ACMap` path. (AoS, scalar CPU.) +- **`ColumnStore`** — the one backend that *must* be a new struct, because it is + structure-of-arrays: coefficients in one contiguous column, keys in plane + blocks. Same hash-join build as the `HashMap`, but `scale` is one vectorized + `*=`, `reduce` is a prefix-sum compaction, and `probe_batch` uses coalesced + gathers. It is `HashMap`'s data re-laid for SIMD / GPU, not a third collection. -Both are expressible **only because no trait signature leaks the array-of-structs -layout**. The rules the whole storage contract holds to: +The whole point of the refactor is to unblock SIMD / GPU, so the traits are +designed so `ColumnStore` is expressible from day one — **only because no trait +signature leaks the array-of-structs layout**. The rules the whole storage +contract holds to: 1. coefficients are a contiguous column and keys are plane blocks (SoA) — so L2 Scale, L1 Reduce, and L3 Pair vectorize and GPU loads coalesce; 2. term I/O is a columnar `TermBatch` / `KeyBatch` (see [The batch contract](#the-batch-contract)), never a scalar per-element - callback — so `HashStore` accepts a batch and processes it scalar internally + callback — so the `HashMap` accepts a batch and processes it scalar internally while `ColumnStore` vectorizes it, from the *same* call; 3. the mutating operations are whole-map (`scale`, `reduce`) or batch (`accumulate_batch`), never per-slot; and @@ -688,14 +695,18 @@ the same `Sum` engine** — they differ only in the key type and the key's own conjugation: ```rust -pub type PauliSum = Sum, P>; -pub type TableauMixture = Sum, P>; +// The storage is a bare container with the algebra traits impl'd on it; the +// alias just fixes the container and bakes in the pass-through hasher. +pub type HashMapStore = HashMap; + +pub type PauliSum = Sum, P>; +pub type TableauMixture = Sum, P>; // TableauMixture replaces the former GeneralizedTableauSum. // A Clifford gate on ANY sum: re-key each key via its own `Clifford` impl. impl Clifford for Sum where - S: SumStorage, + S: Accumulate, S::Key: Clifford + Clone, // PauliWord's is symplectic bits; Tableau's is the inverse-tableau update P: Policy, { @@ -720,46 +731,81 @@ bound from `Word` to `Indexable`. The mixture's coefficient-aware `O(n^2)` measurement remains a `Tableau`-specific algorithm (via `StabilizerFrame`), layered on top rather than merged into the engine. -A `SumStorage` is a new abstraction extracted from the fields currently owned -directly by `PauliSum`: its maps and reusable workspace. It is an actual value, -not a marker configuration: +#### A third instantiation: the generalized tableau -```rust -pub trait SumStorage: Clone { - type Key: Indexable; // PauliWord for PauliSum, Tableau for a mixture - type Coeff: Coefficient; - type Map: Accumulate; +The graded algebra also captures the *single* coefficient-aware tableau, and +finding that it does is strong validation: the current `GeneralizedTableau` +already carries a **second hand-rolled copy of the algebra** in its +`SparseVector` trait, whose `add_or_insert` / `retain` / `iter` are exactly +`Accumulate` / `Retain` / `Support`. - fn data(&self) -> &Self::Map; - fn data_mut(&mut self) -> &mut Self::Map; +A `GeneralizedTableau` represents `U|c⟩`: a Clifford **frame** `U` (a `Tableau`) +times a sparse superposition `|c⟩ = ∑_b c_b |b⟩` over bitstrings. That amplitude +vector is `C[Bitstring]` — the same graded algebra, a third key type: - /// Apply a term producer — a gate — into the map. The producer emits its - /// terms into a `TermSink`; the storage owns whether that sink is a scratch - /// `Vec`, a columnar `TermBatch`, or per-thread buffers, hands the staged - /// batch to `Accumulate::accumulate_batch`, then `reduce`s. This one method - /// subsumes the former `map_add` / `map_insert` / `map_insert_multiple`: - /// they differed only in their producer, not in the map operation. - /// - /// `P` is a type parameter, never `dyn`, so `apply` monomorphizes and the - /// producer's `produce` (marked `#[inline]`) folds into the accumulate loop: - /// after compilation this is a tight loop with the gate body inlined and no - /// per-term allocation, identical to a hand-written loop. - fn apply>(&mut self, producer: P); +```rust +pub struct GeneralizedTableau { + frame: Tableau, // owns loss (Step 7) + amplitudes: Sum)>, P>, // C[bitstring], Vec backend + measurement_record: Vec>, +} +``` - fn reduce(&mut self); +Its gate semantics fall out of the same machinery: + +- **Clifford gate → the frame only.** `G·U|c⟩ = (GU)|c⟩`, so `self.frame.h(q)` + updates the tableau and the amplitude vector is untouched — free on + `C[Bitstring]`. +- **Non-Clifford gate → branch the amplitude `Sum`.** Read the rotation axis from + the frame (the `O(n^2)` decomposition), then `self.amplitudes.apply(producer)` + branches `c → cos·c + i·sin·c'` and accumulates — the identical branch-producer + → `accumulate` → drop-below-threshold pattern `PauliSum` rotations use, down to + the current code spilling the merge into a temporary map. +- **`coefficient_threshold`** is `CoefficientThreshold` `Policy`. + +The `Vec` backend is the right one here: the support is T-count-bounded (small), +so linear-scan `accumulate` beats hashing — the concrete motivation for the `Vec` +container. As with the mixture, the frame↔amplitude coupling and the `O(n^2)` +measurement stay `GeneralizedTableau`-specific; only the *storage and accumulate* +unify — exactly the level at which `PauliSum` fits. + +There is **no `SumStorage` trait, and no owned workspace.** An earlier draft had +a `SumStorage` value bundling the map with a reusable *auxiliary map* and scratch +buffer — but that aux map was an artifact of the *old* mutation model, where a +gate iterated the live map and inserted new keys, and so needed a second map to +stage into and swap. The producer / `TermBatch` model removes that aliasing: a +`TermProducer` *reads* the map through `&` and *writes* produced terms into a +`TermSink` (the batch), then `accumulate_batch` iterates the **batch** — a +separate buffer — and merges into the map. Nothing is mutated while it is +iterated, so there is no aux map to keep. Bundling one into the storage would +double every sum's allocation and clone it on every mixture branch, for a +workspace the design no longer needs. + +`apply` is therefore a thin method on `Sum` itself, over any `Accumulate` +container: + +```rust +impl> Sum { + /// Produce terms into a batch, merge, canonicalize, truncate. `TP` is a + /// type parameter, never `dyn`, so this monomorphizes and the producer's + /// `#[inline]` `produce` folds into the loop — no per-term allocation. + fn apply>(&mut self, producer: TP) { + let mut batch = /* transient, or a driver-owned reusable buffer */; + for (k, c) in self.storage.iter() { producer.produce(&k, &c, &mut batch); } + self.storage.accumulate_batch(&batch); + self.storage.reduce(); + self.policy.truncate(&mut self.storage); + } } ``` -`SumStorage` owns the reusable workspace and the sink. It no longer exposes a -family of closure methods returning `Option>` — that return type -leaked the scratch-`Vec` staging mechanism into the contract. Instead a gate is -a `TermProducer` that pushes into a `TermSink`, so the staging representation -(auxiliary map, scratch vector, columnar batch, per-thread buffers) is entirely -the storage's private choice, and the same gate kernel drives the scalar -`HashStore` and the columnar `ColumnStore` unchanged. `apply` desugars to -`accumulate_batch` + `reduce` on the underlying `Accumulate` map; the -whole-map rewrite (Clifford) and the branching expansion (rotation) are two -`TermProducer`s, per the producer table above. +The only staging that remains is the `TermBatch` the producer fills. It is +**transient by default** (allocated per `apply`), which is exactly right for a +frequently-cloned, small `GeneralizedTableau`. A long-lived driver that applies +millions of gates — `PauliSum` — may instead own **one** reusable batch (and, for +a whole-map Clifford rewrite, a second map to swap into) to amortize allocation. +That reuse is an **opt-in optimization of the driver**, never a field of `Sum` +and never imposed on the tableau side. Because the batch a producer stages is exactly the structure-of-arrays layout prefetched, SIMD, threaded, and offloaded backends consume, the execution @@ -771,8 +817,8 @@ below. #### The pass-through storage contract Because a key's [`key_hash()`](#indexable-values) is already the finalized, -avalanche-quality digest, the provided `SumStorage` backends **guarantee** that -their inner map consumes it directly, through an identity pass-through hasher: +avalanche-quality digest, the provided storage aliases **bake in** an identity +pass-through hasher so the map consumes the digest directly: ```rust #[derive(Default, Clone)] @@ -784,8 +830,8 @@ impl std::hash::Hasher for IdentityHasher { } ``` -The shipped `HashStore` / `DashStore` therefore instantiate their `HashMap` / -`DashMap` with `IdentityBuildHasher`, so `finish() == key.key_hash()` and the +The `HashMapStore` / `DashMapStore` aliases therefore fix their `HashMap` / +`DashMap` to `IdentityBuildHasher`, so `finish() == key.key_hash()` and the digest reaches hashbrown untouched. This is part of the storage contract, not a user responsibility: the user selects the key's *internal* digest algorithm (the `H` in `PauliWord`), which governs distribution quality, and never selects @@ -845,20 +891,21 @@ changed according to whether their underlying responsibility changes: | `Strategy` | `Policy` | Intentional terminology change requested for this redesign; the `Copy` bound is dropped. | | `Coefficient::cutoff` | removed | The truncation predicate moves to `Policy`; `Coefficient::magnitude` exposes only the value property the policy thresholds. | | `Coefficient::sin_cos` | `Angle` | The rotation angle becomes a separate domain, defaulting to the coefficient type. | -| `ACMap` (`ACMapBase`/`Iter`/`AddAssign`/`Insert`/`Retain`/`Consume`) | graded `Support` / `Accumulate` / `Scale` / `Pair` / `Multiply` | The six flat names are replaced by algebra layers over `C[W]`; `Retain` (truncation) leaves the algebra for `Policy`. | -| `PauliSum::map_insert`, `map_add` | one `SumStorage::apply(producer)` → `Accumulate::accumulate_batch` + `reduce` | They differed only in their producer, not the map op; the `Vec<(W,C)>` staging leak is removed. | -| `PauliSum` map pair and `scratch` fields | `SumStorage` | A new abstraction is extracted from currently unnamed storage state. | +| `ACMap` (`ACMapBase`/`Iter`/`AddAssign`/`Insert`/`Retain`/`Consume`) *and* `SparseVector` | graded `Support` / `Accumulate` / `Scale` / `Pair` / `Multiply`, `impl`'d directly on `Vec`/`HashMap` | Two hand-rolled copies of the same abstraction collapse into one over `C[K]`; `Retain` (truncation) leaves the algebra for `Policy`. | +| `PauliSum::map_insert`, `map_add` | one `Sum::apply(producer)` → `accumulate_batch` + `reduce` | They differed only in their producer, not the map op; the `Vec<(W,C)>` staging leak is removed. | +| `PauliSum` aux-map + `scratch` fields | removed | The aux map was an artifact of mutate-while-iterate; the producer/`TermBatch` model needs no owned workspace, so `Sum` owns none. | | `PauliSum` | `PauliSum` over generalized `Sum` machinery | Pauli-facing code keeps its established name; `Sum` names the new cross-algebra engine. | -| `GeneralizedTableauSum` | `TableauMixture = Sum, P>` | The mixture is `C[Tableau]` — the same graded algebra as `PauliSum`, keyed on `Tableau`; storage unifies while the `O(n^2)` measurement stays tableau-specific. | -| `EntryStore`, `VecStorage`, `MapStorage` | unchanged | The proposal uses the existing storage boundary and implementations. | +| `GeneralizedTableauSum` | `TableauMixture = Sum, P>` | The mixture is `C[Tableau]` — the same graded algebra as `PauliSum`, keyed on `Tableau`; storage unifies while the `O(n^2)` measurement stays tableau-specific. | +| `GeneralizedTableau.coefficients: SparseVector` | `Sum)>, CoefficientThreshold>` | The amplitude vector is `C[Bitstring]` — the same algebra, `Vec` backend; see [A third instantiation](#a-third-instantiation-the-generalized-tableau). | +| `EntryStore`, `VecStorage`, `MapStorage`, `SparseVector` backings | the container itself (`Vec`, `HashMap`, `ColumnStore`) | No storage-wrapper trait; the algebra traits are `impl`'d on the collection directly. | | `Config::BuildHasher` | removed; `Indexable::key_hash() -> u64` | The direct-digest model leaves the map no hasher to choose; the key's internal algorithm is a private representation parameter, and the finalized digest is exposed as a value. | | `HashFinalize` | retained, but private to `ppvm-pauli-word` | The per-algorithm/per-width finalization fold still runs inside `key_hash()`; it leaves the algebra-agnostic contract. | -| (new) | `IdentityBuildHasher` + pass-through `SumStorage` contract | The provided storage consumes `key_hash()` directly instead of re-hashing it. | +| (new) | `IdentityBuildHasher` + pass-through storage aliases | The provided `HashMapStore` alias consumes `key_hash()` directly instead of re-hashing it. | | `PauliStorage` | removed | Packed backing storage becomes private to the concrete word representation. | | (new) | `Columnar`, `KeyColumn`, `KeyBatch`, `TermBatch`, `TermSink`/`TermProducer` | Structure-of-arrays term types; the columnar spelling of `Accumulate`/`Pair`. See [Batch execution and the hash-join contract](#batch-execution-and-the-hash-join-contract). | -| (new) | `HashStore` (AoS) and `ColumnStore` (SoA) backends | Two `SumStorage` backends over the same graded traits — the SoA one is expressible from day one because no signature leaks AoS. | +| (new) | `ColumnStore` (SoA) backend | The one storage that must be a new struct (SoA planes); `Vec`/`HashMap` need none. Expressible from day one because no signature leaks AoS. | -Names such as `Word`, `Indexable`, `SumStorage`, and `Sum` are +Names such as `Word`, `Indexable`, `Accumulate`, and `Sum` are therefore new because they denote abstractions that do not exist in the current implementation, not because the existing API is being renamed wholesale. @@ -882,11 +929,10 @@ This rule keeps: implemented by `PhasedPauliWord` and `Tableau`; - `StabilizerFrame`, the role-exclusive tableau operations; - gate and noise traits, implemented across propagation and tableau backends; -- `SumStorage`, and the graded map layers `Support` / `Accumulate` / `Scale` / - `Pair` / `Multiply`, each admitted by a distinct algebraic property *and* a - distinct consumer, and implemented by the scalar `HashStore` and the columnar - `ColumnStore`; -- `EntryStore`, already implemented by `VecStorage` and `MapStorage`; and +- the graded map layers `Support` / `Accumulate` / `Scale` / `Pair` / `Multiply`, + each admitted by a distinct algebraic property *and* a distinct consumer, and + `impl`'d directly on `Vec<(K,C)>` (small support), `HashMap` (large), and + `ColumnStore` (SIMD/GPU); - `Policy`, implemented by independent capacity and truncation behaviors, and consuming the non-algebraic `Retain` capability rather than the map algebra. @@ -908,35 +954,27 @@ Pauli rotation may produce: c P -> c cos(theta) P + c sin(theta) P' ``` -The existing entry can be updated while the active map is traversed, but a new -key cannot generally be inserted into that same map during mutable iteration. -New terms must therefore be staged and merged after the traversal. New keys -may collide with each other or with existing keys, so the merge accumulates -their coefficients. - -Only one staging mechanism is required for correctness. The current engine -uses two because they serve different performance paths: - -- an auxiliary map supports whole-map rewrites, combines output collisions as - they are produced, and retains its allocation across operations; and -- a reusable `Vec<(W, C)>` scratch buffer stages additional terms when existing - entries remain in place, avoiding an auxiliary-map insertion followed by a - second map probe during the merge. +A `TermProducer` *reads* the live map through `&` and *writes* the produced +terms into a `TermSink` — a separate buffer, never the map itself — so there is +no mutate-while-iterate hazard and **no auxiliary map is needed.** After the +producer finishes, `accumulate_batch` merges that buffer into the map, combining +colliding coefficients. This is a deliberate simplification over the old model, +which iterated the live map and inserted into it, and so needed a second map to +stage into and swap. -Conceptually: +The only staging that remains is the single `TermSink` buffer: ```text -whole-map rewrite: active map -> auxiliary map -> swap -in-place branching: active map + scratch buffer -> merge into active map +old model: iterate active map, insert new keys -> needs aux map + swap +new model: read map -> produce into TermSink buffer -> accumulate_batch into map ``` -Neither physical mechanism is part of the public storage contract. Both are the -`TermSink` a `TermProducer` pushes into: a default `SumStorage` may privately -hold both an auxiliary map and the reusable vector, matching the current -`PauliSum` layout; a simpler backend may sink everything through one auxiliary -map; a columnar backend sinks into a `TermBatch`; another may use per-thread -buffers. Whether retaining both mechanisms is worthwhile is a benchmark -decision, not a type-system requirement. +The buffer is **transient by default**, allocated per operation — right for a +small, frequently-cloned `GeneralizedTableau`. A long-lived driver (`PauliSum`) +may own **one** reusable buffer (and, for a whole-map Clifford rewrite, a second +map to swap into) purely to amortize allocation across millions of gates. That +reuse is an opt-in optimization of the driver, not a field of `Sum` and not part +of the storage contract. Whether it is worthwhile is a benchmark decision. ## Batch execution and the hash-join contract @@ -964,7 +1002,7 @@ does not fit in cache, so each probe is a random access that misses to main memory, and the probe phase is bound by memory latency rather than by arithmetic. -`SumStorage::apply` and the `Accumulate`/`Pair` layers describe *what* the merge +`Sum::apply` and the `Accumulate`/`Pair` layers describe *what* the merge computes. But a producer that emitted one `(key, coeff)` at a time would expose no batch to prefetch, no homogeneous run to vectorize, no partition to distribute across threads, and no bulk kernel to hand to a device. That is why a @@ -1129,10 +1167,10 @@ independent of the merge strategy (the systems concern). - `Policy`: truncation already operates on the whole map and is naturally bulk; it should be expressible as a batch retain (via `KeyColumn::gather`) so it composes with a partitioned or offloaded table. -- `SumStorage`: its staging fields become the join's probe-side buffer (the - `TermSink`) and its `apply` desugars to `accumulate_batch` + `reduce`. Whether - it keeps both an auxiliary map and a scratch buffer stays a backend decision, - unchanged from above. +- `Sum::apply`: the producer's `TermSink` *is* the join's probe-side buffer, and + `apply` desugars to `accumulate_batch` + `reduce`. The buffer is transient by + default; a driver may reuse one to amortize — a benchmark decision, unchanged + from above. ### Parallel and offloaded backends @@ -1268,13 +1306,16 @@ do not leak into the shared trait-system design. The intended composition is explicit: ```rust -Sum // e.g. PauliSum = Sum> +Sum +// PauliSum = Sum, P> // C[PauliWord] +// TableauMixture = Sum, P> // C[Tableau] +// GeneralizedTableau amps = Sum)>, CoefficientThreshold> // C[Bitstring] Tableau -GeneralizedTableauSum ``` -Domain-specific aliases or wrappers can preserve `PauliSum` and introduce -`FermionSum` without rebuilding a monolithic configuration trait. +All three are `Sum` over the same graded algebra, differing only in the key type +and the container backend. Domain aliases preserve `PauliSum` and introduce +`FermionSum` without a monolithic configuration trait. ## Non-goals for the first prototype From e9076948dee276d1137a5c6d2aa716306e608741 Mon Sep 17 00:00:00 2001 From: Roger-luo Date: Thu, 30 Jul 2026 21:19:14 -0400 Subject: [PATCH 10/10] docs(design): scope Indexable to hash backends; Key bound is Eq + Clone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolve the key_hash/Indexable over-requirement surfaced by the Vec backend: - The universal key bound (Support::Key, all containers) is now just Eq + Clone — what a linear-scan Vec needs to find and store a key. It never hashes. - Indexable (the digest-bearing hash key) is unchanged but required only where hashing happens: the HashMap and ColumnStore Accumulate impls, and Columnar. - Consequence: a Bitstring used only in a Vec-backed GeneralizedTableau is Eq + Clone and provides no key_hash() it would never use; PauliWord / LossyPauliWord / Tableau, used in hash backends, remain Indexable. - Relax KeyProduct to Eq + Clone (the multiply's backend adds Indexable). No new trait; Indexable keeps its meaning, the digest requirement just lands where the cost is paid. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../traits-2-configuration-and-hashing.md | 27 +++++++++++++++---- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/docs/design/traits-2-configuration-and-hashing.md b/docs/design/traits-2-configuration-and-hashing.md index f2caf0a4d..5fda3b741 100644 --- a/docs/design/traits-2-configuration-and-hashing.md +++ b/docs/design/traits-2-configuration-and-hashing.md @@ -533,7 +533,7 @@ algebra. /// exposed, so a columnar (structure-of-arrays) backend is expressible. `iter` /// is read-only export; a SoA backend synthesizes the pairs from its columns. pub trait Support { - type Key: Indexable; // PauliWord, LossyPauliWord, or Tableau + type Key: Eq + Clone; // minimal; hash backends add `Indexable` type Coeff: Coefficient; fn len(&self) -> usize; fn get(&self, key: &Self::Key) -> Option; @@ -566,7 +566,7 @@ pub trait Pair: Support { /// A key whose set carries a product — the group/monoid structure that lifts /// `C[K]` from a module to an algebra. `PauliWord` implements it (the phased /// Pauli product `v·w = ±i^k (v⊕w)`); a bare `Tableau` mixture key does not. -pub trait KeyProduct: Indexable { +pub trait KeyProduct: Eq + Clone { /// Product of two keys, with the phase it produces (folded onto the coeff). fn key_mul(&self, other: &Self) -> (Self, Phase); } @@ -612,14 +612,20 @@ layout is a separate *layout* choice by execution target: - **`Vec<(K, C)>`** — an unsorted coordinate list, linear-scan `accumulate`. Best for small support (the `GeneralizedTableau` amplitude vector); this is today's - `SparseVector`. + `SparseVector`. Requires only `K: Eq + Clone` — it never hashes. - **`HashMap`** — hash-join `accumulate`. Best for large support (`PauliSum`); this is today's `ACMap` path. (AoS, scalar CPU.) + Requires `K: Indexable` (the direct digest). - **`ColumnStore`** — the one backend that *must* be a new struct, because it is structure-of-arrays: coefficients in one contiguous column, keys in plane blocks. Same hash-join build as the `HashMap`, but `scale` is one vectorized `*=`, `reduce` is a prefix-sum compaction, and `probe_batch` uses coalesced gathers. It is `HashMap`'s data re-laid for SIMD / GPU, not a third collection. + Requires `K: Indexable + Columnar`. + +So `Indexable` is a *per-backend* requirement, not a universal key bound: the +`Vec` path needs only `Eq + Clone`, and only the hash-indexed backends demand the +digest. The whole point of the refactor is to unblock SIMD / GPU, so the traits are designed so `ColumnStore` is expressible from day one — **only because no trait @@ -1192,8 +1198,19 @@ interchangeable rather than separate code paths. Hash-enabled `Word` values and `Tableau` can both be expensive, mostly-stable map keys. Their hashing contract should be expressed independently of any map. -The common capability is intentionally minimal, and it makes the one -load-bearing value — the finalized structural digest — first class: +`Indexable` is **not** the universal key bound, though. The universal +requirement — what the graded algebra's `type Key` bounds on, and all any +container needs to *find* and *store* a key — is just `Eq + Clone`: a linear-scan +`Vec` backend compares by equality and never hashes. Hashing, and therefore the +digest, is a property only a *hash* backend needs. So `Indexable` is required +solely on the `HashMap` / `ColumnStore` `Accumulate` impls (and on `Columnar`), +never on `type Key` itself. A `Bitstring` used only in a `Vec`-backed +`GeneralizedTableau` is `Eq + Clone` and provides no `key_hash()` it would never +use; `PauliWord` / `LossyPauliWord` / `Tableau`, used in hash backends, are +`Indexable`. + +With that scoping, `Indexable` makes the one load-bearing value — the finalized +structural digest — first class: ```rust pub trait Indexable: Clone + Eq + Hash {