docs: add architecture markdown file to docs

CONTRIBUTING.md

@@ -1,5 +1,10 @@
 # Contributing
 
+This page covers the **mechanics** — environment, linting, tests, commit/PR
+conventions, and releasing. For **where in the code to plug in** (adding a trials-table
+column, a QC check, or prototyping in a notebook), see
+[`docs/architecture/extending.md`](docs/architecture/extending.md).
+
 ### Linters and testing
 
 This project uses [uv](https://docs.astral.sh/uv/) to manage its environment and

README.md

@@ -8,18 +8,31 @@
 ![Python](https://img.shields.io/badge/python->=3.12-blue?logo=python)
 
 THIS LIBRARY IS CURRENTLY UNDER DEVLOPMENT AND IS SUBJECT TO CHANGE.
-Tries to follow this [diagram](https://github.com/AllenNeuralDynamics/aind-software-docs/blob/main/docs/source/diagrams/dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg). There are some slight differences so the diagram will be updated to reflect this as this repository further develops.
+Tries to follow this [diagram](https://github.com/AllenNeuralDynamics/aind-software-docs/blob/main/docs/source/diagrams/dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg). The relationship is a two-way street: the diagram guides the design here, and where the code deliberately diverges (or discovers something the diagram didn't anticipate), the diagram will get updated to match. Equally, now that the foundation has been built, the code could be refactored to match the diagram where that's the better fit. Both are treated as living artifacts kept in sync as this repository develops — see [`docs/architecture.md`](docs/architecture.md) for where they currently differ and why.
 
-Tools for processing raw [dynamic foraging](https://github.com/AllenNeuralDynamics/dynamic-foraging-task)
+This library contains tools for processing raw [dynamic foraging](https://github.com/AllenNeuralDynamics/dynamic-foraging-task)
 acquisition data into derived containers for NWB. The package loads raw acquisition
 streams through the
 [aind-behavior-dynamic-foraging](https://github.com/AllenNeuralDynamics/Aind.Behavior.DynamicForaging)
-data contract and assembles higher-level tables (currently the NWB `trials` table)
-from those streams.
+data contract and assembles higher-level tables (currently the NWB `trials` table), arrays, and does QC from those streams.
 
 > **Note:** Data must be acquired in the `aind-behavior-dynamic-foraging` data
 > contract format to be compatible with these tools.
 
+## Architecture
+For the design overview — how the pieces map to the reference diagram, how the stages
+compose, how to extend them, and the rationale behind the design choices — see
+[`docs/architecture.md`](docs/architecture.md).
+
+## Contributing
+Two entry points, depending on what you're doing:
+
+- **How to set up and submit** — environment, linting, tests, commit/PR conventions,
+  and releasing: [`CONTRIBUTING.md`](CONTRIBUTING.md).
+- **Where to add code** — the small, self-contained plug-in points (add a trials-table
+  column, add a QC check, prototype in a notebook), each with a worked example:
+  [`docs/architecture/extending.md`](docs/architecture/extending.md).
+
 ## Inputs and outputs
 **Input** — a raw acquisition directory in the `aind-behavior-dynamic-foraging`
 data contract format (Harp device registers, software events, and the task-logic /
@@ -39,6 +52,12 @@ rig / session input schemas), loaded via `RawDataLoader`.
     so the full raw dataset travels alongside the derived series.
 - **Trials table.** Built by `TrialTableBuilder`, one row per trial written to the
   NWB `trials` table (see [`docs/trials_table_mapping.md`](docs/trials_table_mapping.md)).
+- **Quality control.** A single `aind-data-schema` `QualityControl` object assembled by
+  `build_quality_control` from the raw (contract QC) and processed (behavior metrics)
+  QC stages, written to `quality_control.json`. Each check contributes a metric (a value
+  and pass/fail), and some attach a reference media (e.g. side bias, lick intervals)
+  saved as a PNG alongside the report for the QC portal to render. Unlike the two above,
+  this is a sidecar report rather than an NWB container.
 
 ## Level of Support
  - [x] Supported: We are releasing this code to the public as a tool we expect others to use. Issues are welcomed, and we expect to address them promptly; pull requests will be vetted by our staff before inclusion.

docs/architecture.md

@@ -0,0 +1,50 @@
+# Architecture
+
+An overview of the `dynamic-foraging-processing` design: how it maps to the
+low-level processing **[reference diagram](https://raw.githubusercontent.com/AllenNeuralDynamics/aind-software-docs/325e3284423c24ec8f4f3d5ca960e7410e3bd162/docs/source/diagrams/dynamic_foraging/low_level/dynamic-foraging-low-level-processing.svg)**, how the pieces compose, how to extend it,
+and the rationale behind the design choices.
+
+The alignment sections compare the reference diagram (the target design) with what
+the repo currently implements, call out where they differ, and document the rationale
+— doubling as a checklist for **updating the reference diagram to match the source
+code** (each row/section flags what the diagram should be revised to reflect).
+
+## Contents
+
+This document is split across several files:
+
+- **[Alignment](architecture/alignment.md)** — component-by-component comparison
+  against the reference diagram, and where/why the repo differs (the rationale behind
+  each divergence lives here too).
+- **[Composability & usage](architecture/composability.md)** — how the stages
+  compose, with worked examples at three tiers.
+- **[Extending it](architecture/extending.md)** — contribution points: adding
+  trials columns, QC checks, and prototyping in notebooks.
+- **[Changelog](changelog.md)** — release history.
+
+## Scope
+Delivered across 40+ tracked issues over roughly a four-week period. The work spans
+several distinct areas:
+
+- **Data engineering** — the contract loader, stream navigation, trials-table
+  assembly, NWB packaging (Zarr, reserved-name collisions, `id` handling, JSON-safe
+  serialization).
+- **Domain / behavioral knowledge** — licks, rewards, side bias, and auto-reward
+  semantics, including the meaning of each QC check.
+- **API / architecture design** — the class-vs-function question, composable stages,
+  and the `run_qc` reshaping; these were iterative design decisions rather than
+  one-pass coding.
+- **QC + schema** — `aind-data-schema`, `QualityControl` / `Processing`, and the
+  raw-vs-processed split.
+- **Release engineering** — semantic-release workflow, rulesets, bypass lists,
+  `SERVICE_TOKEN`, branch protection, and the `dev` → `main` flow — a substantial
+  area on its own.
+- **Deployment / orchestration** — built the NWB, QC, and metadata capsules and the pipeline
+  that chains them in Code Ocean, including logging, job-type configuration, and
+  end-to-end debugging many issues that came up.
+- **Testing rigor** — 100% coverage and docstring gates on every change.
+- **Documentation + stakeholder communication** — READMEs, capsule docs, and this
+  divergence-rationale doc for review.
+- **Organizational / coordination layer** — reconciling a reference
+  diagram, scientist requirements, engineering requirements, backward-compatibility constraints, and reviewer
+  feedback.

docs/architecture/alignment.md

@@ -0,0 +1,131 @@
+# Alignment with the reference diagram
+
+Back to the [architecture overview](../architecture.md).
+
+This compares the reference diagram (the target design) with what the repo currently
+implements, calls out where they differ, and documents the rationale — doubling as a
+checklist for updating the reference diagram to match the source code.
+
+The divergences below were deliberate; the reasoning is recorded here so it lives with
+the code, not only in PR descriptions. These are current decisions, not final — they
+are expected to evolve as requirements, tools, and the reference diagram converge, and
+this doc is updated alongside the code.
+
+## TL;DR
+- **Overall shape matches.** Loader → builders → package-to-NWB → QC-from-NWB, with
+  `BaseQC` / `RawQC` / `ProcessedQC`, and `aind-nwb-utils` + `aind-data-schema` as
+  the external writers. All present.
+- **Biggest intentional gap:** the diagram's dedicated **`EventsBuilder` →
+  `EventsTable`** stage is **deferred/blocked**. Per-event data (licks, reward
+  deliveries) is carried as **acquisition time series** — both because that stage
+  isn't built yet *and* because that form **maintains backward compatibility with the
+  existing pipeline** (refer to section 4 below).
+- **Biggest structural difference:** the diagram models the two entry points as
+  **free functions** (`build_df_nwb`, `qc_from_nwb_file`); the repo implements them
+  as **`Pipeline` methods** (`run_nwb`, `run_qc`).
+- **A necessary difference (and the `RawDataDict` is still present):** `RawQC`
+  consumes the `contraqctor` **`Dataset`** (real contract QC), not the plain
+  `RawDataDict` the diagram shows. The dict *is* available (`get_all_raw_data()`) —
+  the builders (`TrialTableBuilder` included) just build from the `Dataset` because
+  it's easier to navigate and access.
+
+## Component-by-component alignment
+
+| Diagram component | Current repo | Status |
+| --- | --- | --- |
+| `RawDataLoader` (`dfp.raw_data`) → `RawDataDict` | `raw_data_loader.RawDataLoader` → `get_all_raw_data()` dict (+ `.dataset`) | ✅ Aligned |
+| `TrialsBuilder` (`dfp.process.trials`) → `TrialsTable` | `processing.TrialTableBuilder` → trials `DataFrame` | ✅ Aligned |
+| `EventsBuilder` (`dfp.process.events`) → `EventsTable` | *(none — deferred/blocked)* | ⛔ Not yet built |
+| `BaseQC` interface | `qc._core.BaseQC` | ✅ Aligned |
+| `RawQC` (`dfp.qc.raw`) | `qc.raw.RawQC` | ✅ Aligned (input differs — refer to section 3 below) |
+| `ProcessedQC` (`dfp.qc.processed`) | `qc.processed.ProcessedQC` | ✅ Aligned (input differs — refer to section 4 below) |
+| `build_df_nwb` function (`dfp.package`) | `pipeline.Pipeline.run_nwb` | ◐ Same role, method vs function |
+| `qc_from_nwb_file` function (`dfp.qc`) | `pipeline.Pipeline.run_qc` | ◐ Same role, method vs function |
+| `NWBFile` (Zarr: acquisition / events / intervals) | NWB (Zarr): acquisition + `trials` (no separate events table) | ◐ Partial (events folded into acquisition) |
+| `aind-nwb-utils` (base NWB) | `create_base_nwb_file` in `run_nwb` | ✅ Aligned |
+| `aind-data-schema` `Processing` / `QualityControl` | `Processing` in `run_nwb`, `QualityControl` in `run_qc` | ✅ Aligned |
+| *(implicit: raw streams written to acquisition)* | explicit `nwb.acquisition.AcquisitionBuilder` | ➕ Extra in repo |
+
+## Where the repo differs, and why
+
+### 1. Entry points: class methods vs. free functions
+- **Diagram:** `build_df_nwb(...)` and `qc_from_nwb_file(...)` are stand-alone functions.
+- **Repo:** `Pipeline(loader).run_nwb(...)` and `.run_qc(nwb_file, ...)`.
+- **Assessment — neutral / mild trade-off.** The `Pipeline` object holds the loader
+  and per-session config (lick sources) once and exposes composable building blocks
+  (`build_acquisition`, `build_trials`, `build_nwb`, `write`), which is convenient
+  for notebooks and tests. The diagram's free functions are simpler for a capsule
+  author (call one function). Both are fine; if the capsule surface ever feels
+  heavy, thin `build_df_nwb` / `qc_from_nwb_file` wrappers around `Pipeline` would
+  reconcile the two with no loss.
+
+### 2. QC-from-NWB
+- **Diagram:** `qc_from_nwb_file` reads the `NWBFile` and runs QC from it.
+- **Repo:** `run_qc(nwb_file, ...)` takes the in-memory `NWBFile` and reads the
+  trials table + lick times back from it; raw QC runs over the loader's dataset.
+- **Assessment — aligned.** This matches the diagram's key idea (QC is driven by the
+  packaged NWB, not by rebuilding from raw). The one nuance is that raw contract-QC
+  still needs the raw dataset, so `run_qc` also relies on the loader — inherent, not
+  a divergence. `run_qc` deliberately combines the raw and processed stages so a single
+  `quality_control.json` carries both (the diagram's intent), accepting that the QC
+  capsule needs both the raw asset and the NWB. The raw-vs-processed split still exists
+  at the stage level — `RawQC`, `ProcessedQC`, and `build_quality_control` are callable
+  and composable on their own (see [Composability & usage](composability.md)); `run_qc`
+  is just the convenience combiner.
+
+### 3. `RawQC` input: `Dataset` vs `RawDataDict`
+- **Diagram:** `RawQC.run(acquisition: RawDataDict)`.
+- **Repo:** `RawQC.run(acquisition: contraqctor.Dataset)`.
+- **Assessment — the `Dataset` is the required input here (and the dict still
+  exists).** The `RawDataDict` is available via `get_all_raw_data()`; it simply isn't
+  what `RawQC` consumes. Contract QC (Harp/camera/CSV/data-contract checks) is defined
+  over the `contraqctor` `Dataset`, not a plain stream-name→DataFrame dict — a
+  `RawDataDict` cannot support those checks — so `RawQC` takes the `Dataset`. The
+  trials-table builder consumes the `Dataset` too, but there it's a choice of
+  convenience rather than necessity: its lazy tree navigation
+  (`dataset.at("Behavior").at("SoftwareEvents").at("TrialOutcome").load()`) and typed
+  models make targeted stream access far easier than digging through a fully
+  materialized dict. The dict stays available for any consumer that prefers it — so
+  the `RawDataDict` the diagram shows *is* present, not a missing piece.
+
+### 4. Processed-events representation (the deferred `EventsBuilder`)
+- **Diagram:** a first-class `EventsBuilder` → `EventsTable`, written into the NWB as
+  a processed events container; `ProcessedQC` takes `(EventsTable, TrialsTable)`.
+- **Repo (interim):** no events table. Per-event data lives as **acquisition time
+  series** (`left/right_lick_time`, `left/right_reward_delivery_time`), and
+  `ProcessedQC.run(trials, left_lick_times, right_lick_times, ...)` takes lick-time
+  arrays rather than an `EventsTable`.
+- **Assessment — deliberate (backward compatibility), plus an interim piece.** Two
+  reasons the event data lives as acquisition series today: (1) the existing pipeline
+  expects licks/rewards in that acquisition-series form, so keeping them there
+  **maintains backward compatibility** with downstream consumers, and (2) the
+  dedicated `EventsBuilder` → `EventsTable` stage is deferred. So this isn't purely a
+  stopgap. A first-class `EventsTable` is still the more standard, queryable
+  representation and would let `ProcessedQC` take a clean `(events, trials)` pair —
+  but even once `EventsBuilder` lands, the acquisition series may need to **remain**
+  (or be produced alongside the table) for backward compatibility. Treat the
+  `EventsTable` as additive, not necessarily a straight replacement.
+
+### 5. Explicit `AcquisitionBuilder`
+- **Diagram:** raw streams are written straight into the NWB acquisition inside
+  `build_df_nwb`; there is no separate acquisition builder.
+- **Repo:** `nwb.acquisition.AcquisitionBuilder` is a distinct class that both
+  packages every raw stream as a `DynamicTable` and derives the lick/reward series.
+- **Assessment — required.** Packaging the raw streams into the NWB acquisition was
+  an explicit **requirement from the scientist** (the raw data must travel in the
+  NWB), so this stage is deliberate, not incidental. It is implemented as an explicit,
+  unit-tested `AcquisitionBuilder` rather than inlined in the packaging function. Its
+  derived lick/reward series overlap with the deferred `EventsBuilder`, but even once
+  that lands, `AcquisitionBuilder` will **keep producing those series in acquisition
+  for backward compatibility** — the `EventsTable` is additive, not a replacement
+  (refer to section 4 above). So this stage
+  stays, packaging both the raw streams and the derived series.
+
+### 6. Module naming
+- **Diagram:** `dfp.raw_data`, `dfp.process.events`, `dfp.process.trials`,
+  `dfp.qc.raw`, `dfp.qc.processed`, `dfp.package`, `dfp.qc`.
+- **Repo:** `raw_data_loader`, `nwb.acquisition`, `processing`, `qc.raw`,
+  `qc.processed`, `pipeline`.
+- **Assessment — naming only.** The QC layout matches; the rest is renamed. The repo
+  has no `process.events` (deferred) and adds `nwb` + `pipeline` packages. No
+  functional difference.

docs/architecture/composability.md

@@ -0,0 +1,74 @@
+# Composability & usage
+
+Back to the [architecture overview](../architecture.md).
+
+Every stage is an independent, callable unit that works **directly off the
+`RawDataLoader`** — the `Pipeline` object is optional convenience, not a requirement.
+You can construct the builders and QC stages straight from the loader, grab any
+single piece, run only raw QC or only processed QC, or assemble your own flow (the
+example notebooks show this). One caveat: the builders and QC stages consume the
+loader (or its `.dataset`), **not** the `RawDataDict` from `get_all_raw_data()`.
+
+```mermaid
+flowchart TD
+  RAW["raw acquisition dir"] --> LOADER["RawDataLoader"]
+  LOADER -->|".dataset"| DATASET["contraqctor Dataset"]
+  LOADER -->|"get_all_raw_data()"| DICT["RawDataDict"]
+
+  subgraph nwbpath["NWB packaging"]
+    DATASET --> ACQB["AcquisitionBuilder.build_acquisition()"]
+    DATASET --> TRB["TrialTableBuilder.build()"]
+    ACQB --> BUILD["build_nwb()"]
+    TRB --> BUILD
+    BUILD --> WRITE["write() → behavior.nwb.zarr"]
+  end
+
+  subgraph qcpath["Quality control"]
+    DATASET --> RAWQC["RawQC().run(dataset)"]
+    NWBIN["behavior.nwb.zarr"] --> PROCQC["ProcessedQC().run(trials, licks)"]
+    RAWQC --> BQC["build_quality_control(metrics)"]
+    PROCQC --> BQC
+    BQC --> QCJSON["quality_control.json"]
+  end
+
+  RUNNWB["Pipeline.run_nwb()"] -.->|wraps| BUILD
+  RUNQC["Pipeline.run_qc(nwb_file)"] -.->|wraps| BQC
+```
+
+**Tier 1 — one-shot (the capsules).**
+```python
+from dynamic_foraging_processing.pipeline import Pipeline
+from dynamic_foraging_processing.raw_data_loader import RawDataLoader
+
+pipeline = Pipeline(RawDataLoader(path="/data/<acquisition>"))
+pipeline.run_nwb("/results")                       # → behavior.nwb.zarr + processing.json
+pipeline.run_qc(nwb_file, "/results")              # → quality_control.json (+ qc/ figures)
+```
+
+**Tier 2 — building blocks, straight off the loader (no `Pipeline` needed).**
+```python
+from dynamic_foraging_processing.raw_data_loader import RawDataLoader
+from dynamic_foraging_processing.nwb.acquisition import AcquisitionBuilder
+from dynamic_foraging_processing.processing import TrialTableBuilder
+
+loader = RawDataLoader(path="/data/<acquisition>")
+acquisition = AcquisitionBuilder(loader).build_acquisition()   # takes the loader
+trials = TrialTableBuilder(loader.dataset).build()             # takes the dataset
+```
+(`Pipeline.build_acquisition()` / `build_trials()` are thin wrappers over exactly
+these — use them if you already hold a `Pipeline`, or the classes directly if you
+don't.)
+
+**Tier 3 — QC à la carte (raw, processed, or both).**
+```python
+from dynamic_foraging_processing.qc import RawQC, ProcessedQC, build_quality_control
+
+raw_metrics = RawQC().run(loader.dataset)                       # raw contract QC only
+proc_metrics = ProcessedQC().run(trials, left_licks, right_licks)  # behavior QC only
+
+# assemble whichever you want into one QualityControl:
+qc = build_quality_control(raw_metrics)                          # raw only
+qc = build_quality_control(proc_metrics)                         # processed only
+qc = build_quality_control([*raw_metrics, *proc_metrics])        # both (what run_qc does)
+```
+(Lick-time arrays come from the NWB acquisition series, or `AcquisitionBuilder.get_lick_times(...)`.)