Skip to content

Commit 145f5a3

Browse files
coopbricoopbri
committed
docs(resonance): add Resonance DAW documentation
Product documentation for Resonance, a free and open-source DAW. Covers audio engine, tracks/routing, mixing, MIDI/piano roll, automation, plugin system, project format, and getting started. Accurately reflects current implementation state: mutex-based threading (lock-free planned), CLAP-only plugins (LV2 planned), WAV/FLAC export (MP3/OGG planned), desktop-only (browser planned).
1 parent b110769 commit 145f5a3

13 files changed

Lines changed: 1215 additions & 0 deletions

‎content/docs/fabric/resonance/audio-engine.mdx‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
---
2+
title: Audio Engine
3+
description: Rust-native audio processing with shared-state threading
4+
---
5+
6+
The Resonance audio engine is written in Rust and built around a pull-based audio graph. It prioritizes deterministic, glitch-free playback with sample-accurate timing.
7+
8+
## Three-Layer Architecture
9+
10+
The engine is organized into two threads:
11+
12+
```
13+
┌──────────────────────────────────────────────────┐
14+
│ Audio Callback (real-time, highest priority) │
15+
│ Pulls samples through the audio graph │
16+
│ Shares AudioGraph/Transport/Metronome via │
17+
│ Arc<Mutex<_>> with the control thread │
18+
└───────────────────────┬──────────────────────────┘
19+
│ Arc<Mutex<_>>
20+
┌───────────────────────┴──────────────────────────┐
21+
│ Control Thread (near-real-time) │
22+
│ Transport state, parameter changes, MIDI routing │
23+
│ Bridges UI events to audio-safe commands │
24+
└──────────────────────────────────────────────────┘
25+
```
26+
27+
### Audio Callback
28+
29+
The audio callback runs at the OS audio callback priority. It processes one buffer at a time (typically 128 or 256 samples) by walking the audio graph from outputs back to inputs. The callback acquires a shared `Arc<Mutex<_>>` to access the `AudioGraph`, `Transport`, and `Metronome`. If the lock is contended (e.g., the control thread is updating state), the callback outputs silence for that buffer to avoid blocking. Lock-free communication via ring buffers is planned for a future release to reduce latency.
30+
31+
### Control Thread
32+
33+
The control thread receives commands from the UI (play, stop, seek, parameter change, track arm) and translates them into audio-safe messages. It also handles MIDI routing, automation playback, and transport state management. Commands are batched per audio buffer to minimize cross-thread traffic.
34+
35+
### I/O Thread (Planned)
36+
37+
A separate I/O thread for disk streaming, audio file decode/encode, and project save/load is planned but not yet implemented. Currently, file I/O is handled on the control thread.
38+
39+
## Audio Graph
40+
41+
The audio graph is a directed acyclic graph (DAG) of processor nodes. Each node has typed input and output ports. The graph is evaluated by pulling from the master output, which recursively pulls from upstream nodes.
42+
43+
```
44+
[Audio File Reader] ──► [Gain] ──► [EQ] ──► [Bus Input]
45+
│
46+
[MIDI Source] ──► [Synth Plugin] ──► [Reverb] ──► [Bus Input]
47+
│
48+
[Bus Sum] ──► [Master]
49+
```
50+
51+
Node types include:
52+
53+
| Node | Purpose |
54+
|---|---|
55+
| `AudioFileReader` | Streams samples from disk |
56+
| `AudioInput` | Captures from hardware input |
57+
| `PluginProcessor` | Wraps a CLAP plugin instance |
58+
| `BuiltinEffect` | Gain, EQ, compressor, reverb, delay |
59+
| `MidiSynth` | Routes MIDI events to instrument plugins |
60+
| `BusSum` | Mixes multiple inputs together |
61+
| `MasterOutput` | Final output to hardware or bounce |
62+
63+
Graph modifications (adding/removing nodes, changing connections) are performed on the control thread and swapped into the audio thread atomically using a triple-buffer scheme.
64+
65+
## Key Data Structures
66+
67+
### TransportState
68+
69+
Tracks playback position, tempo, time signature, loop region, and record state. Shared between the control and audio threads via an atomic snapshot.
70+
71+
```rust
72+
struct TransportState {
73+
playing: bool,
74+
recording: bool,
75+
position_samples: u64,
76+
tempo_bpm: f64,
77+
time_signature: (u32, u32),
78+
loop_enabled: bool,
79+
loop_start: u64,
80+
loop_end: u64,
81+
sample_rate: u32,
82+
}
83+
```
84+
85+
### MixerState
86+
87+
Per-channel state: volume, pan, mute, solo, and the ordered list of plugin slots.
88+
89+
### AutomationLane
90+
91+
A sorted list of breakpoints with interpolation mode (linear or bezier). The control thread evaluates lanes each buffer and sends parameter updates to the audio thread.
92+
93+
### MidiRouter
94+
95+
Maps MIDI input ports and channels to instrument plugin nodes. Supports channel filtering, transposition, and velocity curves.
96+
97+
## Sample-Accurate Timing
98+
99+
All events (note on/off, parameter changes, transport jumps) are tagged with a sample offset within the current buffer. The audio thread applies events at the exact sample position, not at buffer boundaries. This ensures timing precision of 1/sample_rate seconds (approximately 23 microseconds at 44.1 kHz).
100+
101+
## Thread Communication
102+
103+
Currently, the audio callback and control thread share state (`AudioGraph`, `Transport`, `Metronome`) via `Arc<Mutex<_>>`. If the mutex is contended when the audio callback tries to acquire it, the callback outputs silence for that buffer rather than blocking.
104+
105+
Lock-free threading via ring buffers is planned for a future release to reduce latency. The intended design uses SPSC ring buffers for parameter changes and transport commands, and triple-buffering for graph topology swaps.

‎content/docs/fabric/resonance/automation.mdx‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
---
2+
title: Automation
3+
description: Breakpoint envelopes with sample-accurate parameter control
4+
---
5+
6+
Automation allows any parameter in the signal chain to change over time. Volume fades, filter sweeps, effect wet/dry, tempo changes, and plugin parameters can all be automated.
7+
8+
## Automation Lanes
9+
10+
Each track can have multiple automation lanes, one per parameter. Lanes appear below the track in the arrangement view.
11+
12+
To add an automation lane:
13+
14+
1. Click the automation disclosure triangle on a track header
15+
2. Click **Add Lane**
16+
3. Select the parameter to automate (volume, pan, or any plugin parameter)
17+
18+
## Breakpoint Envelopes
19+
20+
Automation data is stored as a series of breakpoints (time-value pairs) with an interpolation mode between each pair.
21+
22+
```toml
23+
[[automation]]
24+
target = "volume"
25+
points = [
26+
{ time = 0, value = -6.0, curve = "linear" },
27+
{ time = 96000, value = 0.0, curve = "linear" },
28+
{ time = 480000, value = 0.0, curve = "bezier" },
29+
{ time = 528000, value = -96.0, curve = "bezier" },
30+
]
31+
```
32+
33+
### Interpolation Modes
34+
35+
| Mode | Behavior |
36+
|---|---|
37+
| **Linear** | Straight line between points |
38+
| **Bezier** | Smooth curve with adjustable curvature (drag the curve handle) |
39+
| **Step** | Instant jump at the second point (hold previous value until the transition) |
40+
41+
## Drawing and Editing
42+
43+
### Drawing
44+
45+
1. Select the **Pencil** tool (or hold **B**)
46+
2. Click and drag in an automation lane to draw a freehand envelope
47+
3. Breakpoints are placed at the grid resolution (adjustable via snap settings)
48+
49+
### Editing
50+
51+
1. Select the **Pointer** tool
52+
2. Click a breakpoint to select it, drag to move
53+
3. Drag the curve between two points to adjust bezier curvature
54+
4. Double-click to add a new breakpoint
55+
5. Select and press **Delete** to remove breakpoints
56+
6. Box-select multiple points and drag to move them together
57+
58+
### Range Operations
59+
60+
- **Copy/Paste**: Select a range of automation, copy, and paste at another position
61+
- **Scale**: Select points and drag the top or bottom edge to scale values proportionally
62+
- **Thin**: Reduce point density while preserving the shape (useful after freehand drawing)
63+
64+
## Automatable Parameters
65+
66+
Any parameter on any node in the audio graph can be automated:
67+
68+
- **Track controls**: Volume, pan, mute
69+
- **Plugin parameters**: Every exposed parameter from CLAP plugins
70+
- **Built-in effects**: All parameters on gain, EQ, compressor, reverb, delay
71+
- **Sends**: Send level per bus
72+
- **Transport**: Tempo (for tempo automation)
73+
74+
The parameter browser shows all available targets in a tree structure grouped by track and plugin.
75+
76+
## Sample-Accurate Automation
77+
78+
Automation values are evaluated per sample, not per buffer. The control thread pre-computes automation values for each buffer and sends them to the audio thread as a list of (sample_offset, value) pairs. The audio thread applies parameter changes at the exact sample position.
79+
80+
This means:
81+
82+
- A volume fade is perfectly smooth, with no stepping artifacts
83+
- A filter cutoff sweep is continuous, not quantized to buffer boundaries
84+
- Multiple parameters can change simultaneously at different rates
85+
86+
For parameters that plugins process per-buffer (not per-sample), the audio thread sends the parameter value at the start of each processing block. CLAP plugins that support per-sample parameter events receive the full event list.

‎content/docs/fabric/resonance/browser-version.mdx‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
---
2+
title: Browser Version
3+
description: WASM-compiled audio engine with AudioWorklet bridge and offline support
4+
---
5+
6+
> **Planned**: The browser version is not yet implemented. This page describes the intended design.
7+
8+
The browser version of Resonance will run at [resonance.omni.dev](https://resonance.omni.dev). It will provide a subset of the desktop functionality, compiled to WebAssembly and running in an AudioWorklet.
9+
10+
## What's Included
11+
12+
- Audio graph engine (WASM-compiled `resonance-core`)
13+
- All built-in effects (EQ, compressor, reverb, delay)
14+
- MIDI input via WebMIDI
15+
- Automation lanes with full editing
16+
- Piano roll and arrangement views
17+
- WASM-compiled CLAP plugins (planned)
18+
- Project export (WAV, FLAC currently; MP3, OGG planned)
19+
- Offline support via service worker
20+
21+
## What's Excluded
22+
23+
- **Native CLAP/LV2 plugins**: No access to local plugin binaries from the browser
24+
- **Low-latency audio**: Browser audio typically has 10-50ms latency (vs 3-10ms on desktop)
25+
- **Filesystem access**: Projects are stored in IndexedDB, not on disk
26+
- **Hardware multi-channel I/O**: Browser audio is typically stereo only
27+
- **ASIO/JACK support**: No direct audio driver access
28+
29+
## WASM Compilation
30+
31+
The `resonance-web` crate compiles a subset of `resonance-core` to `wasm32-unknown-unknown`. The compilation excludes:
32+
33+
- Plugin host code (replaced by WASM plugin host)
34+
- Native audio I/O (replaced by AudioWorklet bridge)
35+
- File I/O (replaced by IndexedDB adapter)
36+
- Thread spawning (single-threaded in WASM)
37+
38+
The resulting `.wasm` binary is approximately 2 MB (gzipped) and loads in under 2 seconds on a modern connection.
39+
40+
## AudioWorklet Bridge
41+
42+
The audio engine runs inside an `AudioWorkletProcessor`:
43+
44+
```
45+
┌─────────────────────────────────────────────┐
46+
│ Main Thread │
47+
│ UI (React), project state, MIDI handling │
48+
│ │
49+
│ ┌─────────────────────────────────────┐ │
50+
│ │ SharedArrayBuffer / MessagePort │ │
51+
│ └──────────────────┬──────────────────┘ │
52+
└─────────────────────┼───────────────────────┘
53+
│
54+
┌─────────────────────┼───────────────────────┐
55+
│ AudioWorklet Thread │
56+
│ ┌──────────────────┴──────────────────┐ │
57+
│ │ resonance-core (WASM) │ │
58+
│ │ Audio graph evaluation │ │
59+
│ │ Built-in DSP │ │
60+
│ │ WASM plugin instances │ │
61+
│ └─────────────────────────────────────┘ │
62+
└─────────────────────────────────────────────┘
63+
```
64+
65+
Communication between the main thread and the AudioWorklet uses:
66+
67+
- **SharedArrayBuffer**: For audio data transfer (when available, with COOP/COEP headers)
68+
- **MessagePort**: For commands and parameter changes (fallback if SAB unavailable)
69+
70+
The AudioWorklet processes 128 samples per callback (the Web Audio API default). The WASM engine is called once per callback to fill the output buffer.
71+
72+
## CLAP-in-WASM
73+
74+
CLAP plugins that are compiled to WASM can run in the browser. The workflow:
75+
76+
1. Plugin developer compiles their CLAP plugin to `wasm32-wasi`
77+
2. The `.wasm` bundle is uploaded to the Resonance plugin registry
78+
3. Users browse and install WASM plugins from within the browser version
79+
4. The plugin runs inside the AudioWorklet alongside the engine
80+
81+
WASM plugins share the same AudioWorklet thread. CPU-heavy plugins may cause audio dropouts since there is no real-time thread priority in the browser.
82+
83+
## Offline Support
84+
85+
The browser version works offline after the initial load:
86+
87+
- **Service Worker**: Caches the app shell, WASM binary, and static assets
88+
- **IndexedDB**: Stores projects, audio files, and plugin presets locally
89+
- **Background Sync**: Queues cloud sync operations for when connectivity returns
90+
91+
The app registers a service worker on first visit. Subsequent visits load from cache, with updates applied in the background.
92+
93+
## Use Cases
94+
95+
The browser version is best suited for:
96+
97+
- **Sketching**: Quickly capture an idea without launching the desktop app
98+
- **Review**: Open a shared project to listen and leave feedback
99+
- **Collaboration**: Join a live session from any device
100+
- **Education**: Learn music production without installing software
101+
- **Mobile**: Basic editing on tablets (limited by screen size and touch input)
102+
103+
For production work (recording, mixing, mastering), the desktop app is recommended.

‎content/docs/fabric/resonance/collaboration.mdx‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
---
2+
title: Collaboration
3+
description: Live session sharing via WebRTC with host authority
4+
---
5+
6+
> **Planned**: Real-time collaboration is not yet implemented. This page describes the intended design.
7+
8+
Resonance supports real-time collaboration, allowing multiple users to work on the same project simultaneously.
9+
10+
## Live Sessions
11+
12+
A live session connects multiple Resonance instances (desktop or browser) over WebRTC:
13+
14+
1. The host opens a project and clicks **Share > Start Live Session**
15+
2. Resonance generates a session link
16+
3. Collaborators open the link (desktop app or browser)
17+
4. All participants see the same arrangement, mixer, and transport state
18+
19+
Live sessions require an Omni account for all participants. Sign in via HIDRA (Omni's identity service).
20+
21+
## Host Authority Model
22+
23+
The session host has full authority over the project:
24+
25+
| Action | Host | Collaborator |
26+
|---|---|---|
27+
| Play/stop/seek | Yes | Yes (forwarded to host) |
28+
| Add/delete tracks | Yes | No |
29+
| Edit clips | Yes | Delegated tracks only |
30+
| Change routing | Yes | No |
31+
| Adjust mixer | Yes | Delegated tracks only |
32+
| Add/remove plugins | Yes | No |
33+
| Export | Yes | No |
34+
| Save | Yes | No |
35+
36+
Collaborators can always listen to the full mix and view the arrangement. Editing permissions are granted per track by the host.
37+
38+
## Track Delegation
39+
40+
The host can delegate control of individual tracks to collaborators:
41+
42+
1. Right-click a track header
43+
2. Select **Delegate to...** and choose a participant
44+
3. The delegated user can now edit clips, adjust volume/pan, and modify plugins on that track
45+
46+
Delegation can be revoked at any time. A track can only be delegated to one collaborator at a time. The host always retains the ability to override.
47+
48+
## Real-Time State Sync
49+
50+
Session state is synchronized using a combination of:
51+
52+
- **State diffs**: Track, mixer, and arrangement changes are transmitted as compact diffs over a WebRTC data channel
53+
- **Transport sync**: Play/stop/seek commands are broadcast with timestamp correction for latency
54+
- **Parameter updates**: Plugin and mixer parameter changes stream in real-time
55+
56+
The host is the source of truth. If the host and a collaborator make conflicting edits, the host's state wins.
57+
58+
## Audio Monitoring
59+
60+
Collaborators hear the mix as rendered by the host:
61+
62+
1. The host's audio engine renders the full mix
63+
2. A compressed audio stream (Opus codec) is sent via WebRTC to each collaborator
64+
3. Collaborators hear the mix with a small delay (typically 50-200ms depending on network)
65+
66+
This approach means collaborators do not need to load plugins or audio files locally. The host bears the full processing load.
67+
68+
### Low-Latency Mode
69+
70+
For recording together, the host can enable low-latency mode:
71+
72+
- Each participant sends their raw audio input via WebRTC
73+
- The host's engine mixes all inputs in real-time
74+
- Latency depends on network quality (typically 20-80ms on good connections)
75+
76+
Low-latency mode requires a stable connection and works best on local networks.
77+
78+
## Requirements
79+
80+
- All participants need an Omni account (free tier is sufficient)
81+
- Modern browser or Resonance desktop app
82+
- WebRTC-capable network (most networks work, some corporate firewalls may block peer-to-peer)
83+
- The Omni signaling server brokers the initial connection, after which traffic is peer-to-peer

0 commit comments

Comments
 (0)